Opened 13 years ago

Closed 12 years ago

#3041 closed task (fixed)

MythMusic scanning improvements

Reported by: stuartm Owned by: stuartm
Priority: minor Milestone: 0.21
Component: mythmusic Version: head
Severity: medium Keywords:
Cc: Ticket locked: no

Description

There are a number of things that need to be done to improve and speed up the mythmusic scanning. Xris is working on indexes and optimising the queries, I'm looking at the code in general.

This ticket just gives me a place to coordinate things.

Attachments (2)

mythmusic_scanner.diff (25.1 KB) - added by stuartm 13 years ago.
ID caching and a couple of other tweaks
mythmusic_lengthcalc_new.diff (5.8 KB) - added by stuartm 13 years ago.
Uses Xing header for VBR mp3 length calculation

Download all attachments as: .zip

Change History (18)

comment:1 Changed 13 years ago by stuartm

(In [12714]) Don't query the artist_id twice when scanning if compilation_artist and artist are the same.

Refs #3041

comment:2 Changed 13 years ago by stuartm

Status: newassigned

Need to implement some clean-up of unused artists, albums and genres to prevent those tables becoming bloated.

comment:3 Changed 13 years ago by stuartm

(In [12744]) Refs #3041

NOTE: This commit may break or negatively impact the performance of mythmusic. I'm committing it in this possibly broken state before it becomes larger and harder to handle.

Includes the following changes:

Change the music_songs lastplay column into a datetime field, it was previously a timestamp which meant it was erroneously updated every time a change was made to the record.

Breaks the scanning code out into it's own class.

Adds a music_directories table to normalise the information currently stored in music_songs filename. The code doesn't take full advantage of this change yet but ultimately it will have some benefits for speed and directory browsing. For the moment, because both new and old code is in use, this change will actually slow down scanning!

Renames all the member variables in metadata.cpp/h to comply with the coding standards and improve readability.

comment:4 Changed 13 years ago by Simon Kenyon

metatdata.cpp doesn't compile at line 417. you have:

if ()

s1; s2;

else

s3;

s1 and s2 need to be within a block.

Changed 13 years ago by stuartm

Attachment: mythmusic_scanner.diff added

ID caching and a couple of other tweaks

comment:5 Changed 13 years ago by stuartm

Attached a patch which has the balance of the scanner changes, it cuts down the number of db queries significantly. The albumid cache doesn't work with compilation albums, but that's not a significant problem unless you have a lot of them. There is still room for improvement and optimisation.

Feedback on the patch (which only applies to HEAD) would be appreciated.

Right now one of the slowest steps in scanning mp3s is calculating the track length. One improvement that I can make is to use the TLEN id3v2 frame if it exists. There just might be a quicker method for obtaining the length but I'll have to look closer at what we're doing now.

comment:6 Changed 13 years ago by anonymous

I've just given this a try and it seems to work fine. My music collection probably isn't a good test for this because I do have a lot of compilation albums and also lots of directories with only one or two tracks in them even so I did see a speed improvement.

With a clean DB scanning 7782 tracks took

  • with HEAD 7min 52secs
  • with patch 6min 03secs

Manually copying an album with 18 tracks into the music directory and performing a rescan

  • with HEAD 37secs
  • with patch 29secs

comment:7 Changed 13 years ago by stuartm

(In [12878]) Refs #3041

Move the QString replace calls outside the loop so they are only done once per file.

comment:8 Changed 13 years ago by stuartm

(In [12876]) Refs #3041

Adds caching of artist, album, genre and directory ids to mythmusic scanning to reduce the number of database queries.

Removed some unnecessary queries and steps in adding or updating a file.

Finally a fix for reading filenames containing non-ascii characters.

comment:9 in reply to:  8 Changed 13 years ago by briand <turbo@…>

Replying to stuartm:

(In [12876]) Refs #3041

Removed some unnecessary queries and steps in adding or updating a file.

I believe some of the chaff removed may have, in fact, been necessary. :-)

My database now contains 8735 artists in music_artists, but only 109 songs in music_songs. Probably not coincidentally, 109 equals the number of new mp3 files I added to mythmusic on Sunday evening. The original 15,267 entries in the music_songs table prior to "scanning for new music" seem to have vanished.

I will agree that this does, in fact, make scanning through my music_songs table considerably faster... but is probably not the optimal solution we were hoping for. :)

I'm at work now, but I'll take a look at what exactly changed in filescanner.cpp when I get home tonight.

comment:10 Changed 13 years ago by stuartm

(In [12887]) If music_directories.path is empty (i.e. the base directory) don't add a trailing slash. Refs #3041

Changed 13 years ago by stuartm

Uses Xing header for VBR mp3 length calculation

comment:11 Changed 13 years ago by stuartm

(In [13104]) Refs #3041

Add reading of length from id3 TLEN tags.

Add calculating length using the frame count in the Xing header for VBR mp3s.

For mp3s using either TLEN or Xing headers this reduces the time to scan the file. A full scan of collections with many such mp3s should be noticably faster - the bigger the mp3, the better the overall improvement.

comment:12 Changed 13 years ago by stuartm

(In [13151]) Replaces libid3tag with taglib, for mp3s only at the moment but that may change to include Vorbis Ogg/Flac? tags.

This removes the dependancy on libid3tag and creates a new one on libtag (aka taglib).

You will need to RE-RUN CONFIGURE and I'd recommend a MAKE DISTCLEAN.

Refs #3041

comment:13 Changed 13 years ago by stuartm

(In [13152]) The libid3hack files should have been deleted in [13151]. Refs #3041

comment:14 Changed 13 years ago by stuartm

(In [13155]) Implements the TLEN frame check with the taglib api and uncomments a line which was accidently left commented in [13151]

Refs #3041

comment:15 Changed 13 years ago by stuartm

(In [13323]) Add some basic doxygen comments to the functions in the filescanner class.

Refs #3041

comment:16 Changed 12 years ago by stuartm

Resolution: fixed
Status: acceptedclosed
Note: See TracTickets for help on using tickets.