MythMusic scanning improvements

[stuartm]

There are a number of things that need to be done to improve and speed up the mythmusic scanning. Xris is working on indexes and optimising the queries, I'm looking at the code in general.

This ticket just gives me a place to coordinate things.

[stuartm]

(In [12714]) Don't query the artist_id twice when scanning if compilation_artist and artist are the same.

Refs #3041

[stuartm]

Need to implement some clean-up of unused artists, albums and genres to prevent those tables becoming bloated.

[stuartm]

(In [12744]) Refs #3041

NOTE: This commit may break or negatively impact the performance of mythmusic. I'm committing it in this possibly broken state before it becomes larger and harder to handle.

Includes the following changes:

Change the music_songs lastplay column into a datetime field, it was previously a timestamp which meant it was erroneously updated every time a change was made to the record.

Breaks the scanning code out into it's own class.

Adds a music_directories table to normalise the information currently stored in music_songs filename. The code doesn't take full advantage of this change yet but ultimately it will have some benefits for speed and directory browsing. For the moment, because both new and old code is in use, this change will actually slow down scanning!

Renames all the member variables in metadata.cpp/h to comply with the coding standards and improve readability.

[Simon Kenyon]

metatdata.cpp doesn't compile at line 417. you have:

if ()

s1; s2;

else

s3;

s1 and s2 need to be within a block.

[stuartm]

ID caching and a couple of other tweaks

[stuartm]

Attached a patch which has the balance of the scanner changes, it cuts down the number of db queries significantly. The albumid cache doesn't work with compilation albums, but that's not a significant problem unless you have a lot of them. There is still room for improvement and optimisation.

Feedback on the patch (which only applies to HEAD) would be appreciated.

Right now one of the slowest steps in scanning mp3s is calculating the track length. One improvement that I can make is to use the TLEN id3v2 frame if it exists. There just might be a quicker method for obtaining the length but I'll have to look closer at what we're doing now.

[anonymous]

I've just given this a try and it seems to work fine. My music collection probably isn't a good test for this because I do have a lot of compilation albums and also lots of directories with only one or two tracks in them even so I did see a speed improvement.

With a clean DB scanning 7782 tracks took

• with HEAD 7min 52secs
• with patch 6min 03secs

Manually copying an album with 18 tracks into the music directory and performing a rescan

• with HEAD 37secs
• with patch 29secs

[stuartm]

(In [12878]) Refs #3041

Move the QString replace calls outside the loop so they are only done once per file.

[stuartm]

(In [12876]) Refs #3041

Adds caching of artist, album, genre and directory ids to mythmusic scanning to reduce the number of database queries.

Removed some unnecessary queries and steps in adding or updating a file.

Finally a fix for reading filenames containing non-ascii characters.

[briand <turbo@…>]

Replying to stuartm:

(In [12876]) Refs #3041

Removed some unnecessary queries and steps in adding or updating a file.

I believe some of the chaff removed may have, in fact, been necessary. :-)

My database now contains 8735 artists in music_artists, but only 109 songs in music_songs. Probably not coincidentally, 109 equals the number of new mp3 files I added to mythmusic on Sunday evening. The original 15,267 entries in the music_songs table prior to "scanning for new music" seem to have vanished.

I will agree that this does, in fact, make scanning through my music_songs table considerably faster... but is probably not the optimal solution we were hoping for. :)

I'm at work now, but I'll take a look at what exactly changed in filescanner.cpp when I get home tonight.

[stuartm]

(In [12887]) If music_directories.path is empty (i.e. the base directory) don't add a trailing slash. Refs #3041

[stuartm]

Uses Xing header for VBR mp3 length calculation

[stuartm]

(In [13104]) Refs #3041

Add reading of length from id3 TLEN tags.

Add calculating length using the frame count in the Xing header for VBR mp3s.

For mp3s using either TLEN or Xing headers this reduces the time to scan the file. A full scan of collections with many such mp3s should be noticably faster - the bigger the mp3, the better the overall improvement.

[stuartm]

(In [13151]) Replaces libid3tag with taglib, for mp3s only at the moment but that may change to include Vorbis Ogg/Flac? tags.

This removes the dependancy on libid3tag and creates a new one on libtag (aka taglib).

You will need to RE-RUN CONFIGURE and I'd recommend a MAKE DISTCLEAN.

Refs #3041

[stuartm]

(In [13152]) The libid3hack files should have been deleted in [13151]. Refs #3041

[stuartm]

(In [13155]) Implements the TLEN frame check with the taglib api and uncomments a line which was accidently left commented in [13151]

Refs #3041

[stuartm]

(In [13323]) Add some basic doxygen comments to the functions in the filescanner class.

Refs #3041