Opened 11 years ago

Closed 10 years ago

#6440 closed enhancement (fixed)

New Utility - Jamu - Just.Another.Metadata.Utility

Reported by: r.d.vaughan@… Owned by: Isaac Richards
Priority: minor Milestone: 0.22
Component: Contributed Scripts & Apps Version: unknown
Severity: low Keywords: metadata utility mythui
Cc: Rob Smith Ticket locked: no

Description

Jamu - Just.Another.Metadata.Utility

This python script performs a variety of utility functions on mythvideo meta data and its associated video files and directories. The focus is to maximize your experience with the new MythUI only available in trunk 0.22. The primary sources of graphics and meta data for movies is themoviedb.com wiki and for TV Series thetvdb.com wiki.

Refer to Jamu's wiki page http://www.mythtv.org/wiki?title=Jamu for full details.

Attachments (12)

jamu_v0.3.1.tar.gz (39.7 KB) - added by r.d.vaughan@… 11 years ago.
Jamu - Just.Another.Metadata.Utility
jamu_v0.3.2.tar.gz (40.5 KB) - added by r.d.vaughan@… 11 years ago.
Jamu - v0.3.2 - Complete replacement of previous versions
jamu_v0.3.3.tar.gz (40.8 KB) - added by r.d.vaughan@… 11 years ago.
New version of Jamu - A complete replacement of previous versions
jamu_v0.3.4.tar.gz (45.2 KB) - added by r.d.vaughan@… 11 years ago.
Jamu v0.3.4 Completely replaces all previous versions !!
jamu_v0.3.5.tar.gz (47.8 KB) - added by r.d.vaughan@… 10 years ago.
Jamu v0.3.5 Completely replaces all previous versions !!
jamu_v0.3.6.tar.gz (48.8 KB) - added by r.d.vaughan@… 10 years ago.
Jamu v0.3.6 Completely replaces all previous versions !!
jamu_v0.3.7.tar.gz (53.7 KB) - added by r.d.vaughan@… 10 years ago.
Jamu v0.3.7 Completely replaces all previous versions !!
jamu_v0.3.8.tar.gz (53.8 KB) - added by r.d.vaughan@… 10 years ago.
Jamu v0.3.8 Completely replaces all previous versions !!
MythVideo_SG_compliant.patch (8.3 KB) - added by r.d.vaughan@… 10 years ago.
MythVideo?.py binding - Storage Group compliance
jamu_v0.3.9.tar.gz (55.5 KB) - added by r.d.vaughan@… 10 years ago.
Jamu v0.3.9 Completely replaces all previous versions !!
jamu_v0.4.0.tar.gz (55.9 KB) - added by r.d.vaughan@… 10 years ago.
Jamu v0.4.0 Completely replaces all previous versions !!
jamu_v0.4.1.tar.gz (56.0 KB) - added by r.d.vaughan@… 10 years ago.
Jamu v0.4.1 Completely replaces all previous versions !!

Download all attachments as: .zip

Change History (31)

Changed 11 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.1.tar.gz added

Jamu - Just.Another.Metadata.Utility

comment:1 Changed 11 years ago by r.d.vaughan@…

!!!!! Jamu v0.3.2 COMPLETELY REPLACES ALL PREVIOUS VERSIONS !!!!

Jamu - v0.3.2

1) Fixed bug where some poster downloads were unnecessary

2) Fixed bug where the mythtv database was updated for no reason

3) Fixed bug in jamu-example.conf "min_poster_size" variable had '=' not ':'

4) Fixed bug where a URL containing Unicode characters would abort the script

5) Using ffmpeg added setting accurate video run time in minutes. A hack but lacked python method to find audio/video properties which did not require additional libraries being installed. Most people have ffmpeg already installed. If ffmpeg is not installed run time from meta data sources is used instead.

Using Jami instructions at http://www.mythtv.org/wiki/Jamu

Changed 11 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.2.tar.gz added

Jamu - v0.3.2 - Complete replacement of previous versions

comment:2 Changed 11 years ago by tsx_5@…

Hi,

Tried to use your script, however, I haven't gotten past go... Here's the error message I get when I attempt to run it. My quess is some funky Anime names, but not totally sure.

Traceback (most recent call last):

File "/usr/local/share/mythtv/mythvideo/scripts/jamu.py", line 3579, in <module>

main()

File "/usr/local/share/mythtv/mythvideo/scripts/jamu.py", line 3564, in main

process.processMythTvMetaData()

File "/usr/local/share/mythtv/mythvideo/scripts/jamu.py", line 2895, in processMythTvMetaData

validFiles = self._findMissingInetref()

File "/usr/local/share/mythtv/mythvideo/scripts/jamu.py", line 2703, in _findMissingInetref

validFiles = self._processNames(allFiles, verbose = self.configdebug_enabled?, movies=self.configmythmovie?)

File "/usr/local/share/mythtv/mythvideo/scripts/jamu.py", line 1781, in _processNames

allEps.append({ 'file_seriesname':movie.encode('utf8'),

UnicodeDecodeError?: 'ascii' codec can't decode byte 0xc2 in position 32: ordinal not in range(128)

Changed 11 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.3.tar.gz added

New version of Jamu - A complete replacement of previous versions

comment:3 Changed 11 years ago by r.d.vaughan@…

Jamu v0.3.3 changes:

1) Added logic to skip any video with a inetref of '99999999'. Protects manually entered meta data and graphics.

2) Fixed issues with file names containing Unicode characters.

3) Fixed other issues with Unicode.

4) Modified regex strings to parse Unicode file names for season and episode numbers.

5) jamu-example.conf was modified (e.g. regex string format). Please update your own Jamu configuration file(s). It is suggested that your Jamu configuration file(s) are recreated using the new jamu-example.conf to avoid compatibility issues.

6) Overall improved multi-language support.

Please note that an additional patch is required for this new version of Jamu. See the Jamu wiki page for updated installation details. http://www.mythtv.org/wiki/Jamu

Changed 11 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.4.tar.gz added

Jamu v0.3.4 Completely replaces all previous versions !!

comment:4 Changed 11 years ago by r.d.vaughan@…

!! Jamu v0.3.4 REPLACES ALL PREVIOUS VERSIONS !!

1) Added logic to skip any secondary source's meta data plot less than 10 words.

2) Fully initialise a new record so warning messages are not displayed.

3) Replace plot meta data line-feeds with a space (e.g. Space Cowboys plot contains line-feeds). Mythvideo does not expect line-feeds in a plot and therefore the plot does not display correctly.

4) Significant improvements in combining meta data between primary and secondary data sources to get the most complete set of meta data possible.

5) Remove 'tmdb.pl' calls and instead use the native tmdb api through a python interface.

6) Added detection of broken symbolic links and fixed/updated those links.

7) Fixed inconsistencies in graphics file extensions (as received from the various sources), made all extensions lower-case and changed ".jpeg" to ".jpg".

8) The interface for the selection of a Movie's reference number has been expanded to allow selection from themoviedb.com or imdb.com. This enhancement covers situations where a movie has not yet been added to themoviedb.com wiki or when a movie title search on themoviedb.com does not yield acceptable results. If the movie's reference number is selected from IMDB, Jamu will still find the movie on themoviedb.com wiki if it exists with that IMDB number.

9) In trunk, there is an intent to have the inetref number reflect the source of the meta data. To accommodate that intent Jamu now sets the inetref field to the source's own reference number (e.g. TMDB number rather than IMDB number when the movie is on themoviedb.com).

Jamu will rename any existing graphics from the imdb# to the source's reference number and update the mythvideo data base record. Any movies that are not yet on themoviedb.com will continue to use their IMDB numbers both in the inetref field and as their graphics name. At a later time if that movie is added to themoviedb.com Jamu will automatically detect this change and rename graphic files and update the mythvideo data base including changing the inetref number.

A limitation to this feature is that if you use a secondary data source (e.g. fetch_poster.py or imdbpy.py) that requires an IMDB number and themoviedb.com does not have the IMDB number for that movie the secondary source will be ignored. At this time the mythtv data base only allows storage of one reference number per video file. If this is an issue then add the IMDB number to the movie's entry on themoviedb.com wiki. Better yet add any other missing meta data or graphics and a secondary source will not be required.

10) The Jamu user configuration file (see jamu-example.conf) has a new variable (%%(imdb)s) for secondary sources. This field indicates that the secondary source requires an IMDB number as input.

11) Added to the Jamu maintenance report is a count of how many movies that are still using IMDB#s as their interef value. Also reported is a list of which movies are still using an IMDB numbers. After running this new version of Jamu (0.3.4) for a second time the new count indicates the number of movies in your collection that have not been entered into themoviedb.com.

comment:5 Changed 10 years ago by r.d.vaughan@…

!! Jamu v0.3.5 REPLACES ALL PREVIOUS VERSIONS !!

1) Fixed a bug when themoviedb.com times out from an api request.

2) Fixed a bug with utf8 directory names.

3) Fixed a bug where a TMDB Poster image was not found when it was really available.

4) A few documentation corrections.

5) Added code to avoid aborting script when themoviedb.com has problems. The issue is reported but the scripts continues processing.

6) Added option (-MV) to download graphics for Scheduled and Recorded programs both TV shows and Movies. See the Jamu wiki page for details: http://www.mythtv.org/wiki/Jamu

7) Modified the (-J) Janitor option to avoid deleting graphics for Scheduled and Recorded videos.

Changed 10 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.5.tar.gz added

Jamu v0.3.5 Completely replaces all previous versions !!

comment:6 Changed 10 years ago by tsx_5@…

Hi again,

Thought I would give this another try.... But when I run 3.5 I get the following error(s):

/usr/lib/python2.6/site-packages/MySQLdb/init__.py:34: DeprecationWarning?: the sets module is deprecated

from sets import ImmutableSet?

CRITICAL - Backend has version 45 and we speak version 39 To search IMDB.com for movies the IMDbPy library must be installed.Check your installation's repository or check the following link.from (http://imdbpy.sourceforge.net/?page=download) ! The MythTV python interface is not installed. MythTV meta data cannot be updated

MythTV.py is located in the same directory as jamu.py

comment:7 Changed 10 years ago by r.d.vaughan@…

tsx_5, The errors you have are symptoms of configuration and/or installation issues. I have emailed you more detailed comments and suggestions.

comment:8 Changed 10 years ago by r.d.vaughan@…

!! Jamu v0.3.6 replaces all previous versions !!

1) Fixed bug when searching themoviedb.com for a movie by title or alternate title.

2) Fixed bug with the Scheduled and Recorded graphic file naming convention for movies.

3) Increased accuracy of TMDB movie searching and matching. Makes it easier for Mass updates and finding graphics for the Scheduled and Recorded programs.

4) If interactive mode is selected but an exact match is found for a movie then the exact match is chosen and user input is not required.

5) Added additional warning messages when there are issues importing the MythTV python bindings.

6) Set up for transition to TMDB's v2.1 api (currently beta) which adds language support among other improvements.

Changed 10 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.6.tar.gz added

Jamu v0.3.6 Completely replaces all previous versions !!

comment:9 Changed 10 years ago by Tim Jordan <tim@…>

Seem to keep getting error like this:

Entry exists in MythDB but category is 0 and year is 1895 (default values). Updating (Sunset Boulevard).

1 -> Sunset Boulevard (1991) # http://www.imdb.com/title/tt0158253 2 -> Sunset Boulevard (2010) # http://www.imdb.com/title/tt0469609 3 -> Sunset Boulevard - 27 Meilen Amerika (1997) # http://www.imdb.com/title/tt0934965 4 -> 'Sunset Blvd.': A Look Back (2002) # http://www.imdb.com/title/tt0392805 5 -> 24/7 - Sunset Boulevard (2002) # http://www.imdb.com/title/tt0934361 6 -> Boulevard des assassins (1982) # http://www.imdb.com/title/tt0083684

Traceback (most recent call last):

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 4666, in <module>

main()

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 4651, in main

process.processMythTvMetaData()

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 3986, in processMythTvMetaData

inetref = self._getTmdbIMDB(available_metadatatitle?)

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 2507, in _getTmdbIMDB

inetref = ui.selectSeries(movies)

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 795, in selectSeries

allSeries = self._displaySeries(allSeries)

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 786, in _displaySeries

allSeries[key]sid?

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 251, in write

self.out.write(obj.encode(self.encoding))

UnicodeEncodeError?: 'ascii' codec can't encode character u'\xe9' in position 22: ordinal not in range(128)

Only seems to happen when the imdb? script return multiple matches.

comment:10 Changed 10 years ago by Tim Jordan <tim@…>

and this is the other error that seems to crop up Entry does not exist in MythDB. Adding (Bambi). Traceback (most recent call last):

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 4666, in <module>

main()

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 4651, in main

process.processMythTvMetaData()

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 3986, in processMythTvMetaData

inetref = self._getTmdbIMDB(available_metadatatitle?)

File "/usr/share/mythtv/mythvideo/scripts/jamu.py", line 2442, in _getTmdbIMDB

TMDB_movies.append({'name': "%s (%s)" % (movietitle?, movieyear?), u'sid': movieinetref?})

KeyError?: 'year'

Changed 10 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.7.tar.gz added

Jamu v0.3.7 Completely replaces all previous versions !!

comment:11 Changed 10 years ago by r.d.vaughan@…

!! Jamu v0.3.7 completely replaces all previous versions !!

1) Cleaned up documentation consistence for Warning and Error messages.

2) Removed redundant code.

3) Sync up with v1.0 of tvdb_api and the new way to assign tvdb api key. The latest version of ttvdb.py (v0.9.6 - ticket http://svn.mythtv.org/trac/ticket/6242) is required.

4) Fixed a bug when TMDB does not have a 'year' for a movie (e.g. 'Bambi')

5) Fixed a bug when ffmpeg cannot find the true time length in minutes of a video file

6) Fixed a bug in the file rename option (-MF) correcting lower case TV series names to the series name as they appears on thetvdb.com

7) Added Jamu compatibility with or without the application of the MythTV.py Ticket #6678

8) Remove all use of the python binding MythVideo?.py "pruneMetadata" function as it is not yet 'Videos' Storage Group aware and would delete records from the Mythvideo table for all video files with a relative filename.

9) Jamu is now restricted to only run on PC's that are a MythTV backend (master or slave). This has been done to avoid potential issues with multiple backends.

10) Except with the "Scheduled and Recorded" option (-MW) Jamu will exit if the backend Jamu is running on, has any Video and/or graphic file storage groups. Currently Jamu does not support these types of storage groups.

11) The Janitor option (-MJ) now checks if the plugins MythGallery, MythGame? and/or MythMusic are using the same graphics directories as MythVideo?. If they share directories the Janitor (-MJ) option will exit without removing any graphics files. Messages are displayed indicating which directories are in conflict.

12) Jamu will now restrict itself to updating only videometadata records whose video files are local to the current host backend. In the case where a user has multiple backends Jamu must run on each of those backends.

13) Added the detection of video or graphic files residing on a NFS mount. If detected Jamu displays a message and exists without any processing. A new option for NFS (-N) when added to any (-M) option string will allow a user to override this check and Jamu will continue processing. The NFS check prevents the situation where a user has more than one MythTV backend using the same directories to store either video files or graphics. Jamu's option (-M) can adversely effect the users MythTV database by mistakenly adding videos from other backends or with the Janitor (-MJ) option mistakenly remove graphics files. If a user only have one backend or does not mix the Video or graphic file NFS directories between backends and still want to use Jamu add the options (N) to your option string e.g. (-MJN), which will skip the NFS check and allow processing to continue.

14) Added an option (-MG) to allow Jamu to perform a "best guess" for a video file's internet reference number from themoviedb.com, imdb.com or thetvdb.com. For accurate guessing the video file name must be very close to the names found on themoviedb, imdb or thetvdb web sites. When more than one possibility exists the video file will be skipped. This option works especially well when the video file named is using Schedule Direct's - TV series: title and subtitle or a Movie's: title and original air date information.

15) Added to the existing TV episode video file renaming (-MF) option. Now movie video files can also be renamed to the format "title (year)" e.g. "The Duchess (2008)". If themoviedb.com has no year for the movie then only the movie title will be used when renaming. Any existing metadata is preserved.

comment:12 Changed 10 years ago by Rob Smith

Cc: Rob Smith added

Changed 10 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.8.tar.gz added

Jamu v0.3.8 Completely replaces all previous versions !!

comment:13 Changed 10 years ago by r.d.vaughan@…

!! Jamu release v0.3.8 replaces all previous version !!

1) Made changes to sync up with MythTV trunk change set [r21138].

2) Jamu now handles TVDB's change from a 5 digit inetref number to 6 digits or more.

If you have a previous version of Jamu installed and update to change set r21138 or higher then you must update to Jamu v0.3.8

Changed 10 years ago by r.d.vaughan@…

MythVideo?.py binding - Storage Group compliance

comment:14 Changed 10 years ago by r.d.vaughan@…

MythVideo?.py binding patch:

All changes are backward compatible except for "pruneMetadata()" as it was broken when applied against files with relative paths.

1) Changed "pruneMetadata()" to be compatible with "Videos" storage groups. Previously any video file that was stored as a relative path was removed whether it really existed or not. These changes support single and multiple videos storage groups for a specific backend. This change forces the prune function to be run on a backend for all videometadata records that have relative file paths.

2) Added the option to specify a backend host name for the following routines: getMetadataId(), getTitleId() and hasMetadata(). This allows a search to target specific back ends and their video storages groups.

3) Fixed a minor bug in getCastId() which ensured that it is backward compatible.

This binding change is a precursor to Jamu's eventual support of storage groups.

comment:15 Changed 10 years ago by robertm

(In [21275]) Refs #6440. Patch from Doug Vaughan to update MythVideo? python bindings to support Video Storage Groups.

Changed 10 years ago by r.d.vaughan@…

Attachment: jamu_v0.3.9.tar.gz added

Jamu v0.3.9 Completely replaces all previous versions !!

comment:16 Changed 10 years ago by r.d.vaughan@…

Jamu v0.3.9 Replaces all previous versions!

1) Fixed a unicode bug with file paths.

2) Fixed a unicode bug with some URLs containing UTF8 characters.

3) Fixed a bug were a bad image file could abort the script.

4) Fixed a bug where a secondary source failed to download a graphic but return a message instead of a file. The message was being incorrectly interpretted as the graphics name.

5) Fixed a bug where a TV series with out a season specific poster or banner would get repeatedly download.

6) Check accessability (Read and Write) to directories and files before including them in files/directories to process.

7) Add the ability to process Storage Groups for all Videos and graphics.

8) Jamu now uses MythVideo?.py binding's Genre and Cast routines

9) Changed all subdirectory cover art to a copied graphic file "folder.jpg/png" to conform to the Storage Group standard. This also works for local subdirs.

Changed 10 years ago by r.d.vaughan@…

Attachment: jamu_v0.4.0.tar.gz added

Jamu v0.4.0 Completely replaces all previous versions !!

comment:17 Changed 10 years ago by r.d.vaughan@…

Jamu v0.4.0 Replaces all previous versions!

1) Removed a few lines of debugging code.

2) Fixed the check that confirms that all Video and graphic directories are read and writeable.

3) Fixed a bug where under rare circumstances a graphic would be repeatedly downloaded.

4) Made the installation of the python IMDbPy library mandatory.

5) For all movies IMDB numbers will be used instead of converting to TMDB numbers. This is done to maintain consistency with MythVideo? movie inetref numbers.

Changed 10 years ago by r.d.vaughan@…

Attachment: jamu_v0.4.1.tar.gz added

Jamu v0.4.1 Completely replaces all previous versions !!

comment:18 Changed 10 years ago by r.d.vaughan@…

Jamu v0.4.1 Replaces all previous versions!

1) Fixed an obscure video file rename issue with the (-MF) option

comment:19 Changed 10 years ago by robertm

Resolution: fixed
Status: newclosed

(In [21506]) Closes #6440. Adds JAMU, Just Another Metadata Utility, to MythVideo?. This is a great python script that will massively ease the transition to MythVideo?'s new metadata sources in .22. Does mass metadata grab from open and legit sites (TVDB and TMDB). Renames filenames if the user wishes. Downloads fanart/banners/screenshots/covers for all items. Populates metadata for all MythVideo? items.

This script also can be cron'ed to download fanart/etc. for upcoming and current recordings, and to clean up unused metadata images, amongst many, many other functions. Please see the jamu.README file and the wiki for more details.

Note: See TracTickets for help on using tickets.