Opened 10 years ago

Closed 10 years ago

Last modified 9 years ago

#9859 closed Bug Report - General (fixed) grabber script takes too long to process shows with thousands of episodes

Reported by: espenaf@… Owned by:
Priority: minor Milestone:
Component: MythTV - General Version: 0.24-fixes
Severity: low Keywords:
Cc: Ticket locked: no


The metadata lookup in mythvideo seems to have a 30 second timeout. When trying to lookup metadata for "The Daily Show with Jon Stewart" the timeout kills the process before it is finished fetching the data.

I don't have a slow connection, nor have I hammered the server.The metadata lookup has been tried at 5 to 10 different instances in the span of 2 weeks. All requests timeout for this tv show only. I might have been extremely unlucky with the timing of all my request tries, but I doubt it. Here's a frontend log of one of the tests:

2011-05-28 17:48:11.375 Using protocol version 63 2011-05-28 17:48:17.627 Running Grabber: /usr/share/mythtv/metadata/Television/ -l en -M The Daily Show with Jon Stewart 2011-05-28 17:48:18.511 Running Grabber: /usr/share/mythtv/metadata/Television/ -l en -D 71256 16 46 QProcess: Destroyed while process is still running. 2011-05-28 17:48:48.558 No results found for The Daily Show with Jon Stewart 16 46 After asking about this in the users mailing list Raymond Wagner answered with the following:

"My guess is that it's trying to pull the entire series and process it all at once, before returning the desired episode. Most shows only have a few dozen or maybe a hundred or two episodes allowing fairly quick processing. The Daily Show on the other hand is running into the thousands of episodes. I'll take a closer look at this later and see if it can be improved."

Seems Raymond is busy or forgot, so I am just filing a bug for this issue.

Change History (5)

comment:1 Changed 10 years ago by Raymond Wagner

Owner: set to Raymond Wagner
Status: newaccepted
Summary: Mythvideo ttvdb metadata lookup exits to grabber script takes too long to process shows with thousands of episodes

The metadata update does not time out too quickly. 30 seconds is a perfectly sane limit. The script should not be taking that long to run.

comment:2 Changed 10 years ago by Raymond Wagner

Owner: Raymond Wagner deleted
Status: acceptednew

When pulling data for The Daily Show, the script spends five seconds downloading the metadata, and another eighty seconds lost in lxml and some XSLT application. This is either going to require some serious optimization in the transform script, or a complete rewrite of the result generation code.

comment:3 Changed 10 years ago by Github

Milestone: unknown0.25
Resolution: fixed
Status: newclosed

ttvdb grabber performance improvements Change Log: 1) Removed complex element tree searches from the XSLT stylesheet and substituted XPATH lxml functions 2) Minor documentation corrections

Fixes #9859

Branch: master Changeset: 774f0d3ee833039f42f368685a5360b98f55e801

comment:4 Changed 10 years ago by Raymond Wagner


Backported to 0.24 in 8f8d6620bb3a6

comment:5 Changed 9 years ago by stuartm

Milestone: 0.24.2

Milestone 0.24.2 deleted

Note: See TracTickets for help on using tickets.