Opened 13 years ago
Closed 13 years ago
Last modified 12 years ago
#9859 closed Bug Report - General (fixed)
ttvdb.py grabber script takes too long to process shows with thousands of episodes
Reported by: | Owned by: | ||
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | MythTV - General | Version: | 0.24-fixes |
Severity: | low | Keywords: | |
Cc: | Ticket locked: | no |
Description
The metadata lookup in mythvideo seems to have a 30 second timeout. When trying to lookup metadata for "The Daily Show with Jon Stewart" the timeout kills the process before it is finished fetching the data.
I don't have a slow connection, nor have I hammered the server.The metadata lookup has been tried at 5 to 10 different instances in the span of 2 weeks. All requests timeout for this tv show only. I might have been extremely unlucky with the timing of all my request tries, but I doubt it. Here's a frontend log of one of the tests:
2011-05-28 17:48:11.375 Using protocol version 63 2011-05-28 17:48:17.627 Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -M The Daily Show with Jon Stewart 2011-05-28 17:48:18.511 Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -D 71256 16 46 QProcess: Destroyed while process is still running. 2011-05-28 17:48:48.558 No results found for The Daily Show with Jon Stewart 16 46 After asking about this in the users mailing list Raymond Wagner answered with the following:
"My guess is that it's trying to pull the entire series and process it all at once, before returning the desired episode. Most shows only have a few dozen or maybe a hundred or two episodes allowing fairly quick processing. The Daily Show on the other hand is running into the thousands of episodes. I'll take a closer look at this later and see if it can be improved."
Seems Raymond is busy or forgot, so I am just filing a bug for this issue.
Change History (5)
comment:1 Changed 13 years ago by
Owner: | set to Raymond Wagner |
---|---|
Status: | new → accepted |
Summary: | Mythvideo ttvdb metadata lookup exits to early → ttvdb.py grabber script takes too long to process shows with thousands of episodes |
comment:2 Changed 13 years ago by
Owner: | Raymond Wagner deleted |
---|---|
Status: | accepted → new |
When pulling data for The Daily Show, the script spends five seconds downloading the metadata, and another eighty seconds lost in lxml and some XSLT application. This is either going to require some serious optimization in the transform script, or a complete rewrite of the result generation code.
comment:3 Changed 13 years ago by
Milestone: | unknown → 0.25 |
---|---|
Resolution: | → fixed |
Status: | new → closed |
ttvdb grabber performance improvements Change Log: 1) Removed complex element tree searches from the XSLT stylesheet and substituted XPATH lxml functions 2) Minor documentation corrections
Fixes #9859
Branch: master Changeset: 774f0d3ee833039f42f368685a5360b98f55e801
The metadata update does not time out too quickly. 30 seconds is a perfectly sane limit. The script should not be taking that long to run.