Ticket #9859 (closed Bug Report - General: fixed)
Opened 2 years ago
Last modified 15 months ago
ttvdb.py grabber script takes too long to process shows with thousands of episodes
| Reported by: | espenaf@… | Owned by: | |
|---|---|---|---|
| Priority: | minor | Milestone: | |
| Component: | MythTV - General | Version: | 0.24-fixes |
| Severity: | low | Keywords: | |
| Cc: | Ticket locked: | no |
Description
The metadata lookup in mythvideo seems to have a 30 second timeout. When trying to lookup metadata for "The Daily Show with Jon Stewart" the timeout kills the process before it is finished fetching the data.
I don't have a slow connection, nor have I hammered the server.The metadata lookup has been tried at 5 to 10 different instances in the span of 2 weeks. All requests timeout for this tv show only. I might have been extremely unlucky with the timing of all my request tries, but I doubt it. Here's a frontend log of one of the tests:
2011-05-28 17:48:11.375 Using protocol version 63 2011-05-28 17:48:17.627 Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -M The Daily Show with Jon Stewart 2011-05-28 17:48:18.511 Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -D 71256 16 46 QProcess: Destroyed while process is still running. 2011-05-28 17:48:48.558 No results found for The Daily Show with Jon Stewart 16 46 After asking about this in the users mailing list Raymond Wagner answered with the following:
"My guess is that it's trying to pull the entire series and process it all at once, before returning the desired episode. Most shows only have a few dozen or maybe a hundred or two episodes allowing fairly quick processing. The Daily Show on the other hand is running into the thousands of episodes. I'll take a closer look at this later and see if it can be improved."
Seems Raymond is busy or forgot, so I am just filing a bug for this issue.
Attachments
Change History
comment:1 Changed 2 years ago by wagnerrp
- Owner set to wagnerrp
- Status changed from new to accepted
- Summary changed from Mythvideo ttvdb metadata lookup exits to early to ttvdb.py grabber script takes too long to process shows with thousands of episodes
comment:2 Changed 23 months ago by wagnerrp
- Owner wagnerrp deleted
- Status changed from accepted to new
When pulling data for The Daily Show, the script spends five seconds downloading the metadata, and another eighty seconds lost in lxml and some XSLT application. This is either going to require some serious optimization in the transform script, or a complete rewrite of the result generation code.
comment:3 Changed 23 months ago by Github
- Status changed from new to closed
- Resolution set to fixed
- Milestone changed from unknown to 0.25
ttvdb grabber performance improvements Change Log: 1) Removed complex element tree searches from the XSLT stylesheet and substituted XPATH lxml functions 2) Minor documentation corrections
Fixes #9859
Branch: master Changeset: 774f0d3ee833039f42f368685a5360b98f55e801
comment:4 Changed 23 months ago by wagnerrp
- Milestone changed from 0.25 to 0.24.2
Backported to 0.24 in 8f8d6620bb3a6
The metadata update does not time out too quickly. 30 seconds is a perfectly sane limit. The script should not be taking that long to run.