Modify
Warning Please read the Ticket HowTo before creating or commenting on a ticket. Failure to do so may cause your ticket to be rejected or result in a slower response.

Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#9859 closed Bug Report - General (fixed)

ttvdb.py grabber script takes too long to process shows with thousands of episodes

Reported by: espenaf@… Owned by:
Priority: minor Milestone:
Component: MythTV - General Version: 0.24-fixes
Severity: low Keywords:
Cc: Ticket locked: no

Description

The metadata lookup in mythvideo seems to have a 30 second timeout. When trying to lookup metadata for "The Daily Show with Jon Stewart" the timeout kills the process before it is finished fetching the data.

I don't have a slow connection, nor have I hammered the server.The metadata lookup has been tried at 5 to 10 different instances in the span of 2 weeks. All requests timeout for this tv show only. I might have been extremely unlucky with the timing of all my request tries, but I doubt it. Here's a frontend log of one of the tests:

2011-05-28 17:48:11.375 Using protocol version 63
2011-05-28 17:48:17.627 Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -M The Daily Show with Jon Stewart
2011-05-28 17:48:18.511 Running Grabber: /usr/share/mythtv/metadata/Television/ttvdb.py -l en -D 71256 16 46
QProcess: Destroyed while process is still running.
2011-05-28 17:48:48.558 No results found for The Daily Show with Jon Stewart 16 46

After asking about this in the users mailing list Raymond Wagner answered with the following:

"My guess is that it's trying to pull the entire series and process it all at once, before returning the desired episode. Most shows only have a few dozen or maybe a hundred or two episodes allowing fairly quick processing. The Daily Show on the other hand is running into the thousands of episodes. I'll take a closer look at this later and see if it can be improved."

Seems Raymond is busy or forgot, so I am just filing a bug for this issue.

Attachments (0)

Change History (5)

comment:1 Changed 3 years ago by wagnerrp

  • Owner set to wagnerrp
  • Status changed from new to accepted
  • Summary changed from Mythvideo ttvdb metadata lookup exits to early to ttvdb.py grabber script takes too long to process shows with thousands of episodes

The metadata update does not time out too quickly. 30 seconds is a perfectly sane limit. The script should not be taking that long to run.

comment:2 Changed 3 years ago by wagnerrp

  • Owner wagnerrp deleted
  • Status changed from accepted to new

When pulling data for The Daily Show, the script spends five seconds downloading the metadata, and another eighty seconds lost in lxml and some XSLT application. This is either going to require some serious optimization in the transform script, or a complete rewrite of the result generation code.

comment:3 Changed 3 years ago by Github

  • Milestone changed from unknown to 0.25
  • Resolution set to fixed
  • Status changed from new to closed

ttvdb grabber performance improvements
Change Log:
1) Removed complex element tree searches from the XSLT stylesheet and
substituted XPATH lxml functions
2) Minor documentation corrections

Fixes #9859

Branch: master
Changeset: 774f0d3ee833039f42f368685a5360b98f55e801

comment:4 Changed 3 years ago by wagnerrp

  • Milestone changed from 0.25 to 0.24.2

Backported to 0.24 in 8f8d6620bb3a6

comment:5 Changed 2 years ago by stuartm

  • Milestone 0.24.2 deleted

Milestone 0.24.2 deleted

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'new'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.