Opened 4 years ago

Closed 4 years ago

Last modified 2 years ago

#12573 closed Patch - Bug Fix (fixed)

Update to Mythnetvision twit.tv script by Auric

Reported by: William L. DeRieux IV <williamderieux@…> Owned by: Karl Egly
Priority: minor Milestone: 0.27.6
Component: Plugin - MythNetvision Version: Unspecified
Severity: medium Keywords:
Cc: Ticket locked: no

Description

The twit.tv site had undergone some significant changes to its layout and as a result the current twit.tv.pl script fails to find any videos.

The included update will scrape there rss feed page: http://wiki.twit.tv/wiki/TWiT_Show_Feeds

It will only download videos for currenty active shows.

For each rss feed you have a choice of the following: Video-LO / Video-Hi / Video-HD.

The script is hard-coded for Video-Hi, but it could be refined by changing: my $video_size = $video_large; to either: my $video_size = $video_small my $video_size = $video_hd

This update provides the following features: 1) Videos can be successfully added to the UI 2) Low impact on the twit.tv server 3) Extremely fast:

  • We only make at most 1 call to generate links to the rss feeds.
  • We only make 1 call for each RSS feed (currently that is 25 rss feeds)
  • the original script would make at least 600 calls or more per show

I will try to include the updated script with this ticket (the attached version will be a cleaned up version of the one I am linking to) but if that does not work you can review the changes here: https://github.com/neoh4x0r/mythtv/commit/244514f7fe65d2e824a46daf9568bc72d94a3d32

Attachments (1)

twit.tv.pl (12.0 KB) - added by William L. DeRieux IV <williamderieux@…> 4 years ago.
Updated version of twit.tv.pl script for mythnetvision (clean version)

Download all attachments as: .zip

Change History (6)

Changed 4 years ago by William L. DeRieux IV <williamderieux@…>

Attachment: twit.tv.pl added

Updated version of twit.tv.pl script for mythnetvision (clean version)

comment:1 Changed 4 years ago by William L. DeRieux IV <william@…>

Owner: set to William L. DeRieux IV <william@…>
Resolution: fixed
Status: newclosed

In 56769ba842219e9c689b665ae7b3c6c74201dc01/mythtv:

update twit.tv.pl script to latest version

-- the twit.tv site has undergone some recent changes
Now the script scrapes the wiki's rss feeds instead of twit.tv directly

New dependency XML::TreeBuilder?

Fixes #12573
(cherry picked from commit 244514f7fe65d2e824a46daf9568bc72d94a3d32)

comment:2 Changed 4 years ago by Karl Egly

Milestone: unknown0.27.6
Priority: criticalminor
Severity: highmedium
Type: Patch - FeaturePatch - Bug Fix

Committed after really light testing.

The returned XML has some room for improvement though.

<item>
  <title>2015/08/03 [EP#211] - Mark Johnson is the CEO and co-founder at Descartes Labs. He has a track record of translating complex technologies into usable and successful products. Recently he was the CEO of Zite which he sold twice. First time was to CNN and the second time was to Flipboard. Mark was also a product manager who trained at SAP followed by a string of successful search startups. Download or subscribe to this show at twit.tv/tri. Thanks to Cachefly for the bandwidth for this show.</title>
  <mythtv:subtitle></mythtv:subtitle>
  <author>leo@twit.tv (Leo Laporte)</author>
  <pubDate>2015/08/03</pubDate>
  <description>&lt;p&gt;&lt;img src=&quot;http://twit.cachefly.net/coverart/tri/tri144videohi.jpg&quot; align=&quot;right&quot; hspace=&quot;20&quot; vspace=&quot;20&quot; border=&quot;0&quot; title=&quot;Triangulation&quot; alt=&quot;Triangulation&quot;/&gt;&lt;/p&gt;&lt;p&gt;Mark Johnson is the CEO and co-founder at Descartes Labs. He has a track record of translating complex technologies into usable and successful products. Recently he was the CEO of Zite which he sold twice. First time was to CNN and the second time was to Flipboard. Mark was also a product manager who trained at SAP followed by a string of successful search startups.&lt;/p&gt; &lt;p&gt;Download or subscribe to this show at &lt;a href=&quot;http://twit.tv/tri&quot;&gt;twit.tv/tri&lt;/a&gt;.&lt;/p&gt; &lt;p&gt;Thanks to &lt;a href=&quot;http://cachefly.com/&quot; target=&quot;_blank&quot;&gt;Cachefly&lt;/a&gt; for the bandwidth for this show.&lt;/p&gt;</description>
  <link>http://twit.tv/tri/211</link>
  <player></player>
  <playerargs></playerargs>
  <download></download>
  <downloadargs></downloadargs>
  <media:group>
    <media:thumbnail url="http://twit.cachefly.net/coverart/tri/tri144videohi.jpg"/>
    <media:content url="http://www.podtrac.com/pts/redirect.mp4/twit.cachefly.net/video/tri/tri0211/tri0211_h264m_864x480_500.mp4" length="" duration="1:04:05" width="864" height="480" lang="eng"/>
  </media:group>
  <rating></rating>
  <mythtv:country>usa</mythtv:country>
  <mythtv:season></mythtv:season>
  <mythtv:episode>211</mythtv:episode>
  <mythtv:customhtml>no</mythtv:customhtml>
</item>

Resetting some values to defaults while here.

comment:3 Changed 4 years ago by William L. DeRieux IV <william@…>

In a62fc0da9e9d8152a5b83cb6e8898faf67d6cbab/mythtv:

update twit.tv.pl script to latest version

-- the twit.tv site has undergone some recent changes
Now the script scrapes the wiki's rss feeds instead of twit.tv directly

New dependency XML::TreeBuilder?

Fixes #12573
(cherry picked from commit 244514f7fe65d2e824a46daf9568bc72d94a3d32)

(cherry picked from commit 56769ba842219e9c689b665ae7b3c6c74201dc01)

comment:4 Changed 4 years ago by Karl Egly

Owner: changed from William L. DeRieux IV <william@…> to Karl Egly

comment:5 Changed 2 years ago by churufrijo@…

Instead of scraping a wiki page, you should take the list of feeds from TWiT's feeds OPML file: http://feeds.twit.tv/twitfeeds.opml

Much more reliable!

Note: See TracTickets for help on using tickets.