TV Metadata Lookup Fails

[Steven Côté <steven.cote@…>]

When attempting to retrieve metadata of television programs in MythVideo?, it will always time out.

I've dug into this problem and the culprit seems to be the baseurl used by the tvdb_api.py script.

The script sets the base url to "​http://www.thetvdb.com" on line 395. This should be "​http://thetvdb.com". This was fixed in the upstream version of this file (​https://github.com/dbr/tvdb_api).

According to the tvdb programmers api guide, this script should be looking up the base url from the mirrors api (​http://thetvdb.com/wiki/index.php?title=Programmers_API#Example_Usage). This only returns a single URL that doesn't seem to be geographically significant, so I guess there's not a lot of harm in hard coding it, but I thought I'd mention it any way.

[Raymond Wagner]

> dig www.thetvdb.com

; <<>> DiG 9.8.3-P4 <<>> www.thetvdb.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62781
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.thetvdb.com.               IN      A

;; ANSWER SECTION:
www.thetvdb.com.        300     IN      CNAME   thetvdb.com.

The www.thetvdb.com subdomain is nothing but a pointer to thtvdb.com, so which one we choose makes no difference. ttvdb.py works fine here. Could you give some more information on what's happening?

[Steven Côté <steven.cote@…>]

The forum discussion that the upstream patch referred to goes into a lot of detail. ​http://forums.thetvdb.com/viewtopic.php?f=4&t=12909

The tl;dr version is that depending where in the world you're making the query from, www.thetvdb.com isn't consistently translated into thetvdb.com. To answer why would require someone with superior knowledge of DNS than myself.

It seems to have something to do with the reliability of cloudflare (who are tvdb's dns providers?). Looking at the forums, it looks like there's a bunch of new posts in and around the beginning of this month which is when I started having problems.

For me personally (based in the UK), the tvdb script is timing out because it's unable to resolve the url with the "www" prefixed. I'll double check if it's the 503 http error mentioned in the forum that everyone else is reporting when I get home from work tonight.

[Steven Côté <steven.cote@…>]

Well that was weird. I tried running dig a few times from the same box I have my MythTV installation. On three different attempts, I got three different answers back!

I think there may be something going on with their servers at the moment, so I think I'll leave it a couple days and try again once things settle down.

[Raymond Wagner]

You should be seeing a list of five addresses. The addresses themselves will remain the same, but their ordering will be different each time. It's a simple form of load balancing.

[Steven Côté <steven.cote@…>]

It's all really quite strange. From the computers at work, I'm seeing the five addresses and the pointer. From home the output of dig is:

$ dig www.thetvdb.org

; <<>> DiG 9.8.1-P1 <<>> www.thetvdb.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64732
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;www.thetvdb.org.               IN      A

;; ANSWER SECTION:
www.thetvdb.org.        120     IN      A       54.235.159.97

The actual result of this is that when I try to run ttvdb.py, it times out.

$ /usr/share/mythtv/metadata/Television/ttvdb.py -d -l en -a GB -M "The Walking Dead"
opts {'usage': False, 'configure': '', 'screenshot': False, 'num_seasons': False, 'language': 'en', 'poster': False, 'mythvideo': False, 'list': True, 'collection': False, 'fanart': False, 'banner': False, 'raw': False, 'version': False, 'numbers': False, 'country': 'GB', 'test': False, 'data': False, 'debug': True, 'interactive': False}

args [u'The Walking Dead']
[u'The Walking Dead']
####################
# series_season_ep array( [u'The Walking Dead'] )
####################
# Starting tvtvb
# Processing (The Walking Dead) Series
# ..got tvdb mirrors
# Start to process series or series_season_ep
####################
2013-10-19 07:24:59,830) DEBUG Searching for show The%20Walking%20Dead
2013-10-19 07:24:59,831) DEBUG Retrieving URL http://www.thetvdb.com/api/GetSeries.php?seriesname=The%20Walking%20Dead&language=en
! Error: Could not connect to server: <urlopen error [Errno 101] Network is unreachable>

Trying that URL in the browser, it times out, but dropping the www prefix returns the expected XML.

[Steven Côté <steven.cote@…>]

Not a lot of movement here. For the time being I'm working around my specific issue by using the Google public DNS which is giving me much more consistent results than my own ISP.

However, the actual bug was to raise the issue that the tvdb_api.py script has fallen out of sync with upstream, which is still true.

[jasongillis]

It looks like this issue was probably fixed in Aug 2017 with the change to the JSON API (commit e60c0283eda6f2e094a0b002eaea72f0ba06455b).

Also, this ticket was probably logged against code using the XML API which isn't supported by TVDB anymore.

[Stuart Auchterlonie]

Closing out old tickets.