mythfilldatabase download error

[Greg Woods <greg@…>]

This is for the recent "schedulesdirect not working" threads on the mailing list. The gist of it is that I get no guide data because mythfilldatabase errors out with a download error. Things work fine if I use xmltv utilities to grab the data from Schedules Direct, then manually load it with "mythfilldatabase --file". Several others on the list are reporting the same issue, although it does seem to be working for most people. I filed a Schedules Direct ticket, and Robert Kulagowski investigated and determined there is nothing wrong with my SD account or with my ability to get the data from Tribune, and he suggested I file a MythTV ticket, so here it is.

I will attach a log of a mythfilldatabase run showing the error.

This started immediately after my upgrade to 0.25, things were working fine before that.

[Greg Woods <greg@…>]

My attempt to attach the log failed, it was rejected by a spam filter. The log can be viewed at:

​http://www.gregandeva.net/mythfilldatabase.20120424133425.30422.log.txt

[beirdo]

Please try the attached patch, and let me know if that solves it. If it doesn't, please generate another log file with it in place. Thanks.

[Greg Woods <greg@…>]

Sorry to say that the patch did not fix it for me, I still get a download error. The new log is at:

​http://www.gregandeva.net/mythfilldatabase.20120427152000.23018.log.txt

[beirdo]

Please try again with the same logging as your first run (-v file,network --loglevel debug) The log messages I put in are in file/debug.

[Greg Woods <greg@…>]

The new log is at:

​http://www.gregandeva.net/mythfilldatabase.20120428095501.25294.log.txt

[beirdo]

Ahh. Thanks. I see. OK, in the code that was patched, change the 60 to something much higher (say give it 600 for 10min) and see if it still times out.

I have noticed in the past that the way DataDirect? transfers happen, there is a small burst of data up front, and then you wait for the server to compile all the listings for you, and then it continues the transfer. Seems that in your case, that delay is > 1min. I think once we figure out what the timeout would have to be, we might want to be pushing back on TMS through Schedules Direct to see if that long of a delay is "normal" or not.

[beirdo]

That change would be on line 800 of mythdownloadmanager.cpp.

[beirdo]

Sorry, and if it works or not, please post a log with -v file,network --loglevel debug

Thanks,

[Greg Woods <greg@…>]

I changed the 60 (as in the patch) to 600, and it still fails after a 10 minute wait:

​http://www.gregandeva.net/mythfilldatabase.20120428125438.15182.log.txt

I'm not certain exactly what all it is waiting for during that time. I ran "time tv_grab_na_dd" on my gateway system. That entire operation takes about 11 minutes, but the download part takes just 170 seconds. If I didn't have an important recording scheduled to start in 15 minutes, I'd try it again with a 20 minute timeout. I'll do that later tonight. I can't install new packages on my backend because it's running Fedora 14 which is no longer supported, so I had to install xmltv elsewhere and copy the file over to do a manual guide update. The gateway system is a slower 32-bit system so I would expect the mythfilldatabase processing to be faster on the actual backend, so I don't expect going to 20 minutes to make a difference, but I'll try it later anyway just to be sure.

[beirdo]

Hmmm, we may need to look at packet dumps from tcpdump to determine what's going on. In particular, as root:

tcpdump -s0 -i any -w /tmp/capture.pcap host 144.142.232.53

Capture the whole failing attempt, and then I'd need to peruse it. Don't worry, your username/password is sent in htdigest form, and is not visible in cleartext.

[Greg Woods <greg@…>]

Log:

mythfilldatabase.20120429075139.18419.log

Packet capture:

capture.pcap

Both at www.gregandeva.net (can't include full links or the &*#(*@(!@ spam filter rejects me)

[beirdo]

Thank you. That is very useful. It seems that somehow, it is not hitting the authentication callback, or the callback isn't working. This is odd. Please try adding this new patch, we should only need a logfile this time :)

Thanks for your patience, BTW.

[beirdo]

This new patch just adds more logging around the callback to see if it is actually hitting.

[Greg Woods <greg@…>]

Log with datadirect-2 patch is at www.gregandeva.net/mythfilldatabase.20120429120905.15752.log.txt

[beirdo]

What version of Qt are you running?

Also, please verify that your schedule direct username and password are set correctly (don't know why it would have changed in an upgrade, but please verify).

[beirdo]

From your pcap:

Authorization: Digest username="greg", realm="gregandeva.net", nonce="97c2cbdfe2b03f1807970e2cd4ba0696", uri="/schedulesdirect/tvlistings/xtvdService", opaque="2b99186424025b388825c77349e75a2b", response="84f7bc1b9b7f82bc69bb75661dbe0582", qop=auth, nc=00000001, cnonce="ab193f6bb95e3b46bb34453fe339e249"

OK. That is not right. the realm is being changed. I'll assume your username for SchedulesDirect? is in fact "greg", and not "Greg"?

[Greg Woods <greg@…>]

I am running qt-4.7.1-7.fc14.x86_64

I have verified (by looking in the videosource table) that my SchedulesDirect? userid and password are set correctly (which isn't a surprise given previous discussion in the thread; at least one other person experiencing this problem already tried resetting and verifying his login info several times).

My SD username is actually "greg@…".

[Greg Woods <greg@…>]

greg@… (don't know why that got cut off...)

[Greg Woods <greg@…>]

It's my e-mail address.

[beirdo]

Ooooooh. I think I'm understanding the issue now ;) Now how to fix it....

It seems when you set the user to a user in the form of an email... Qt takes the host part as being the realm, which is stupid. I'll have to go read their code to see how to get around this mess.

[beirdo]

​https://bugreports.qt-project.org/browse/QTBUG-15070

This was apparently fixed in Qt4.7.2

[beirdo]

In fact, by the report it is ONLY a problem in Qt4.7.1

[danielk]

Can we just set the realm after setting the user as a workaround for Qt 4.7.1?

[beirdo]

No. There is not "setRealm" and the setUser() strips off everything from the first @ on. It's a bug in Qt that has been fixed. The way to work around it is to downgrade to Qt 4.7.0 or upgrade to 4.7.2 or 4.8.

Or, you can go edit the code for Qt itself and recompile, but that takes a very very long time.

[Gavin Hurlbut <ghurlbut@…>]

In e778f23d0eeabf99be23121cebc190191aa7041f/mythtv:

Add more debug logs in the DataDirect? and MythDLMgr code

Refs #10662

Turns out the problem we are seeing is a known bug in Qt 4.7.1 that is fixed
in 4.7.2 (and wasn't there before 4.7.1).

In particular, if your Schedules Direct username is an email address, Qt4.7.1
stupidly puts the username as the part before the @ and the realm as the
portion after the @, even though the realm was already provided.

SO, if you have a Schedules Direct username that is an email address, either
upgrade to 4.7.2 (or higher), or downgrade to 4.7.0 (or lower).

[danielk]

As discussed on #mythtv there is a workaround that doesn't involve patching Qt.

Go to ​https://www.schedulesdirect.org/account/edit Change the login name so that it does not have an "@" sign in it. Run mythtv-setup and change the Schedules Direct login name to match the new login name.

As such I'm closing this as an upstream bug.

[Gavin Hurlbut <ghurlbut@…>]

In 9e59131ec6d352a6fdafc810b004765ba43588a7/mythtv:

Add more debug logs in the DataDirect? and MythDLMgr code

Refs #10662

Turns out the problem we are seeing is a known bug in Qt 4.7.1 that is fixed
in 4.7.2 (and wasn't there before 4.7.1).

In particular, if your Schedules Direct username is an email address, Qt4.7.1
stupidly puts the username as the part before the @ and the realm as the
portion after the @, even though the realm was already provided.

SO, if you have a Schedules Direct username that is an email address, either
upgrade to 4.7.2 (or higher), or downgrade to 4.7.0 (or lower).
(cherry picked from commit e778f23d0eeabf99be23121cebc190191aa7041f)