Opened 12 years ago

Closed 12 years ago

Last modified 12 years ago

#10662 closed Bug Report - General (Upstream Bug)

mythfilldatabase download error

Reported by: Greg Woods <greg@…> Owned by: beirdo
Priority: minor Milestone: unknown
Component: MythTV - Mythfilldatabase Version: 0.25-fixes
Severity: medium Keywords:
Cc: Ticket locked: no

Description

This is for the recent "schedulesdirect not working" threads on the mailing list. The gist of it is that I get no guide data because mythfilldatabase errors out with a download error. Things work fine if I use xmltv utilities to grab the data from Schedules Direct, then manually load it with "mythfilldatabase --file". Several others on the list are reporting the same issue, although it does seem to be working for most people. I filed a Schedules Direct ticket, and Robert Kulagowski investigated and determined there is nothing wrong with my SD account or with my ability to get the data from Tribune, and he suggested I file a MythTV ticket, so here it is.

I will attach a log of a mythfilldatabase run showing the error.

This started immediately after my upgrade to 0.25, things were working fine before that.

Attachments (2)

downloadmanager.diff (2.1 KB) - added by beirdo 12 years ago.
datadirect-2.diff (1.2 KB) - added by beirdo 12 years ago.

Download all attachments as: .zip

Change History (30)

comment:1 Changed 12 years ago by Greg Woods <greg@…>

My attempt to attach the log failed, it was rejected by a spam filter. The log can be viewed at:

http://www.gregandeva.net/mythfilldatabase.20120424133425.30422.log.txt

comment:2 Changed 12 years ago by beirdo

Owner: changed from stuartm to beirdo
Status: newassigned

comment:3 Changed 12 years ago by beirdo

Status: assignedinfoneeded

Please try the attached patch, and let me know if that solves it. If it doesn't, please generate another log file with it in place. Thanks.

Changed 12 years ago by beirdo

Attachment: downloadmanager.diff added

comment:4 Changed 12 years ago by Greg Woods <greg@…>

Sorry to say that the patch did not fix it for me, I still get a download error. The new log is at:

http://www.gregandeva.net/mythfilldatabase.20120427152000.23018.log.txt

comment:5 Changed 12 years ago by beirdo

Please try again with the same logging as your first run (-v file,network --loglevel debug) The log messages I put in are in file/debug.

comment:6 Changed 12 years ago by Greg Woods <greg@…>

comment:7 Changed 12 years ago by beirdo

Ahh. Thanks. I see. OK, in the code that was patched, change the 60 to something much higher (say give it 600 for 10min) and see if it still times out.

I have noticed in the past that the way DataDirect? transfers happen, there is a small burst of data up front, and then you wait for the server to compile all the listings for you, and then it continues the transfer. Seems that in your case, that delay is > 1min. I think once we figure out what the timeout would have to be, we might want to be pushing back on TMS through Schedules Direct to see if that long of a delay is "normal" or not.

comment:8 Changed 12 years ago by beirdo

That change would be on line 800 of mythdownloadmanager.cpp.

comment:9 Changed 12 years ago by beirdo

Sorry, and if it works or not, please post a log with -v file,network --loglevel debug

Thanks,

comment:10 Changed 12 years ago by Greg Woods <greg@…>

I changed the 60 (as in the patch) to 600, and it still fails after a 10 minute wait:

http://www.gregandeva.net/mythfilldatabase.20120428125438.15182.log.txt

I'm not certain exactly what all it is waiting for during that time. I ran "time tv_grab_na_dd" on my gateway system. That entire operation takes about 11 minutes, but the download part takes just 170 seconds. If I didn't have an important recording scheduled to start in 15 minutes, I'd try it again with a 20 minute timeout. I'll do that later tonight. I can't install new packages on my backend because it's running Fedora 14 which is no longer supported, so I had to install xmltv elsewhere and copy the file over to do a manual guide update. The gateway system is a slower 32-bit system so I would expect the mythfilldatabase processing to be faster on the actual backend, so I don't expect going to 20 minutes to make a difference, but I'll try it later anyway just to be sure.

comment:11 Changed 12 years ago by beirdo

Hmmm, we may need to look at packet dumps from tcpdump to determine what's going on. In particular, as root:

tcpdump -s0 -i any -w /tmp/capture.pcap host 144.142.232.53

Capture the whole failing attempt, and then I'd need to peruse it. Don't worry, your username/password is sent in htdigest form, and is not visible in cleartext.

comment:12 Changed 12 years ago by Greg Woods <greg@…>

Log:

mythfilldatabase.20120429075139.18419.log

Packet capture:

capture.pcap

Both at www.gregandeva.net (can't include full links or the &*#(*@(!@ spam filter rejects me)

comment:13 Changed 12 years ago by beirdo

Thank you. That is very useful. It seems that somehow, it is not hitting the authentication callback, or the callback isn't working. This is odd. Please try adding this new patch, we should only need a logfile this time :)

Thanks for your patience, BTW.

Changed 12 years ago by beirdo

Attachment: datadirect-2.diff added

comment:14 Changed 12 years ago by beirdo

This new patch just adds more logging around the callback to see if it is actually hitting.

comment:15 Changed 12 years ago by Greg Woods <greg@…>

Log with datadirect-2 patch is at www.gregandeva.net/mythfilldatabase.20120429120905.15752.log.txt

comment:16 Changed 12 years ago by beirdo

What version of Qt are you running?

Also, please verify that your schedule direct username and password are set correctly (don't know why it would have changed in an upgrade, but please verify).

comment:17 Changed 12 years ago by beirdo

From your pcap:

Authorization: Digest username="greg", realm="gregandeva.net", nonce="97c2cbdfe2b03f1807970e2cd4ba0696", uri="/schedulesdirect/tvlistings/xtvdService", opaque="2b99186424025b388825c77349e75a2b", response="84f7bc1b9b7f82bc69bb75661dbe0582", qop=auth, nc=00000001, cnonce="ab193f6bb95e3b46bb34453fe339e249"

OK. That is not right. the realm is being changed. I'll assume your username for SchedulesDirect? is in fact "greg", and not "Greg"?

comment:18 Changed 12 years ago by Greg Woods <greg@…>

I am running qt-4.7.1-7.fc14.x86_64

I have verified (by looking in the videosource table) that my SchedulesDirect? userid and password are set correctly (which isn't a surprise given previous discussion in the thread; at least one other person experiencing this problem already tried resetting and verifying his login info several times).

My SD username is actually "greg@…".

comment:19 Changed 12 years ago by Greg Woods <greg@…>

greg@… (don't know why that got cut off...)

comment:20 Changed 12 years ago by Greg Woods <greg@…>

It's my e-mail address.

comment:21 Changed 12 years ago by beirdo

Ooooooh. I think I'm understanding the issue now ;) Now how to fix it....

It seems when you set the user to a user in the form of an email... Qt takes the host part as being the realm, which is stupid. I'll have to go read their code to see how to get around this mess.

comment:22 Changed 12 years ago by beirdo

https://bugreports.qt-project.org/browse/QTBUG-15070

This was apparently fixed in Qt4.7.2

comment:23 Changed 12 years ago by beirdo

In fact, by the report it is ONLY a problem in Qt4.7.1

comment:24 Changed 12 years ago by danielk

Can we just set the realm after setting the user as a workaround for Qt 4.7.1?

comment:25 Changed 12 years ago by beirdo

No. There is not "setRealm" and the setUser() strips off everything from the first @ on. It's a bug in Qt that has been fixed. The way to work around it is to downgrade to Qt 4.7.0 or upgrade to 4.7.2 or 4.8.

Or, you can go edit the code for Qt itself and recompile, but that takes a very very long time.

comment:26 Changed 12 years ago by Gavin Hurlbut <ghurlbut@…>

In e778f23d0eeabf99be23121cebc190191aa7041f/mythtv:

Add more debug logs in the DataDirect? and MythDLMgr code

Refs #10662

Turns out the problem we are seeing is a known bug in Qt 4.7.1 that is fixed
in 4.7.2 (and wasn't there before 4.7.1).

In particular, if your Schedules Direct username is an email address, Qt4.7.1
stupidly puts the username as the part before the @ and the realm as the
portion after the @, even though the realm was already provided.

SO, if you have a Schedules Direct username that is an email address, either
upgrade to 4.7.2 (or higher), or downgrade to 4.7.0 (or lower).

comment:27 Changed 12 years ago by danielk

Resolution: Upstream Bug
Status: infoneededclosed

As discussed on #mythtv there is a workaround that doesn't involve patching Qt.

Go to https://www.schedulesdirect.org/account/edit Change the login name so that it does not have an "@" sign in it. Run mythtv-setup and change the Schedules Direct login name to match the new login name.

As such I'm closing this as an upstream bug.

comment:28 Changed 12 years ago by Gavin Hurlbut <ghurlbut@…>

In 9e59131ec6d352a6fdafc810b004765ba43588a7/mythtv:

Add more debug logs in the DataDirect? and MythDLMgr code

Refs #10662

Turns out the problem we are seeing is a known bug in Qt 4.7.1 that is fixed
in 4.7.2 (and wasn't there before 4.7.1).

In particular, if your Schedules Direct username is an email address, Qt4.7.1
stupidly puts the username as the part before the @ and the realm as the
portion after the @, even though the realm was already provided.

SO, if you have a Schedules Direct username that is an email address, either
upgrade to 4.7.2 (or higher), or downgrade to 4.7.0 (or lower).
(cherry picked from commit e778f23d0eeabf99be23121cebc190191aa7041f)

Note: See TracTickets for help on using tickets.