Modify

Opened 23 months ago

Last modified 3 months ago

#11739 assigned Bug Report - General

EIT over the air program guide title description mismatch FIXED/SOLVED

Reported by: deadletterfile@… Owned by: dekarl
Priority: minor Milestone: 0.28
Component: MythTV - ATSC Version: Master Head
Severity: medium Keywords: EIT title description
Cc: stuarta Ticket locked: no

Description

Summary: Eliminating two lines of code from EITHelper::ProcessEvents (void) eliminates 99% of Mythtv generated title/description mismatches in the over the air program guide

  • eitList_lock.unlock();
  • eitList_lock.lock();

I believe the use of QMutexLocker in function uint EITHelper::ProcessEvents (void) as currently implemented is incorrect, allowing a threaded CPU to run concurrent processing of events (individual program listings). For a given channel, this allows descriptions from other programs to be associated with the wrong program title.

libs/libmythtv/programdata.cpp For the same program listing as one previously processed, tests the length of new program description against the current stored description, replacing old with new if new is longer. I had a case where a long incorrect description was not replaced by a subsequent, shorter correct description. Possible fix in the patch.

The patch is against the master branch: commit 619d87bdc8a (Aug 8, 2013)

My fix for ticket 11476 is also in the patch as it stabilized use of the EIT scanner for me.

sha1sum /data/wrkbin/computer/libmythtv.patch 922fb3eedc55c65de7b97328a7c5215d37b13060

Anyone testing the patch should be vigilant in examination before applying. Code changes (ignoring logging) are few enough to be made manually. Place the patch in the mythtv/libs/libmythtv directory and apply:

patch -Np1 < libmythtv.patch

With this patch installed, my EIT scanner does not unnecessarily abort. Event/program information taken from the broadcast data stream correctly populates the Mythtv program guide. Titles and descriptions match. I am very happy with the results and (until proven otherwise) consider this issue fixed/solved/closed. I have even installed custom recording rules based on description for the first time. Note that I do not believe these .cpp files have been altered in some time and a clean patch is perhaps likely on older 'master' branches or even 0.26 and 0.26/fixes.

Attachments (3)

libmythtv.patch (13.4 KB) - added by Ronald L Humble (Royboy626) <deadletterfile@…> 23 months ago.
patch for <tv_rec programdata eitscanner eithelper>.cpp
libmythtv.140128.patch (15.5 KB) - added by deadletterfile@… 17 months ago.
patch to stabilize EIT over-the-air broachast schedule
libmythtv.140430.patch (15.1 KB) - added by deadletterfile@… 14 months ago.
astc EIT fixes #11476 #11739; off commit befdc4d4b9

Download all attachments as: .zip

Change History (29)

Changed 23 months ago by Ronald L Humble (Royboy626) <deadletterfile@…>

patch for <tv_rec programdata eitscanner eithelper>.cpp

comment:1 Changed 23 months ago by deadletterfile@…

I am an idiot and a fool. Alas, the Internet is forever.

I still hope the programmers will look at the QMutexLocker implementation in ProcessEvents as I believe this change has merit.

However my original coding removed:

if (match.description.length() >= ldesc.length())

ldesc = match.description;

from processdata.cpp. This worked well until a block of channel listing lost their program descriptions. The patch solution was no better and I have returned the above code to service. It appears a bad program description once broadcast by a channel, but later corrected, will likely not be correct in the Mythtv EIT program listings.

I changed complex code in programdata.cpp before understanding it. I feel like further improvement in title/description mismatches can be made in programdata.cpp, and will make the attempt in the coming months. I still think I can contribute, but will be more careful and measured in my response.

I apologize to the Mythtv community and especially the programmers.

comment:2 Changed 18 months ago by stuarta

  • Owner set to stuarta
  • Status changed from new to assigned

I'm guessing from the previous comment that this ticket can be closed?

comment:3 Changed 18 months ago by deadletterfile@…

Thank you for you response. The problem persists. I was simply stupid and hasty in working with the code. I hope someone might still address the program/description mismatches that occur with EIT data in Mythtv. I have found the atsc_epg program from linuxtv.org dvb-apps useful. I am still working with it but it seems to provide superior results to Mythtv. After several months, I am finally looking at this issue again. Obviously it is your call as to closing the ticket, but program/description mismatch using broadcast EIT data it still a problem for me with the latest stable version of Mythtv.

I do not think #11476 has been addressed either. My changes have completely stabilized the display of EIT program data for me. Let me know if I can provide any assistance on either of these issues. -- Royboy626

comment:4 Changed 18 months ago by stuarta

Hi,

Not planning to close it, I was trying to better understand the problem in order to verify that your fixes are correct. I need to understand the signalmonitor changes better and try to work out why they have an influence on your issue.

comment:5 Changed 18 months ago by stuarta

  • Milestone changed from unknown to 0.28

comment:6 Changed 18 months ago by deadletterfile@…

I apologize for my patch being a unified patch of my Mythtv changes regarding bugs 11739 and 11476. It confused things for the Mythtv programmers trying to address these potential issues.

I have two avenues to which you might address your attention, in addition to any you investigate.

1) I reverted back to eliminating the three lines of code mentioned in comment #1. I am not sure of the programmer's original intent, but have been pleased with the result of omitting these lines in eliminating mismatches in Mythtv.

2) At the same time, I changed set up in mythtv-setup to only use one tuner card for EIT data collection. I believe multiple cards collecting EIT data can cause program/description mismatches.

I still strongly believe the implementation of the QMutexLocker is flawed in eithelper.cpp -> EITHelper::ProcessEvents. I eliminated these two lines:

eitList_lock.unlock(); eitList_lock.lock();

from the function. It appears to me use of the above lines would apply to QMutex code, and should not be used. If I am correct, perhaps this bug exists elsewhere in the code. Removal of these lines seems to solve 2) for me.

http://qt-project.org/doc/qt-4.8/qmutexlocker.html

Details...

I made two changes after posting comment #1 and have been running on those changes since then and am very pleased with the result. 1) I went back to commenting out the three lines of code in programdata.cpp DBEvent::UpdateDB as noted below:

if (match.subtitle.length() >= lsubtitle.length())

lsubtitle = match.subtitle;

// if (match.description.length() >= ldesc.length()) // if (match.description.length() >= 2) // ldesc = match.description;

if (lcategory.isEmpty() && !match.category.isEmpty())

lcategory = match.category;

2) I have 22 stations on about 15 transponders. I have been using only one tuner card to collect EIT data for months. Over 48 hours, in casually attempting to find a program/description mismatch, I could find only one. Last night I set all tuner cards to collect EIT data. This morning, program/description mismatches were common, but only on one transponder (with three stations). This was decidedly NOT the behavior months ago, it was over all transponders. However this was a short test. I returned to using only one card for EIT collection, waited an hour for EIT data refresh, and all program/description mismatches resolved themselves. I have never had mismatched descriptions cross channels or transponders. If a program had a mismatched description, it was always from another program on that channel.

After making this test, I discovered the two lines from 2) had crept back into my code. I recompiled and retested with multiple cards gathering EIT data. It has been two hours and no program/mismatches have been found. I admit this is a short test but hints at 2) being an issue if multiple cards are gathering EIT data. Thank you for your time.

I look forward to your investigation of these issues. I will be happy to assist if requested.

Changed 17 months ago by deadletterfile@…

patch to stabilize EIT over-the-air broachast schedule

comment:7 Changed 17 months ago by deadletterfile@…

mythbackend version: (detached from f19d9f8) [v0.28-pre-755-gf19d9f8-dirty]

Attached is libmyth.140128.patch. It is a unifed patch of my changes which have stablized the use of over-the-air broadcast information on an atsc broadcast stream. The patch is against the master branch updated to commit f19d9f8062 (Jan 27,2014). As it includes proposed fixes for submitted bugs 11476 and 11739 I will attempt to briefly comment only on changes related to title/description mismatches here.

The latest adjustment was the elimination of the ETT cache in EITHelper::AddETT

What I theorize has been happening is EIT/ETT mismatches have occurred at times when tables were updated in the broadcast stream. I do not think EIT or ETT are checked against the version_number in the latest MGT, therefore potentially an old table could be used. This theory matches the observation of others that the problem gets worse with time (as more stations encounter this problem), and after reboot, there are initially no new title/description mismatches (with the UpdateDB function adjusted as below).

I have run the patch on gf19d9f8-dirty for four days (and for 48 hours on my prior build) and have seen no degradation in service. At least in my case, the ETT caching seems unnecessary. Alternately, caching could be checked and selectively purged before use, or the Add<ETT EIT> functions could be synced and process data only after the latest ETTs and EITs are known to exist.

As earlier reported, I continue to eliminate the overwriting of program/event descriptions based on length in the 'uint DBEvent::UpdateDB' function, although I admit the original intent of the code escapes me.

Other changes to eithelper.<h cpp> eitscanner.cpp and atscstreamdata.cpp have to do with converting QMutex to QMutexLocker or with bug 11476 and my attempts to resolve that issue. I admit to being less than a newbie with locking. The attempt was to clarify code and reduce the risk of thread collisions. I do not vouch for their integrity, however I am having no issues.

At this time, with this patch installed, all my broadcast schedule issues are solved. Scheduling is working optimally for me.

Thank you for your time. - RoyBoy626

comment:8 Changed 17 months ago by stuarta

  • Cc stuarta added
  • Owner changed from stuarta to dekarl

Changed 14 months ago by deadletterfile@…

astc EIT fixes #11476 #11739; off commit befdc4d4b9

comment:9 Changed 13 months ago by deadletterfile@…

libmythtv.140430.patch patches v0.27.1 (commit ecfefabd22 - May 26,2014). ATSC EIT Program Guide works with no detected issues.

comment:10 Changed 10 months ago by dekarl

  • Component changed from MythTV - General to MythTV - ATSC

comment:11 Changed 10 months ago by deadletterfile@…

libmythtv.140430.patch (Comment #7) patches v0.27.3 (commit 9d871a999c - July 4,2014). ATSC EIT Program Guide works with no detected issues.

comment:12 Changed 8 months ago by deadletterfile@…

libmythtv.140430.patch (Comment #7) patches v0.27.4 (commit e4f65c8797 - Oct 15, 2014). ATSC EIT Program Guide works with no detected issues.

comment:13 Changed 4 months ago by sagaciouskjb@…

I just applied this patch against v.0.27.4 ( forgive me but I don't know how to reference commit number, I just pulled it from the repository last night ) and noticed that the "browse" mode still sometimes mismatches the description when watching LiveTV. In my case it mismatched the description of "Adventure's of Superman" with the episode of "Bonanza" that had aired on the same channel the previous hour ( a full hour ). However when I loaded the actual EPG it showed the correct description, and when I dropped back from the EPG to the browse mode it had updated as well.

comment:14 Changed 4 months ago by deadletterfile@…

Thanks for the report. Did you wait until completely new listings were imported? If a station listing goes out 12 hours, wait 14 hours or so and check the listings. I have 23 stations, and in casual almost daily use over ten months, I have noted only two mismatches. There is still at least one bug I have not discovered, but for me, the improvement by applying libmythtv.140430.patch has been dramatic. In my amateur opinion, one problem that greatly reduced mismatches was in EITHelper::AddETT. The program caches ETT data, but the data can be accessed when it is old. A missing return statement in TVRec::TuningSignalCheck? (I think) causes the EIT grabber to crash. Third (and where undiscovered bugs perhaps are located) was improper mutex locking.

comment:15 Changed 4 months ago by sagaciouskjb@…

Yeah, but it did happen shortly after I thought they were all done, so perhaps some were still download/updating. I haven't noticed a repeat of the behavior since. Seems to be working 100% in the 24 hours since my comment. Thanks for the fix

comment:16 Changed 3 months ago by geoffp@…

For what it's worth, I've been running the most recent patch for this against the current Ubuntu build (mythtv-0.27.1+fixes.20140624.aa822f5) for about a month with no apparent issues. It's a dramatic improvement for me.

Even if it's not 100% correct yet (though it appears correct for me), it's a major UX improvement over every description being wrong all the time. Kudos to deadletterfile and all testers.

comment:17 Changed 3 months ago by ironstorm@…

I'm also running Ubuntu, however my results are not good.

I tried:

  • mythtv-0.27.1+fixes.20140624.aa822f5 + libmythtv.140430.patch (stock ubuntu mythtv package)
  • mythtv_0.27.4+fixes.20150305.f1115fc-0ubuntu0mythbuntu2 + libmythtv.140430.patch (mythbuntu package)

The results are the same with each, EIT information is fixed each time the patch is applied - which is good because the vast majority of my EIT info is garbage right now (it's rare when any titles and descriptions match up). However after a few minutes of running, mythbackend begins to consume all available CPU (~144% on my dual core chromebox), this continues even after the backend goes "idle" (no recording, flagging, transcoding, watching, etc).

A restart brings the CPU back in line, but it quickly goes back up again. The original un-patched packages do not suffer from this problem. This level of CPU consumption make watching on a local mythfrontend impossible due to choppiness, drop outs and errors. Reverting back to the stock packages makes the CPU problem go away.

My set-up is:

  • Asus Chromebox serving as backend and front end
  • 1 HDHomeRun Plus (onboard transcoding) -> OTA antenna
  • 1 HDHomeRun Original (no transcoding) -> same OTA antenna

comment:18 Changed 3 months ago by geoffp@…

My set up is:

  • Separate backend on Ubuntu Server, Kodi frontend
  • HDHomeRun Dual (ATSC/North America)

Maybe the differential is the second tuner? Or a second, very similar tuner on the same source?

comment:19 Changed 3 months ago by Wolfgange <s.matthew.31@…>

I'm a bit confused, where exactly is the mythtv folder that has libs/libmythtv? I cannot find a libs folder in /var/lib/mythtv/.

comment:20 Changed 3 months ago by ironstorm@…

@geoffp,

I just did a test, disconnected the power to my HDHomeRun Original (it's a Dual Tuner, but is white with 2 coax ports), then I reinstalled the patched packages of mythtv_0.27.4+fixes.20150305.f1115fc-0ubuntu0mythbuntu2 and started everything up.

So far mythbackend has been behaving for the past 2 hours without any trace of the CPU issue. My guess is the changes in this patch relating to locking/unlocking/mutexes tuners have some kind of problem.

@Wolfgange, its inside the mythtv source folder... Here's how I built it and installed it

cd ~/source
curl -O https://code.mythtv.org/trac/raw-attachment/ticket/11739/libmythtv.140430.patch
sudo apt-get update
sudo apt-get source mythtv
cd mythtv-0.27.4+fixes*/mythtv/libs
cat ../../../libmythtv.140430.patch | patch -p0
cd ../..
sudo debian/rules binary
cd ..
for m in $(dpkg --get-selections | grep myth | cut -f1); do sudo dpkg -i $m*.deb; done

comment:21 Changed 3 months ago by ironstorm@…

@geoffp, forget what I said ... It's back to using 130%+ CPU doing nothing... looks like the patch is bad, just a matter of timing before myth backend goes nuts. :(

29010 mythtv     20   0 3658M 63612 29128 S 132.  0.4 29:27.21 /usr/bin/mythbackend --syslog local7 --user mythtv --daemon --noupnp

comment:22 Changed 3 months ago by ironstorm@…

@geoffp, nevermind... now I'm getting the CPU overload with my one tuner on the stock packages... maybe this is something to do with the HDHomerun Plus... :(

comment:23 Changed 3 months ago by stuartm

For dual tuners, try disabling EIT collection on just one of them. Some drivers/hardware don't handle repeated retuning on two tuners simultaneously. Technically it's possible for this to occur through normal usage, but the probability increases greatly with EIT scanning since it's constantly changing frequencies.

Last edited 3 months ago by stuartm (previous) (diff)

comment:24 Changed 3 months ago by geoffp@…

Good sleuthing, though, @ironstorm -- what appeared to be a problem with the patch was found not to be. That gets us closer to the truth.

comment:25 Changed 3 months ago by geoffp@…

Interestingly -- I did glance at CPU usage on my backend, and to my surprise, mythbackend was at 100%, but it subsided after a few minutes. I seem to remember that CPU usage spikes when it's doing rescheduling. I've been watching to see if I can catch it happening again so I can correlate it with log entries, but I haven't seen it happen since.

I have seen plenty of EIT scanning activity in the log since then, though, which does *not* correlate with a CPU usage spike. Therefore, I can say with very high confidence that with patches applied, EIT scanning activity doesn't always spike CPU, and I think I can say with pretty decent confidence that I've never even seen it happen once.

comment:26 Changed 3 months ago by geoffp@…

Update: I caught one of the CPU spikes. It lasted about 14 minutes, and correlates with this block of log entries:

Mar 31 10:02:52 basementcat mythbackend: mythbackend[10810]: I Scheduler scheduler.cpp:2139 (HandleReschedule) Reschedule requested for MATCH 0 1 4 2015-04-02T14:50:00Z EITScanner
Mar 31 10:02:52 basementcat mythbackend: mythbackend[10810]: I Scheduler scheduler.cpp:2252 (HandleReschedule) Scheduled 44 items in 0.0 = 0.01 match + 0.00 check + 0.03 place
Mar 31 10:05:34 basementcat mythbackend: mythbackend[10810]: N Expire autoexpire.cpp:264 (CalcParams) AutoExpire: CalcParams(): Max required Free Space: 90.0 GB w/freq: 15 min
Mar 31 10:08:15 basementcat mythbackend: mythbackend[10810]: I Scheduler scheduler.cpp:2139 (HandleReschedule) Reschedule requested for MATCH 0 1 10 2015-04-03T14:00:00Z EITScanner
Mar 31 10:08:15 basementcat mythbackend: mythbackend[10810]: I Scheduler scheduler.cpp:2252 (HandleReschedule) Scheduled 44 items in 0.0 = 0.01 match + 0.00 check + 0.03 place
Mar 31 10:13:38 basementcat mythbackend: mythbackend[10810]: I Scheduler scheduler.cpp:2139 (HandleReschedule) Reschedule requested for MATCH 0 1 1 2015-04-02T14:30:00Z EITScanner
Mar 31 10:13:38 basementcat mythbackend: mythbackend[10810]: I Scheduler scheduler.cpp:2252 (HandleReschedule) Scheduled 44 items in 0.0 = 0.01 match + 0.00 check + 0.03 place

The backend CPU is an Intel(R) Core(TM)2 Duo E8400 @ 3.00GHz. Not a speed demon by today's standards.

Add Comment

Modify Ticket

Action
as assigned The owner will remain dekarl.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.