EIT/EPG improvements for DVB-C

[klaas.de.waal@…]

The EPG of MythTV is not correct for the DVB-C signal from provider Ziggo in The Netherlands.

The EIT/EPG data from the DVB-C signal is like this:

• There is a EPG data for up to 3 or 4 days ahead
• If there is no program data yet there are dummy program entries with the title "Zender verstrekt geen informatie" ("Broadcaster does not provide information") or with other titles that convey a similar message.
• For 1 to 3 days ahead there are program entries with title/starttime/endtime.
• For 1 day ahead, at least 24 hours, all program entries have also a subtitle and a description.

Processing the received EIT entries and updating the EPG is implemented in libs/libmythtv/programdata.cpp. In this implementation each new entry is compared with what is already in the database and based on the result of the comparison the new entry can be:

• inserted in the database
• used to update an existing entry, complete or partially
• or completely discarded!

The result is that quite often the MythTV Guide shows for today the dummy entries that were broadcasted three days ago instead of the actual programs.

My TV does not have any EPG memory and only starts collecting EPG data after it is switched on. It always has a correct and complete EPG within 30 to 60 seconds. This suggests to me that all comparisons and selective updating that is in the current implementation is not correct, at least not for my DVB-C signal.

I have greatly simplified the code in programdata.cpp so that it always accepts all new program entries and that it deletes everything in the database that conflicts with the new data. With this code the MythTV EPG shows exactly the same programs as my TV.

The file programdata.cpp is attached.

In order to test this I have added a lot of debug output. This can be enabled like this: $ mythbackend --loglevel debug -v eit.

The new log output looks like this:

2013-10-18 04:02:21.932963 D  EIT: update chanid 1008 2013-10-18T11:15:00Z 'Ranger dierenjournaal vanuit Emmen'
2013-10-18 04:02:21.936359 D  EIT: update chanid 1008 2013-10-18T11:20:00Z 'Ultimate Spider-man'
2013-10-18 04:02:21.939324 D  EIT: delete chanid 1901 2013-10-21T00:03:00Z 'Zender verstrekt geen informatie'
2013-10-18 04:02:21.941013 D  EIT: insert chanid 1901 2013-10-21T00:03:00Z 'NOS Tekst tv'
2013-10-18 04:02:21.954900 D  EIT: update chanid 1311 2013-10-18T12:31:00Z 'Doctor Cheezy'
2013-10-18 04:02:21.967634 D  EIT: update chanid 1311 2013-10-18T12:57:00Z 'SpangaS'
2013-10-18 04:02:22.380320 D  EIT: update chanid 1311 2013-10-18T13:17:00Z 'Het Klokhuis'

Note that I have only tested this with my local DVB-C signal.

N.B. The Ziggo DVB-C signal is also discussed in ticket #10136, although the subject of that ticket is the "duplicate key entry" messages caused by a bug in the current implementation of programdata.cpp.

[klaas.de.waal@…]

"newer is better" implementation for EIT/EPG

[bas-t <tycholursen@…>]

Hi Klaas, is this ment to use in addition to your patch in #10136?

I tried compiling without your previous patch in branch fixes/0.26. That did not compile. the error was:

programdata.cpp: In member function ‘uint DBEvent::GetOverlappingPrograms(MSqlQuery&, uint, std::vector<DBEvent>&) const’:
programdata.cpp:253:9: error: ‘CategoryType’ is not a member of ‘ProgramInfo’
programdata.cpp:253:35: error: expected ‘;’ before ‘category_type’
programdata.cpp:261:13: error: ‘category_type’ was not declared in this scope
make[2]: *** [programdata.o] Fout 1
make[2]: *** Wachten op onvoltooide taken...
make[2]: Map '/usr/src/mythtv/mythtv/libs/libmythtv' wordt verlaten
make[1]: *** [sub-libmythtv-make_default] Fout 2
make[1]: Map '/usr/src/mythtv/mythtv/libs' wordt verlaten
make: *** [libs] Fout 2

[Klaas de Waal <klaas@…>]

Replying to bas-t <tycholursen@…>:

Hi Klaas, is this ment to use in addition to your patch in #10136?

I tried compiling without your previous patch in branch fixes/0.26. That did not compile. the error was:

programdata.cpp: In member function ‘uint DBEvent::GetOverlappingPrograms(MSqlQuery&, uint, std::vector<DBEvent>&) const’:
programdata.cpp:253:9: error: ‘CategoryType’ is not a member of ‘ProgramInfo’
programdata.cpp:253:35: error: expected ‘;’ before ‘category_type’
programdata.cpp:261:13: error: ‘category_type’ was not declared in this scope
make[2]: *** [programdata.o] Fout 1
make[2]: *** Wachten op onvoltooide taken...
make[2]: Map '/usr/src/mythtv/mythtv/libs/libmythtv' wordt verlaten
make[1]: *** [sub-libmythtv-make_default] Fout 2
make[1]: Map '/usr/src/mythtv/mythtv/libs' wordt verlaten
make: *** [libs] Fout 2

Hi Bas, as mentioned in the ticket this is done on the 0.27-fixes, not on the 0.26. It should compile cleanly on a 0.27. BTW, you better ask this on the dev mailing list or in an email directly to me ( at gmail.com or klaas at kldo dot nl) because the ticket is not the mailing list and the devs will lock the ticket!

[klaas.de.waal@…]

After running for a few days more with this fix there are sometimes programs that are present in the EPG of my Samsung TV but that are not present ("Unknown") in the MythTV EPG. The "newer is better" behaviour implemented in this patch is still a big improvement compared with the 0.27 implementation but it is not perfect. Maybe the EITcache is also of influence here.

[klaas.de.waal@…]

The simplified approach in which all overlapping or conflicting programs were deleted did in the end lead to a lot of "holes" (shown as "unknown") in the EPG. To fix this I have put back the original code that does the adjustment of start- and endtimes of the adjacent programs. In order to be able to completely trace what is happening to each EIT update I have added a lot of debug output that appears when mythbackend is started with the "-v general,eit --loglevel debug" options. The changes in this version of programdata.cpp, compared with today's 0.27-fixes version, are:

• The patch described in and attached to ticket #10136 to prevent the database errors
• Do not insert or update programs that are completely in the past
• Extended the query to find overlapping programs to include the case where the new program is completely "inside" an existing program
• Do not update a program with a modified starttime when the starttime is in the past. This solves the problem that changing the starttime of a program that is already being recorded causes another recording of the same program.
• In the debug log output, standardize the program names to be always truncated to 35 characters and to be always within single quotes. Previously it was sometimes the full program name and sometimes truncated to 30 characters, and sometimes between single quotes and sometimes not.
• Added lots of comments.

The above contains only fixes that IMHO cannot cause problems with other EIT/EPG sources. Of course, this should be tested by somebody as I can only test against the Ziggo DVB-C cable TV signal. There will be two files attached: programdata.cpp.diff.v10 which is the result of the "git diff" command, and the complete source file programdata.cpp.v10.

[Stuart Auchterlonie]

Patch looks okay on initial reading

[Klass De Waal <klaas.de.waal@…>]

In 95652b68a022a5ce2350952779f77891e5d1ded1/mythtv:

EIT improvements for DVB

Fixes #11914, Refs #10136

Signed-off-by: Stuart Auchterlonie <stuarta@…>