Incorrect EIT CRID IMI processing on split programme segments

[lucylangthorne55@…]

When a programme is split for news, mythtv incorrectly only records a random half of the programme since the other half is incorrectly marked as a duplicate due to incorrect IMI handling.

When using EIT, programmes that have a crid that includes an IMI (Instance Metadata ID) can be used to identify programmes where a single piece of content has been split in to more than one part.

An example document stating that the split should be considered one programme is in ​http://www.freeviewnz.tv/media/1055/freeview_dtt_transmission_rules_2_1.pdf which says in §7.1.7:

"A CRID in the CID shall be a programme CRID (crid type 0x31) with an IMI extension. Where two events have the same CRID and IMI value and the gap between each event is less than 3 hours (measured from the end of the preceding event to the start of the next event) then they shall be considered to be segments of a single item of content. An item of content may be split across more than two events as long as the gap between each event remains less than 3 hours."

So using dvbsnoop I can see that my film has:

(Part 1 of film):
    Event_ID: 2364 (0x093c)
    Start_time: 0xe263190000 [= 2017-07-21 19:00:00 (UTC)]
    Duration: 0x0011000 [=  01:10:00 (UTC)]
    ...
            DVB-DescriptorTag: 118 (0x76)  [= content_identifier_descriptor]
            descriptor_length: 14 (0x0e)
            crid_type: 49 (0x31)  [= user defined]
            crid_location: 0 (0x00)  [= Carried explicitly within descriptor]
            crid_len: 12 (0x0c)
            crid_bytes: 
                 0000:  2f 32 35 36 37 32 34 37  38 38 23 31               /256724788#1
...
(News)
    Event_ID: 2365 (0x093d)
    Start_time: 0xe263201000 [= 2017-07-21 20:10:00 (UTC)]
    Duration: 0x0000500 [=  00:05:00 (UTC)]
...
(Part 2 of film):
    Event_ID: 2366 (0x093e)
    Start_time: 0xe263201500 [= 2017-07-21 20:15:00 (UTC)]
    Duration: 0x0010000 [=  01:00:00 (UTC)]
...
            DVB-DescriptorTag: 118 (0x76)  [= content_identifier_descriptor]
            descriptor_length: 14 (0x0e)
            crid_type: 49 (0x31)  [= user defined]
            crid_location: 0 (0x00)  [= Carried explicitly within descriptor]
            crid_len: 12 (0x0c)
            crid_bytes: 
                 0000:  2f 32 35 36 37 32 34 37  38 38 23 31               /256724788#1

Note that crid and IMI for both parts are identical.

Sinced the crid is identical, mythtv thinks the two halves of the programme are in fact the same identical programme rather than two parts of the programme.

Ideally the broadcaster and spec would have used #1 and #2, but that would be far too sensible.

I have not yet seen a programme split in to three segments, even though that is valid according to the above spec, nor a case where there isn't exactly one programme in between the parts such as "part1, <news>, part2".

Looking in my database, I noticed that mythtv currently strips the IMI. That means a simple query can not locate such records.

For full backwards compatibility, we probably can't store the IMI in the database programid since it would mean many existing programmes would no longer match the same crid. Besides, a programme may be split today but not split on a subsequent showing.

Anyway, a query to show duplicates (since the split is always short so we can work around lack of IMI):

select distinct P1.title, P1.partnumber, P1.programid, P1.starttime,
P1.endtime, P2.starttime, P2.endtime from program P1, program P2 where
P1.programid <> '' and P1.programid=P2.programid and
P1.chanid=P2.chanid and P1.starttime < P2.starttime and
timestampdiff(MINUTE, P1.endtime, P2.starttime) < 15 and 
P1.title not in ('Teleshopping', 'Home Shopping', 'News') and 
P1.starttime > now();

Even though I use EIT, I thought perhaps manually setting partnumber to 1 and 2 and parttotal to 2 in the database would work, but it didn't work and I assume the scheduler ignores those fields since programid is identical and it is mythfrontend that is reconstructing the description to contain "(Part 1 of 2)". Plus I haven't looked at exactly when ProgramData? would overwrite the data anyway but assume it only overwrites the whole data if broadcast EIT was marked as changed.

So, I'm not sure exactly how the bug could easily be fixed since it wasn't immediately obvious to me how eitcache interacts with the program table.

Since this splitting seems only to be used for movies, perhaps the full crid could be put in to seriesid and a fake (but consistent) crid (such as crid+start_hour_min) generated for programid so that at least "record all" rules would work, on the premise that it's better to record too much than to only have half and that certain types of movie are always on at the same times in the day. Unfortunately with all the +1 hour channels it would mean every film recorded twice, but still arguably better than only half a movie.

I created this bug purely to get it documented since it has annoyed me for a few years and I finally got time to look in to it.

[Stuart Auchterlonie]

Closing all old tickets in trac.

If your issue still persists, please open an issue in Github ​https://github.com/MythTV/mythtv/issues

and reference the existing trac ticket.