Opened 14 years ago

Closed 12 years ago

#1035 closed task (fixed)

EIT rewrite

Reported by: Stuart Auchterlonie Owned by: danielk
Priority: minor Milestone: 0.21
Component: eit Version: head
Severity: medium Keywords:
Cc: Ticket locked: no

Description

This ISN'T a wishlist. This is what we need to do to rewrite the EIT handling for 0.20

#778 - Astra 19.2E EPG aka. remove Kristian Kalweit fix

#892 - Handling channels with high EIT data rates.

#913 - Handling of EIT data with additional data

#1025 - Handling of unicode characters within EIT data.

Split all eitFixup routines out of siparser. Use them on EIT data only after we have decided that this is new/updated information.

Keep track of which events we have sent to the database. ie. do our own caching of EIT data, getting rid of that currently done for events in siparser (also #943) (GreyFoxx has done most of this already)

EIT data prioritization. which is better EIT or xmltv? Within EIT, lower table numbers always seem to be better data. 0x4e & 0x4f now & next, 0x50 & 0x51 future, 0x60 & 0x61 far future, and other longer term EIT data from various providers

Fixup routines sometimes try and put a subtitle in the DB that exceeds the length of the subtitle field in the DB, so silent truncation of information occurs.

Attachments (7)

9664-gdb.txt (5.9 KB) - added by chris [@… 14 years ago.
9666-gdb.txt (5.8 KB) - added by chris [@… 14 years ago.
9666-myth.log (2.6 KB) - added by chris [@… 14 years ago.
9664-myth.log (2.6 KB) - added by chris [@… 14 years ago.
malta_channel_dump (13.6 KB) - added by btsglw@… 13 years ago.
Channel dump as requested by Daniel
malta_dtv_multiplex_dump (3.4 KB) - added by btsglw@… 13 years ago.
DTV_mulitplex dump as requested by Daniel
early_eit_fixup.diff (1.7 KB) - added by anonymous 13 years ago.

Download all attachments as: .zip

Change History (66)

comment:1 Changed 14 years ago by Stuart Auchterlonie

Owner: changed from danielk, stuarta to anonymous
Status: newassigned

comment:2 Changed 14 years ago by Stuart Auchterlonie

Owner: changed from anonymous to danielk, stuarta
Status: assignednew

comment:3 Changed 14 years ago by mythdev@…

you probably knew this anyway - but just in case it helps with any design decisions:

a) In UK dvb-t the EITFixupStyle2 function can account for ca 40% of the total processing time in LiveTV - its called ~40 times/sec - those Qt regexes really hurt in this function :)

b) For dvb-t I don't believe the scan needs to look at every channel - just every multiplex

I was about to look see if it was practical to skip ParseDVBEIT for say 10% of the packets or even to skip it completely during recording/livetv - ymwv!

comment:4 Changed 14 years ago by danielk

Owner: changed from danielk, stuarta to danielk

Reassigning, so this shows up on my ticket list

comment:5 Changed 14 years ago by danielk

(In [9062]) References #1035. Cleanup of EITFixUp.

This keeps most of the regular expressions around across calls to the fixup routines so that we don't need to depend on Qt's regular expression caching.

comment:6 Changed 14 years ago by danielk

(In [9074]) References #1035. SIParser refactoring.

comment:7 Changed 14 years ago by danielk

(In [9098]) References #1035. Eliminates the last ATSC EIT descriptor parsers in siparser.

comment:8 Changed 14 years ago by danielk

(In [9099]) References #1035. Removes DVB Content Descriptor parser from SIParser.

comment:9 Changed 14 years ago by danielk

(In [9100]) References #1035. Removes DVB Component Descriptor parser from SIParser.

comment:10 Changed 14 years ago by danielk

(In [9101]) References #1035. Removes DVB Short and Extended Event Descriptor parsers from SIParser.

comment:11 Changed 14 years ago by danielk

(In [9103]) References #1035. Couple missing pieces from [9100].

comment:12 Changed 14 years ago by danielk

(In [9110]) References #1035. Removes DVB EIT parser from SIParser.

comment:13 Changed 14 years ago by danielk

(In [9111]) References #1035. Removes unused ConvertDVBDate() method.

comment:14 Changed 14 years ago by danielk

(In [9112]) References #1035. Removes STTHandler. The System Time table table is always in a single segment, and we want to see the updates...

comment:15 Changed 14 years ago by danielk

(In [9116]) References #1035. Uses ATSCStreamData to call MGT, VCT, STT, and ETT handlers.

comment:16 Changed 14 years ago by danielk

(In [9117]) References #1035. Uses ATSCStreamData to call HandleEIT().

comment:17 Changed 14 years ago by danielk

(In [9118]) Fixes #1375, References #1035. Allow each A/V stream to use multiple Conditional Access methods.

Apparently this is used in Australia. I'm not sure if they use different methods for the video, audio, and data streams, or if they just list all the supported methods and hope one is sticks. Multiple CA methods per PMT was already supported.

comment:18 Changed 14 years ago by danielk

(In [9135]) References #1035, fixes some NIT descriptor parsing bugs.

comment:19 Changed 14 years ago by danielk

(In [9139]) References #1035. Removes a few more descriptor parsers from siparser.

This should also allow a DVB scans to pick up a few more transports correctly.

comment:20 Changed 14 years ago by danielk

(In [9227]) References #1035. DVB EIT versioning + fix for PMT pull.

The EIT versioning is the bulk of the patch, but there were also some problems with the PMT handling in [9224] discovered by Oscar which this should fix.

comment:21 Changed 14 years ago by danielk

(In [9228]) References #1035 and closes #1427, by applying patch from #1427.

This implements an event signature cache, so that we avoid processing an event again during any one EIT scan.

This should lower the memory requirements and CPU load of the EIT Scan somewhat.

comment:22 Changed 14 years ago by danielk

(In [9664]) References #1035. Minor EIT cleanup & ATSC EIT bug fix.

This removes some now unused signals and slots, and removes all signals and slots in EITHelper.

This also processes the MGT and VCT in siparser. This was disabled by commit a week or two ago, and breaks EIT in ATSC Land.

comment:23 Changed 14 years ago by danielk

(In [9668]) References #1035. ATSC EIT refactor.

This moves most of the EIT parsing for ATSC out of SIParser and into the eit classes. This also adds language awareness and preference to the ATSC EIT parsing.

comment:24 Changed 14 years ago by chris [@…

Hi,

I am getting a crash on startup with 9664-8. I've tried 9663 which is fine. Looking at the GDB output, it appears to be a problem in the EIT scanner.

I've attached some backtraces from 9666 and 9666.

HTH

Chris

Changed 14 years ago by chris [@…

Attachment: 9664-gdb.txt added

Changed 14 years ago by chris [@…

Attachment: 9666-gdb.txt added

Changed 14 years ago by chris [@…

Attachment: 9666-myth.log added

Changed 14 years ago by chris [@…

Attachment: 9664-myth.log added

comment:25 Changed 14 years ago by danielk

(In [9669]) References #1035. This moves most of the EIT parsing for DVB out of SIParser and into the eit classes.

I haven't ported the fixup functions over yet. The new structure depends on being able to de-link eit event processing from the channel/transport we are currently on in the SIParser. But a lot of the "PrivateType?" stuff for the fixups is not cached.

comment:26 Changed 14 years ago by danielk

(In [9671]) References #1035. Restores EITFixUp's for DVB EIT.

comment:27 Changed 14 years ago by danielk

(In [9672]) References #1035. Fixes a copy-n-paste error in program credits update in [9671].

comment:28 Changed 14 years ago by danielk

(In [9673]) Closes #1667, References #1035. Fix for off-by-one error in myth listings category lookup found by Janne.

comment:29 Changed 14 years ago by danielk

(In [9674]) Fixes #1668. References #1035. Fixes copy-n-paste error.

What should have been the index in the loop was copied from another part of the function that used a result of the loop.

comment:30 Changed 14 years ago by danielk

(In [9688]) References #1035. Removes --disable-dvb-eit compile-time option.

This moves the eit stuff outside the DVB umbrella, it is now almost usable by the HDTVRecorder (for those HD-2000 folks).

comment:31 Changed 14 years ago by danielk

(In [9689]) References #1035, Fixes #892. Implements rate limiting for EIT events.

comment:32 Changed 14 years ago by danielk

(In [9701]) References #1035. Fixes timezone conversion for ATSC EIT events. This was messed up earlier this week (timezone compensation was applied twice).

comment:33 Changed 14 years ago by danielk

(In [9743]) References #1035, References #1701.

You may need to do a distclean. This removes signals and slots from the channels and recorders.

I've tested this with every recorder I have access to, but I don't have access to a firewire or dbox2 recorder.

If you experience any problems please do a distclean before reporting the problem.

The reasons for getting rid of signals and slots in non gui elements are because they require delayed deletes and other extra complexity to avoid segfaults, and because they are not thread safe and are incompatible with virtual inheritence and multiple inheritence.

comment:34 Changed 14 years ago by danielk

(In [9763]) refs #1035, refs #1688. Preferred language fix for DVB EIT.

comment:35 Changed 14 years ago by danielk

(In [9767]) Refs #1035, refs #1688. Always do some basic fixups with DVB EIT.

comment:36 Changed 14 years ago by danielk

(In [9798]) Refs #1035. Refs #1049. This is a continuation of the refactor started in [9790].

This makes it possible for DVBRecorder and SIParser to use a DVBStreamData in addition to a ATSCStreamData.

This allows us to move more of the DVB specific EIT stuff from SIParser to DVBStreamData.

I also referenced #1049, because this lets us use a DVBStreamData in DVBSignalMonitor and then pass it along to both the DVBRecorder and DVBSIParser; which simplifies the implementation of NIT/SDT monitoring.

This also adds SetStreamData?() and GetStreamData?() stubs to DTVRecorder, this allows us to get rid of a few of the ifdef's in TVRec. TVRec has also been modified to use a DVBStreamData rather than an ATSCStreamData when starting a signal monitor for a non-ATSC digital channel.

comment:37 Changed 14 years ago by danielk

(In [9807]) Refs #1035. More EIT refactoring.

This removes all the EIT handling from SIParser except for the pid filtering function.

comment:38 Changed 14 years ago by danielk

(In [9816]) Refs #1035. More EIT refactoring for EIT branch.

The major change is the removal of the DVBSIParser and the code it depends on. DTVSignalMonitor picks up it's small remaining role in active EIT scanning. This in turn enables EIT scanning for all recorders with a DTVSignalMonitor based signal monitor.

This patch looks a lot bigger than it is. Many of the changes are deletions of files we no longer use now that [DVB]SIParser is gone.

By replacing the dvbdevice() function, which was our only remaining dependence on dvbdev aside from dvbci, we can also handle more than 4 DVB devices now. Admitedly this is not a common problem, but there is no good reason for this limitation.

comment:39 Changed 14 years ago by danielk

(In [9830]) Refs #1035. Adds a convenience methods that allow you to query which channels are cached in the stream data.

comment:40 Changed 14 years ago by danielk

(In [9831]) Refs #1035. Refactors DVB recording a bit to work better with StreamData? classes (which is being used for more generalized EIT handling).

In order allow the recorder to monitor the PAT & PMT for the stream this makes DVBRecorder and not get confused by multiple PMTs on a pid this moves the HandlePAT() and HandlePMT() methods from the DVBChannel to the DVBRecorder.

This moves some the channel table DB stuff from TVRec to ChannelUtil? as a cleanup. This also to allows us to more easilly check whether two channels share a multiplex by untie these queries from the current channel in the channel class.

This also adds the other pids in the PAT to the pids being monitored in DVBRecorder, this allows us to more quickly change between channels on the same transport by simply querying if we have the needed PSIP tables and using them if they are present.

So far this has been tested with the HDHomeRun and DVB recorders, but it should also speed up FireWire? and HDTV recorder channel changing since the use the StreamData? classes.

comment:41 Changed 14 years ago by danielk

(In [9841]) Refs #1035, refs #1728. Fixes a cut-n-paste error in DVBRecorder which caused a segfault in DVB Land (w/ patch from Janne).

comment:42 Changed 14 years ago by danielk

(In [9844]) Refs #1035. Fixes #1049. Adds SDT monitoring to DTVSignalMonitor in mythtv-eit branch.

What this does is require a matching SDT table for DVB channels before decide that we are on the right transport. This especially helps with DVB-S tuning, but also lets us collect the SDT table in the DTVSignalMonitor which allows us to dependably collect DVB EIT now that the SIParser class is gone.

This also changes the scanning for ATSC/MPEG a bit. We've been inserting a networkid of '1' instead of '0' for these transports for a while. There is a db update in dbcheck to fix existing transports and some changes in siscan to insert these properly in future channel scans.

Please test/debug so that I can merge this with SVN-head soon.

comment:43 Changed 14 years ago by danielk

(In [9847]) Refs #1035. merged trunk/mythtv changes r9833:9846 into the mythtv-eit branch.

comment:44 Changed 14 years ago by danielk

(In [9957]) Refs #1035. Big EIT branch merge.

This moves the EIT handling out of SIParser and mostly into the DVB and ATSC stream handlers, allowing recorders other than the DVBRecorder to handle EIT without massive code duplication. This also allows us to get rid of a bunch of dependencies on the dvbdev stuff. The only thing we are still using there is the CAM code.

This also fixes several problems in the MPEG PES packet assembly code that were uncovered when we started using it for EIT in addition to using it for the tuning tables.

Since SIParser is gone this also allows us to use the table caching in the stream handler code to speed up intra-transport tuning. If we already have the needed tuning tables we skip the signal monitoring step and begin recording the new channel immediately.

This also fixes various bugs in EIT handling, and lowers the EIT processors memory consumtion further.

This also removes the arbitrary limit of four DVB cards per backend. Now you should be able to add as many DVB devices as your hardware allows.

I recommend a "make distclean".

comment:45 Changed 14 years ago by danielk

(In [9973]) Refs #1035. Fixes #1801. Fixes a copy-n-paste error that resulted in an ignored EIT fixup regular expression in the UK EIT fixup.

comment:46 Changed 14 years ago by danielk

(In [9976]) Refs #1035. Fixes #1728. Applies Janne's EIT Cache 'save to DB' patch.

This allows the EIT version cache to be save to the DB so that it can be shared amoung recorders on several backends.

comment:47 Changed 13 years ago by danielk

(In [10238]) Refs #1035. Refs #1728. Refs #1801. Refs #1884. Refs #1615. EIT branch merge.

  • Improves the EIT Fixup functions for the UK and Sweden.
  • Improves the EIT version cache (lowering memory and CPU requirements).
  • Fixes various small memory leaks and lowers memory requirements with earlier deletes.
  • Fixes a spurious error message in filldata with EIT only sources.

comment:48 Changed 13 years ago by danielk

(In [10244]) Refs #1035. Undo overly Janne's leak fix for StreamData? in tv_rec. It improperly deletes stream data in some cases.

comment:49 Changed 13 years ago by danielk

Milestone: 0.200.21

comment:50 Changed 13 years ago by danielk

Priority: majorminor

comment:51 Changed 13 years ago by Stuart Auchterlonie

(In [10584]) Refs #1035. Sync's mythtv-eit branch back to head.

comment:52 Changed 13 years ago by Stuart Auchterlonie

(In [10614]) Refs #1035, increases the wait time in the eit event loop

This increases the wait time in the eit event loop from 200ms to 400ms, which makes the eit processing less aggressive and more like background processing.

comment:53 Changed 13 years ago by danielk

(In [11752]) Refs #1035. Several European DVB providers don't set the character set for their EIT listings as required by the DVB standards so we have been applying an override to set it to a sensible value; but, for some reason the 'sensible value' we chose was the Turkish character set. In all likelyhood they are using something closer to the Western European character set, so I've changed the character set appropriately.

The two character sets are very similar, they differ mostly in how they handle some special characters which is why this wasn't caught earlier.

Changed 13 years ago by btsglw@…

Attachment: malta_channel_dump added

Channel dump as requested by Daniel

Changed 13 years ago by btsglw@…

Attachment: malta_dtv_multiplex_dump added

DTV_mulitplex dump as requested by Daniel

comment:54 Changed 13 years ago by anonymous

Change set 11752 introduced an eit bug with Bell Expressvu. The early eit fixups for the character set is overwriting previous defined fixups.

Changed 13 years ago by anonymous

Attachment: early_eit_fixup.diff added

comment:55 Changed 13 years ago by Stuart Auchterlonie

(In [11817]) Refs #1035. Fixes the problem with the Bell Expressvu eit fixups that were introduced in [11752]

comment:56 Changed 13 years ago by Janne Grunau

(In [12820]) Refs #2342, #1035. Adds channel looking to eit cache

This should prevent duplicate eit insertions. The channel looks are removed when the master frontend starts or after 24 hours. The eit cache will now written back to the DB after every channel change. I hope this isn't too much stress for anyone's database / backend server. I didn't notice the higher load.

comment:57 Changed 13 years ago by Janne Grunau

(In [12831]) Refs #1035. Optimize DVB EIT insertion.

One DVB event information table caries only events of a single channel => look the chanid only once per section up.

Simplifies the eit cache by removing duplicate chanid lookup.

Fixes eit cache issues with the same channel on different sources (reported by Otto Kolsi)

comment:58 Changed 13 years ago by danielk

Component: mythtveit

comment:59 Changed 12 years ago by danielk

Resolution: fixed
Status: newclosed

We'll open a new ticket for the next EIT rewrite..

Note: See TracTickets for help on using tickets.