DVB Scanning Issues

[f.lange@…]

Channel scan returns rather cryptic channel names and channel numbers. Program info via EIT won't work, though watching TV and channel switching with arrow keys works fine. Output of -v siparser attached.

[f.lange@…]

output mythtv-setup -v siparser

[danielk]

[invalid] No indication of an error occurring.

Did you find an SDT on this transport with the old scan?

If so have you tried extending the timeouts?

Please answer in dev mailing list.

[danielk]

(In [9318]) References #1485. Make sure SDT pid is added on stream data reset.

[f.lange@…]

mythtv-setup -v siparser,channel

[f.lange@…]

Version 9318: same channel list as version 9313. Log of complete scan of all transports attached.

[danielk]

(In [9344]) Closes #1485. SDT was being disabled in siscan.

We used to disable the SDT until we had seen the NIT. This was because we weren't caching the SDT's at the time and we wanted to see the UK channel number map, if it existed, before inserting any SDTs. But since we do cache the tables now this code was just causing problems without any benefits.

[anonymous]

Sorry, not yet fixed.

[danielk]

ticket reopened without additional information (like the requested logs).

[stuart@…]

f.lange@… added the logs on 10/03/2006 but just in case those weren't OK I've attached my own.

Quick summary to any questions you have asked on the subject: Timeouts have been increased far beyond what they were previously. Yes scanning worked before and yes the SDT was found.

[stuart@…]

mythtv-setup -v siparser,channel

[danielk]

(In [9464]) Closes #1485. Hopefully fixes the missing sdt pid issue for good.

[anonymous]

Problem still unchanged after 9464. I don't know what else I can add...

[stuart@…]

Between Stuart A and myself we've worked out all the current DVB scanning bugs. We'll supply the patches as we do the work.

Starting with a patch to fix the inversion of the HasFreeCA flag in siscan. HasFreeCA has also been renamed to HasCA which is less ambiguous.

[stuart@…]

Reassigning to stuarta and changing the summary. Will resolve all the _current_ DVB scanning issues under this ticket.

[stuart@…]

Disables call to GetBetterMplexID in UpdatePATinDB

[stuart@…]

Checks for existing of multiplex in DB before adding it (could be improved)

[stuart@…]

Disables insertion of additional frequencies from the frequency_list_descriptor

[stuart@…]

Added three more patches.

updatepat.patch - We don't have the networkid in UpdatePATinDB so GetBetterMplexID will always fail. For now just avoid calling GetBetterMplexID. This prevents the "Found channel, but it doesn't match existing tsid." error.

duplicatemultiplex.patch - Checks that a transport/multiplex doesn't exist in the database before adding it. This stops us ending up with 6 copies of each multiplex, 1 per NIT x 6 NITs.

frequency_list.patch - This disables processing of the frequencies in the frequency_list_descriptor. The additional frequencies stored in this descriptor don't include guard_interval, fec etc Those values would come from the delivery_system_descriptor. Eventual goal would be to pick the best frequency (signal/quality) and use that in the database. For now though disabling this prevents multiplexes with duplicate transportid and NULL guard_interval etc

[anonymous]

If I understand you correctly "Eventual goal would be to pick the best frequency (signal/quality) and use that in the database" - you may not have considered how these frequencies are used for dvb-t in the UK.

Presumably in order to prevent interference from different transmitters - each transmitter uses the centre frequency OR a single frequency selected from this list - if the other_frequency flag is set then the centre frequency is not used on that transmitter and one of the frequency list is used. The only case where two frequencies would be transmitting the same transport is in regions that were in range of two transmitters.

The issue that makes scanning a pain is that the frequency list and main frequencies for each transport may overlap - so you could test the signal on that frequency and get a valid strong signal - but it would actually be for a different transport - so selecting the strongest signals could actually lose you a transport.

I tried to explain this (very longwindedly) in #841 - I haven't tried to update that because I was waiting to understand the design of the new signal monitoring daniel was planning for .20

• good luck!

[stuart@…]

Temporary fix for SDT problem.

[stuart@…]

Nasty hack fix to stop more duplicate multiplexes being added to db and prevent some errors.

[danielk]

(In [9510]) References #1485. Rename HasFreeCA() to IsEncrypted?().

The old name matches the DVB standards' name for the field, but the IsEncrypted?() is clearer.

[stuart@…]

sdt.patch - Two connected problems prevented us from using the SDT data and thus from getting channels names/numbers/callsigns. In almost every scan we are seeing the SDT before the NIT. We can insert channels based on the SDT alone but the transports from the NIT need to go into the DB first. So the patch just ignores the SDT until we see the NIT. It might be faster to cache the first SDT we see and use it once we have run HandleNIT, which is why I've labelled this a temporary fix.

The second problem was a flaw in HasCachedAllSDT. We weren't seeing the NIT before the SDT so this function would always fail, but it also wrongly compares the number of transports in the NIT to the number of cached SDTs. If scanning just one transport at a time we will see six transports in the NIT but only one SDT. HasCachedAllSDT would make sense if we cached the SDTs from all transports but not if we are only examining one transport at a time.

getbettermplex.patch - (*current).mplexid (HandleDVBDBInsertion) never seems to correspond to an existing mplexid. This would cause GetBetterMplexID to always fail in UpdateSDTinDB and prevent it from completing. Created a new method "GetExistingMplexID" to replace it gives the intended behaviour. Daniel, you might want to approach this another way, I went for the quick fix.

[danielk]

(In [9532]) References #1485. Adds a method HasCachedSDT() to address one of the scanning issues.

We were waiting for the transport to contain all the SDT's associated with the NIT, but these can of course be located on different transports. HasCachedSDT() just checks to see that we have at least one complete set of SDT sections.

Also, if we do not find all the SDTs we are waiting on. I've enabled a bit of code in HandlePostInsertion?() which will insert what we do have and not insert stations based on the PAT if an NIT is found.

[danielk]

(In [9533]) References #1485, by applying modified version of the 'getbettermplex.patch'

I'm not sure that this will work correctly, but the existing GetBetterMPlex() is buggy so this shouldn't make things any worse.

[danielk]

Stuart, I applied all the patches that looked ready. I didn't add the first part of the sdt patch as it would break the ability of the dvbstreamdata to be used for tuning, and probably wouldn't work when things show up in a different order. But I believe I fixed the cached sdt check, which should allow the insert once the NIT is seen.

[stuart@…]

Thanks Daniel. There are two serious issues remaining and just a couple of cosmetic ones from my end of things. I'm told there are one or two further issues with DVB-C, I've yet to look closely at those and I'm unable to debug them so no guarentees.

Any comments on what is needed to get something like the duplicatemultiplex.patch implemented? I've actually noticed after submitting this patch that a similar approach is used elsewhere when inserting transports into the database (possibly old siparser code).

StuartA was looking at the right solution for the frequency_list.patch. I don't want to step on his toes but if I find some time during the week I might give that a go myself.

[danielk]

(In [9537]) References #1485. Adds duplitate checking code to insert_dtv_multiplex.

This is less trusting of the pre-existing transports than the patch in #1485, but should still prevent most duplicate insertions of transports. Probably for this to work better in DVB land an existing transport that didn't match the insert data exactly should be updated, but I didn't address that issue in this changeset.

[Stuart Auchterlonie]

I believe the correct solution to the other_frequency list problem it to provide a method where we can throw a list of frequencies to check for a given tsid back up to the scanner. This can then be used for two things.

1. Optimize scanning time by rewriting the scan list (this should be

a new country specific scan

2. Allow us to verify that the tsid we've tuned to is correct (since it

is possible in fringe areas to recieve a signal from an adjacent area on one of the alternate frequencies given in the NIT, but for a different tsid)

Also the correct way to handle these frequencies is to tune to each alternate frequency in turn, stopping once we find one where the tsid matches the tsid we know is associated with this transport.

From the NIT at moely parc

Transport_stream_ID: 4158
Center frequency: 0x03412140 (= 546000.000 kHz)
Other frequency descriptors:
               Centre_frequency: 0328f85b  (= 530166.067 kHz)
               Centre_frequency: 04724e40  (= 746000.000 kHz)
               Centre_frequency: 0303d725  (= 505833.033 kHz)
               Centre_frequency: 044daf40  (= 722000.000 kHz)
               Centre_frequency: 0410e75b  (= 682166.067 kHz)
               Centre_frequency: 04661940  (= 738000.000 kHz)
               Centre_frequency: 0328f85b  (= 530166.067 kHz)
               Centre_frequency: 04354540  (= 706000.000 kHz)

The freq in use is 738000 kHz which is well down the list, and you can see that they are not ordered according to frequency but the most likely one you will use.

Daniel, one thing I noticed while looking at this problem is the dvbdescriptor parsing code is not handling interdependecies between descriptors.

In this specific case the alternate frequency descriptors are ONLY present iff the other_frequency_flag = 1, and each one only provides a new frequency, the rest of the information comes from the original terrestrial_delivery_descriptor which preceeded it.

[Janne <janne-mythtv@…>]

First to general comments. The scan progress display alternates between 6% (might be 1/number of transports) and the right value. It goes to 6% after the finished processing message and goes back after adding services.

The new scanner overwrites channel name and callsign of existing services.

DVB-C, scan of existing transports: Several data services are picked up (Set top box updates, custom epg providers)

DVB-T, scan of existing transports: Data services. On some multiplexes I get "found channel, but it doesn't match existing tsid. ..." and the channel names are not picked up from SDT. Instead I get names like C38#1.

DELETED ALL CHANNELS

DVB-C, Full scan (tuned): is not working, seems that the scanner never tunes to the given parameters

DVB-T, Full scan: same as scan of existing transports. It is much slower than before.

DVB-C, scan of existing transports: same as before deleting all channels

After scanning the dtv_multiplex table holds for every DVB-T mplex a second one with garbled inversion value.

I'll look into this issues and attach patches or extensive log.

[Janne <janne-mythtv@…>]

fix handling of inversion

[Janne <janne-mythtv@…>]

The inversion_fix.patch fixes several problems with the inversion parameter.

Inversion is not only an DVB-T parameter (insert_dtv_multiplex() part). Defaults inversion to auto for multiplex from *DeliverySystemDescriptor?. exists_dtv_multiplex() returns true if multiplexes differ only in inversion.

This fixes also the double inserts for DVB-T.

The broken channel names are fixed by the sdt.patch.

[danielk]

(In [9539]) References #1485. Checks each param to dtv_multiplex get/insert for validity before use.

This should fix some issues with uninitialized values when inserting a new multiplex.

This also fixes a problem with the CreateMultiplex?() functions created by [9537]. Because we did not always insert a multiplex when requested in insert_dtv_multiplex() the function that got the multiplex to return from CreateMultiplex?() returned the wrong multiplex. We now return the multiplex from insert_dtv_multiplex so that CreateMultiplex?() has something valid to return.

Finally, mythfilldatabase.pro did not have the libmythtv in it's depend path so changes to channelutil.h caused linking errors, this has been fixed.

[Janne <janne-mythtv@…>]

falls back to auto inversion since 9539 insert nothing into dtv_multiplex.inversion

[Janne <janne-mythtv@…>]

signed char gives the number, casting to char

[Janne <janne-mythtv@…>]

fallback_auto_inversion.patch and signed_char_fix.patch solve together with r9539 the same issue as fix_inversion.patch

[danielk]

(In [9545]) References #1485, applies signed char patch from janne.

For the comparisons to work on a PPC Mac the chars must be signed, but they must be cast back to the native "char" signedness for QString to print it as a character rather than as a small number.

[danielk]

(In [9546]) References #1485. Applies Janne's fallback to auto inversion patch.

Instead of failing when we can't parse the inversion parameter to the DVBTuning::parse methods we set the inversion to 'a'uto and print a warning message. My understanding is that using automatic inversion detection works on all DVB cards but just slows down the tuning a bit.

[anonymous]

On my system I am still having sdt problems [9624]. Channel names are not allocated if the SDT is seen before the NIT.

HasCachedSDT is being called before the NIT has been seen/cached. Resulting in the _cached_nit.empty returning true, and hence giving the HasCachedSDT a false value, even though the SDT has actually been cached.

Changing the 'if (_cached_nit.empty)' statement to 'return true;', allows my sytem to tune properly, with all the correct channel names. However I presume this is not the correct solution?

[danielk]

changeset [9635] referenced this ticket, it added a call to HandleDVBDBInsertion in HandleNIT, so that NIT and SDT arrival order does not matter.

[Stuart Auchterlonie]

additional verbose NIT printing

[Stuart Auchterlonie]

additional verbose printing - dvbtables

[Stuart Auchterlonie]

correct other frequency handling - siparser

[Stuart Auchterlonie]

correct other frequency handling - channelutil

[Stuart Auchterlonie]

correct dtv_multiplex handling in channelutil

[Stuart Auchterlonie]

channelutil - create multiplexes from NIT

[Stuart Auchterlonie]

scanning fixes, utilizing all the other fixes

[Stuart Auchterlonie]

A whole series of patches to fix scanning on DVB. Series applies as follows

1. verbose-print-nit.diff
2. dvbtable-debug-printing.diff
3. siparser-other-frequency-handling.diff
4. channelutil-other-freq-handling.diff
5. channelutil_dtv_multiplex-updates.diff
6. channelutil-create-mplexes-fromNIT.diff
7. dvb_scanning_fixes.diff

1 & 2 are extra debugging printing. Finishes off a few methods that were just stubs.

3 & 4 correct the behaviour of siparser & channelutil when the other_frequency descriptor is present in the NIT

5 corrects the handling of multiplexes where the settings of the mplex have been changed.

6 allows the settings from the NIT to propagate into the mplex table without changing the frequency of the mplex. See note 1.

7 makes sure we wait for all the tables that we need to be able to correctly fill in the dtv_multiplex & channel tables.

Note 1. Some information about the other_frequency descriptors. other_frequency descriptors are only found in terrestrial delivery descriptors if the other frequency flag is set. In this case the other_frequency descriptor just specifies an alternative centre frequency and all the other settings are provided in the terrestrial delivery descriptors.

So patch 6 implements the logic, that if the mplex already exists in the dtv_multiplex table, then it was put there when we tuned to that multiplex previously.

This needs testing from someone on a mplex which uses the alternate frequencies. If you are on Moel y Parc (UK) this is you.

[danielk]

(In [9808]) Refs #1485. Adds debugging VERBOSE VB_SIPARSER macros for NIT tables to scan.

[danielk]

(In [9809]) Refs #1485. Implements two toString() stubs to describe Frequency List and Service Descriptors.

[danielk]

slightly modified multiplex update

[danielk]

(In [9813]) Refs #1485. Applies Stuart A's patch for handling DVB-T "other_frequencies".

This is used in places where there are two transmitters for the same set of channels near each other.

[danielk]

modified dvb scanning patch

[danielk]

(In [9822]) Refs #1485. Makes multiplex updates smarter.

This avoids inserting redundant multiplexes by updating existing ones when possible and not requiring an exact match on some of the optional params (i.e. if we has 'auto' before but now know the exact value we don't insert a new multiplex, we just update the existing one.

This has been tested with DVB, ATSC & raw MPEG transports.

[danielk]

(In [9824]) Refs #1485. Tests for SDTs in addition to NITs before falling back to PMT channel insertion and also avoids adding NIT entry's PID in PAT as a PMT PID.

The second part is from Stuart's dvb_scanning patch, which hasn't been completely applied yet..

[danielk]

(In [9839]) Refs #1485. This increases the timeout when scanning for DVB tables.

The NIT table is only required to be sent once every 10 seconds. I've added 22 seconds to the time we spend on each transport when scanning using the hardware used in Europe and Australia. This gives us two opportunities to see any single NIT table.

I've also added a ModulationDB() method to the transport scan item, so that the correct modulation can be inserted for multiplexes when scanning using the HDTV or HDHomeRun recorders and MythTV is not compiled with DVB driver support. Before, we always inserted '8vsb', but the HDHomeRun device should support QAM soon.

[danielk]

(In [9842]) Refs #1485. Removes old frequency_list insertion code (it was already commented out).

This is now handled from SIScan where we know about some additional important params, like the modulation.

[skamithi]

frequencytables.cpp fails to compile if you don't use DVB. attached is the output i get.

from frequencytables.h
=========================
// MythTV includes
#
#ifdef USING_DVB
#include "dvbchannel.h"
#include "dvbtypes.h"
#else // if ! USING_DVB
#define QAM_64   3
#define QAM_128  4
#define QAM_256  5
#define QAM_AUTO 6
#define VSB_8    7
#define VSB_16   8
#endif // USING_DVB
====
[skarundi@mwanzo1 libmythtv]$ make
g++ -c -pipe -march=pentium4 -Wall -W -g -I/usr/include/freetype2 -D_REENTRANT -DPIC -fPIC  -DMMX -Di386 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -DPREFIX=\"/usr\" -DLIBDIR=\"/usr/lib\" -D_LARGEFILE_SOURCE -DUSING_OSS -DUSING_OPENGL_VSYNC -DUSING_XV -DUSING_FRONTEND -DUSING_V4L -DUSING_DBOX2 -DUSING_HDHOMERUN -DUSING_IVTV -DUSING_BACKEND -DQT_THREAD_SUPPORT -DQT_SHARED -DQT_NO_DEBUG -I/usr/lib/qt3/mkspecs/default -I. -I/usr/include -I/usr/X11R6/include -I../.. -I.. -I../libmyth -I../libavcodec -I../libavutil -I../libmythmpeg2 -Idvbdev -Impeg -I/usr/lib/qt3//include -I/usr/X11R6/include -I/usr/X11R6/include -o frequencytables.o frequencytables.cpp
/usr/lib/qt3//include/qsqldatabase.h:63: warning: 'class QSqlDriverCreatorBase' has virtual functions but non-virtual destructor
frequencytables.cpp: In member function 'QString TransportScanItem::ModulationDB() const':
frequencytables.cpp:140: error: 'QPSK' was not declared in this scope
frequencytables.cpp:142: error: 'QAM_16' was not declared in this scope
frequencytables.cpp:143: error: 'QAM_32' was not declared in this scope
make: *** [frequencytables.o] Error 1
[skarundi@mwanzo1 libmythtv]$

[danielk]

(In [9843]) Refs #1485. Temporary fix for compile issue reported by skamithi.

There is a namespace conflict with the CARD_TYPES enumeration which stymies a proper fix.

I'll fix this properly before I commit the 8PSK support in #1330.

[einarb@…]

Output from dvbsnoop

[einarb@…]

Output from mythtv-setup --verbose siparse,channel,record

[Stuart Auchterlonie]

add's VERBOSE LOC's to channel util

[Stuart Auchterlonie]

add's current tsid & netid to the createmultiplex calls.

[Stuart Auchterlonie]

add's routines to getcurrent tsid & netid from signalmonitor

[Stuart Auchterlonie]

use's the centre freq from the NIT if there are no other_freq

[Stuart Auchterlonie]

add a series of patches to improve scanning.

1) channelutil-verbose-loc.diff
2) head_signalmonitor-support-current-tsid-n-netid.diff
3) add-tsid-n-nid-to-createmultiplexes-v3.diff
4) use-centre-freq.diff

1) just adds the LOC macro to channelutil. simplies sharing 2 between head and mythtv-eit branch
2) adds support to the signalmonitor for retrieving the tsid & networkid of the currently tuned mux.
3) uses 2 to add the current tsid & networkid to the CreateMultiplex? calls. this allows CreateMultiplex? to identify correctly which transport it is tuned to and be able to locate it in dtv_multiplex table.
4) make sure we use the centre freq if there are no other_freq described by the NIT (also corrects uint/int on the tmp variable mplex)

[Stuart Auchterlonie]

replaces use-centre-freq.diff, fixes other_freq handling

[Stuart Auchterlonie]

replaces use-centre-freq.diff, corrects other_freq handling

[Stuart Auchterlonie]

Added use-correct-frequency.diff, replaces use-centre-freq.diff. This corrects the handling of other_freq's when the NIT is inserted into the database.

TODO:

• Work out why HandleDVBDBInsertion is continuing when only the NIT has been seen

• Fix tuned scan so that it scans all the mplexes it finds from the NIT.

[Stuart Auchterlonie]

fixes compile error in previous use-correct-frequency patch

[danielk]

(In [9963]) Refs #1485. Just adds LOC macro to channelutils, used by later #1485 patches.

[danielk]

(In [9964]) Refs #1485. Adds the network id and transport id to a couple createmultiplex calls.

This allows us to find these multiplexes later and know if the are the right ones.

[danielk]

(In [9966]) Fixes #1485. Refs #1704. Adds an MPEG1/MPEG2/MPEG4-AVC IP network recorder.

This is another contribution from the Queen of Canada and the good folks at CRC.

There are instructions in crcipnetworkrecorder.cpp on how to broadcast your own channel using vlc.

The MPEG4-AVC keyframe detection code is in DTVRecorder so it could be used by the DVBRecorder to generate a correct keyframe map for the UK HDTV trial streams.

[danielk]

Commit [9966] should have referenced #1787, and this ticket was closed by commit [9965].