National characters in polish EPG (Cyfra+ and Polsat) won't work :(

[marcin.jachurski@…]

Polish (Hotbird) DVB-S operators give EPG encoded in the ISO-8859-2 (Polsat) and in ISO-6937 (Cyfra+). Unfortunately does not give information about encoding. Polsat is not suitable byte identifying coding. Cyfra+ gives the wrong marker.On the programs broadcast together we get different coding, both wrong.

MythTV allows to specify the encoding, but only for the transponder, which is transmitted program (in eithelper.cpp). In this case it is necessary to specify the encoding to the network or the transponder from which the EPG has been received.

I wrote an patch, which adds forcing iso-8859-2 and iso-6937, depending on the source transport and/or network id, not on TSID() and OriginalNetworkID() from existing DVBEventInformationTable.

There is someone who had similar problems and have any comments? Maybe I am wrong to do?

This new encodings: ISO-8859-2 for source network id 113 ISO-6937 for source network id 318

Also other channels from Hotbird:

TV POLONIA PULS TVS ITV TELE5 TVP Kultura TV Biznes POLONIA1 EDUSAT Mango 24 CSBTV TVR Rebel TV 4fun.TV Eska TV VIVA Polska. Czworka

[marcin.jachurski@…]

It is quote from ​http://mythtv.org/pipermail/mythtv-dev/2008-December/064048.html

but still isn't fixed.

[Stuart Auchterlonie]

I'm still waiting for the patch mentioned in the email to be posted. Until then, i'm closing the ticket...

[marcin.jachurski@…]

But it was at Mon Dec 29 11:44:02 UTC 2008 and no one fix the problem until 0.24 version. If you need screenshots or database backup or any log. Please let me know. Polish people are frustrated at polish forums. And said that "We don't have full support for polish language :( "

Please fix it in the future version of MYTH TV. I am happy with that software so that's why i am asking you.

Good solution will be create field in channel table in database with EPG encoding from the air. After that all problems with any encodings in EPG will be solved.

THX.

[robertm]

Please don't re-open the ticket without a patch.

[voyo <voyo.mythtv@…>]

Im attaching a patch originally wrote by Wagabunda, this is old thread and description: http://www.mythtv.org/pipermail/mythtv-dev/2008-December/064048.html I only fixed this to patch cleanly against 0.24-fixes. This is not best possible solution because it *might* break on other satellites, its using some probably reserved bits. However works great for me.

[warpme@…]

Hi, It is great to have addressed PL chars on polish sat EIT. This patch works great for me with passive scan (2 weeks of tests). However with active scan I have quite frequent random errors with PL chars in prog titles or desc. I'm almost sure it is correlated with active scan as last 2 weeks I was on passive mode without any issues. Last evening I turned on active scan and now I have a lot of progs. with bad pl chars. I'm on 0.24-fixes 20110415. I'll be more that happy to help debug problem.

[stuartm]

Should be simple, patch attached to do encoding fixups for Polish EIT.

[warpme@…]

Stuart, I can't find patch You re referring to. Do I miss something ?

[paulh]

There is a patch attached so hopefully someone can look at this for 0.27

[warpme@…]

Updated patch for current master.

[warpme@…]

Patch for current master

[Karl Egly]

Sorry for taking so long to get back to this. If I remember correctly I posted a patch that moved the bit from 1<<15 to 1LL<<48 because service_id is a 16bits value. But I vaguely remember negative test results for that change.

So lets move forward. Can someone provide samples of the actual sections transmitted for both values of original_network_id?

Basically one DVB-EIT section captured with dvbsnoop including the hexdump of the section. (So I can add a unit test for it) Bonus points if it is two copies of the same event with different sources / encodings, so I can add a test that verifies both are equal.

[warpme@…]

dekarl, may You pls advice how to quickly gather data You need? Also - will You include patch to official code?

[Karl Egly]

Basically the output of "dvbsnoop -n 1 0x12" including the hexdump while tuned to one or the other transport. The tricky part may be capturing a nice small sample of both variants where the encoding does make a difference.

[Karl Egly]

Oh, if your patch still works when you change from 1<<15 to 1LL<<48, that would be a good hint, too. (Might as well just apply it then)

[warpme@…]

@dekarl, I'm attaching updated patch for current master. I tested it with 24h continuous active scan over 17 various mplexes and in my opinion it works OK. I have Q however: recently major SAT platform in Poland introduced "improvement" in EIT EPG. They moved episode num. to prog.title. Example: was "Pielęgniarki". Now is "Pielęgniarki 3 (odc. 38)". Is it possible to add in EIT fixups parsing in string title thing like "(odc. XX)" and add "xx" to episode num? Ideally fixup should:

1.find "(odc. XX)" in title

2.copy "xx" to episode num.

3.remove "(odc. XX)" from title

4.remove all trailing spaces from title

If we will have such mfixup, EIT in Polish sat will be v.good (in sense proper following episodes & duplicates when mythtv is asked to automatically record whole series).

[warpme@…]

Patch updated for current master

[Karl Egly]

Replying to warpme@…:

I'm attaching updated patch for current master. I tested it with 24h continuous active scan over 17 various mplexes and in my opinion it works OK.

I'll see how I can manually merge that, so its easier to test/review.

Example: was "Pielęgniarki". Now is "Pielęgniarki 3 (odc. 38)". Is it possible to add in EIT fixups parsing in string title thing like "(odc. XX)" and add "xx" to episode num?

For this I really need a real sample (output of dvbsnoop -n 1 0x12) and the information where all the pieces go. e.g. in that sample, what is " 3" and what is " (odc. 38)"? Is that episode 3 of 38? Or is that "episode 38" of "season 3"? (maybe with the episode as absolute number instead of the number in the third season).

I have seen all kinds of variants in real broadcasters data, so I really need help to identify the variant that is used to come up with a good conversion. For your example it appears to be "Pielęgniarki 3" "Episode 38" or "Pielęgniarki" "Season 3" "Episode 38". ​http://www.filmweb.pl/serial/Piel%C4%99gniarki-2014-713378/episodes#

Please open a new ticket with

• the samples from "dvbsnoop -n 0x12", including details about how it should be fixed up
• the information where the fixup needs to be applied

[warpme@…]

@dekarl, I added #12614 for EPG fixups.

[Karl Dietz <dekarl@…>]

In c4fe5210fa6db3879e0d941b50826c9b82af646a/mythtv:

extend character encoding fixups for Cyfra+ and Polsat

This extends the framework to allow applying fixups based on the
transport that the data is carried on, instead of the transport that the
data is about.
It is now possible to remove or replace character encoding signalling.
The old code would only add, but not replace the signalling.

Based on a patch by Wagabunda ​http://lists.mythtv.org/pipermail/mythtv-dev/2008-December/064048.html
Updated by Voyo and Piotr Oniszczuk

Fixes #9480

[Karl Egly]

Please review and test, as this is a generic rewrite of the submitted patch.

I don't think this can go into fixes/0.27 anytime soon because it changes the existing encoding fixups when the transmitted string already has an encoding signaling. (Now the fixups replace the encoding signaling instead of doing nothing)

[warpme@…]

It looks something is wrong. EPG on all channels has wrong PL chars. dvbsnoop -n 1 0x12 from TVN & Polsat attached.

[warpme@…]

PolsatHD dvbsnoop log

[warpme@…]

TVNHD dvbsnoop log

[Karl Dietz <dekarl@…>]

In 770f9ebbb23d171f7c7a01b567c4171c564987f1/mythtv:

extend unit tests to catch issue with c4fe5210fa

Sadly all tests pass, so the cause is still unclear.

Refs #9480
Refs #12618

[warpme@…]

dekarl, if tests are OK then it think problem isn't in implementation but rather lays in design. As Your's commit is not only 1:1 backport of original ticket but it is also re-desing of it, I think maybe it is worth to carefully compare original ticket working logic with Your's implementation logic. There must be difference as original patch works well.

[Karl Dietz <dekarl@…>]

In 67f13feb41e1928718eb3461634a8ee366b49449/mythtv:

Revert "remove unneeded casts"

Refs #9480
Refs #12618

This reverts commit a11d8cb3c74c0340d4d76dbf34d8ccb98075b4e4.

[Karl Dietz <dekarl@…>]

In e60471d5ef3008882fd65cf694b2f09fe706e4e1/mythtv:

Revert "extend unit tests to catch issue with c4fe5210fa"

Refs #9480
Refs #12618

This reverts commit 770f9ebbb23d171f7c7a01b567c4171c564987f1.

[Karl Dietz <dekarl@…>]

In 05159b0a12238e8f4ed525ae50943a8eb2ba6355/mythtv:

Revert "extend character encoding fixups for Cyfra+ and Polsat"

Refs #9480
Refs #12618

This reverts commit c4fe5210fa6db3879e0d941b50826c9b82af646a.

[Karl Egly]

I'm moving work to a branch at ​https://github.com/MythTV/mythtv/tree/devel/eit-encoding-fixes so master is now back in the state that it was before I started working on this.