Modify

Opened 5 years ago

Closed 4 months ago

Last modified 4 months ago

#9480 closed Patch - Bug Fix (fixed)

National characters in polish EPG (Cyfra+ and Polsat) won't work :(

Reported by: marcin.jachurski@… Owned by: dekarl
Priority: minor Milestone: 0.28
Component: MythTV - EIT Version: 0.24
Severity: medium Keywords:
Cc: Ticket locked: no

Description

Polish (Hotbird) DVB-S operators give EPG encoded in the ISO-8859-2 (Polsat) and in ISO-6937 (Cyfra+). Unfortunately does not give information about encoding. Polsat is not suitable byte identifying coding. Cyfra+ gives the wrong marker.On the programs broadcast together we get different coding, both wrong.

MythTV allows to specify the encoding, but only for the transponder, which is transmitted program (in eithelper.cpp). In this case it is necessary to specify the encoding to the network or the transponder from which the EPG has been received.

I wrote an patch, which adds forcing iso-8859-2 and iso-6937, depending on the source transport and/or network id, not on TSID() and OriginalNetworkID() from existing DVBEventInformationTable.

There is someone who had similar problems and have any comments? Maybe I am wrong to do?

This new encodings: ISO-8859-2 for source network id 113 ISO-6937 for source network id 318

Also other channels from Hotbird:

TV POLONIA PULS TVS ITV TELE5 TVP Kultura TV Biznes POLONIA1 EDUSAT Mango 24 CSBTV TVR Rebel TV 4fun.TV Eska TV VIVA Polska. Czworka

Attachments (5)

eit_pl.110307.diff (7.9 KB) - added by voyo <voyo.mythtv@…> 5 years ago.
0002-ticket9480_pl_eit_encodings.patch (7.6 KB) - added by warpme@… 17 months ago.
Patch for current master
0002-ticket9480_pl_eit_encodings.2.patch (7.7 KB) - added by warpme@… 4 months ago.
Patch updated for current master
polsat2hd-decoded-v2.txt (6.7 KB) - added by warpme@… 4 months ago.
PolsatHD dvbsnoop log
tvnhd-decoded-v2.txt (5.5 KB) - added by warpme@… 4 months ago.
TVNHD dvbsnoop log

Download all attachments as: .zip

Change History (38)

comment:1 Changed 5 years ago by marcin.jachurski@…

comment:2 Changed 5 years ago by stuarta

  • Resolution set to Invalid
  • Status changed from new to closed

I'm still waiting for the patch mentioned in the email to be posted. Until then, i'm closing the ticket...

comment:3 Changed 5 years ago by marcin.jachurski@…

  • Resolution Invalid deleted
  • Status changed from closed to new

But it was at Mon Dec 29 11:44:02 UTC 2008 and no one fix the problem until 0.24 version. If you need screenshots or database backup or any log. Please let me know. Polish people are frustrated at polish forums. And said that "We don't have full support for polish language :( "

Please fix it in the future version of MYTH TV. I am happy with that software so that's why i am asking you.

Good solution will be create field in channel table in database with EPG encoding from the air. After that all problems with any encodings in EPG will be solved.

THX.

comment:4 Changed 5 years ago by robertm

  • Resolution set to Invalid
  • Status changed from new to closed

Please don't re-open the ticket without a patch.

Changed 5 years ago by voyo <voyo.mythtv@…>

comment:5 Changed 5 years ago by voyo <voyo.mythtv@…>

Im attaching a patch originally wrote by Wagabunda, this is old thread and description: http://www.mythtv.org/pipermail/mythtv-dev/2008-December/064048.html I only fixed this to patch cleanly against 0.24-fixes. This is not best possible solution because it *might* break on other satellites, its using some probably reserved bits. However works great for me.

comment:6 Changed 5 years ago by voyo <voyo.mythtv@…>

  • Resolution Invalid deleted
  • Status changed from closed to new

comment:7 Changed 5 years ago by warpme@…

Hi, It is great to have addressed PL chars on polish sat EIT. This patch works great for me with passive scan (2 weeks of tests). However with active scan I have quite frequent random errors with PL chars in prog titles or desc. I'm almost sure it is correlated with active scan as last 2 weeks I was on passive mode without any issues. Last evening I turned on active scan and now I have a lot of progs. with bad pl chars. I'm on 0.24-fixes 20110415. I'll be more that happy to help debug problem.

comment:8 Changed 4 years ago by stuartm

  • Milestone changed from unknown to 0.25
  • Type changed from Bug Report to Patch - Bug Fix

Should be simple, patch attached to do encoding fixups for Polish EIT.

comment:9 Changed 4 years ago by warpme@…

Stuart, I can't find patch You re referring to. Do I miss something ?

comment:10 Changed 4 years ago by stuarta

  • Milestone changed from 0.25 to 0.26

comment:11 Changed 4 years ago by kenni

  • Milestone changed from 0.26 to 0.26.1

comment:12 Changed 3 years ago by paulh

  • Milestone changed from 0.26.1 to 0.27

There is a patch attached so hopefully someone can look at this for 0.27

comment:13 Changed 3 years ago by stuartm

  • Owner changed from stuarta to dekarl
  • Status changed from new to assigned

comment:14 Changed 3 years ago by dekarl

  • Milestone changed from 0.27 to 0.27.1

comment:15 Changed 17 months ago by warpme@…

Updated patch for current master.

Changed 17 months ago by warpme@…

Patch for current master

comment:16 Changed 16 months ago by dekarl

  • Milestone changed from 0.27.2 to 0.27.5

comment:17 Changed 12 months ago by stuarta

  • Milestone changed from 0.27.5 to 0.27.6

comment:18 Changed 6 months ago by dekarl

Sorry for taking so long to get back to this. If I remember correctly I posted a patch that moved the bit from 1<<15 to 1LL<<48 because service_id is a 16bits value. But I vaguely remember negative test results for that change.

So lets move forward. Can someone provide samples of the actual sections transmitted for both values of original_network_id?

Basically one DVB-EIT section captured with dvbsnoop including the hexdump of the section. (So I can add a unit test for it) Bonus points if it is two copies of the same event with different sources / encodings, so I can add a test that verifies both are equal.

comment:19 Changed 4 months ago by warpme@…

dekarl, may You pls advice how to quickly gather data You need? Also - will You include patch to official code?

comment:20 Changed 4 months ago by dekarl

Basically the output of "dvbsnoop -n 1 0x12" including the hexdump while tuned to one or the other transport. The tricky part may be capturing a nice small sample of both variants where the encoding does make a difference.

comment:21 Changed 4 months ago by dekarl

Oh, if your patch still works when you change from 1<<15 to 1LL<<48, that would be a good hint, too. (Might as well just apply it then)

comment:22 follow-up: Changed 4 months ago by warpme@…

@dekarl, I'm attaching updated patch for current master. I tested it with 24h continuous active scan over 17 various mplexes and in my opinion it works OK. I have Q however: recently major SAT platform in Poland introduced "improvement" in EIT EPG. They moved episode num. to prog.title. Example: was "Pielęgniarki". Now is "Pielęgniarki 3 (odc. 38)". Is it possible to add in EIT fixups parsing in string title thing like "(odc. XX)" and add "xx" to episode num? Ideally fixup should:

1.find "(odc. XX)" in title

2.copy "xx" to episode num.

3.remove "(odc. XX)" from title

4.remove all trailing spaces from title

If we will have such mfixup, EIT in Polish sat will be v.good (in sense proper following episodes & duplicates when mythtv is asked to automatically record whole series).

Changed 4 months ago by warpme@…

Patch updated for current master

comment:23 in reply to: ↑ 22 Changed 4 months ago by dekarl

Replying to warpme@…:

I'm attaching updated patch for current master. I tested it with 24h continuous active scan over 17 various mplexes and in my opinion it works OK.

I'll see how I can manually merge that, so its easier to test/review.

Example: was "Pielęgniarki". Now is "Pielęgniarki 3 (odc. 38)". Is it possible to add in EIT fixups parsing in string title thing like "(odc. XX)" and add "xx" to episode num?

For this I really need a real sample (output of dvbsnoop -n 1 0x12) and the information where all the pieces go. e.g. in that sample, what is " 3" and what is " (odc. 38)"? Is that episode 3 of 38? Or is that "episode 38" of "season 3"? (maybe with the episode as absolute number instead of the number in the third season).

I have seen all kinds of variants in real broadcasters data, so I really need help to identify the variant that is used to come up with a good conversion. For your example it appears to be "Pielęgniarki 3" "Episode 38" or "Pielęgniarki" "Season 3" "Episode 38". http://www.filmweb.pl/serial/Piel%C4%99gniarki-2014-713378/episodes#

Please open a new ticket with

  • the samples from "dvbsnoop -n 0x12", including details about how it should be fixed up
  • the information where the fixup needs to be applied

comment:24 Changed 4 months ago by warpme@…

@dekarl, I added #12614 for EPG fixups.

comment:25 Changed 4 months ago by Karl Dietz <dekarl@…>

  • Resolution set to fixed
  • Status changed from assigned to closed

In c4fe5210fa6db3879e0d941b50826c9b82af646a/mythtv:

extend character encoding fixups for Cyfra+ and Polsat

This extends the framework to allow applying fixups based on the
transport that the data is carried on, instead of the transport that the
data is about.
It is now possible to remove or replace character encoding signalling.
The old code would only add, but not replace the signalling.

Based on a patch by Wagabunda http://lists.mythtv.org/pipermail/mythtv-dev/2008-December/064048.html
Updated by Voyo and Piotr Oniszczuk

Fixes #9480

comment:26 Changed 4 months ago by dekarl

  • Milestone changed from 0.27.6 to 0.28

Please review and test, as this is a generic rewrite of the submitted patch.

I don't think this can go into fixes/0.27 anytime soon because it changes the existing encoding fixups when the transmitted string already has an encoding signaling. (Now the fixups replace the encoding signaling instead of doing nothing)

comment:27 Changed 4 months ago by warpme@…

It looks something is wrong. EPG on all channels has wrong PL chars. dvbsnoop -n 1 0x12 from TVN & Polsat attached.

Changed 4 months ago by warpme@…

PolsatHD dvbsnoop log

Changed 4 months ago by warpme@…

TVNHD dvbsnoop log

comment:28 Changed 4 months ago by Karl Dietz <dekarl@…>

In 770f9ebbb23d171f7c7a01b567c4171c564987f1/mythtv:

extend unit tests to catch issue with c4fe5210fa

Sadly all tests pass, so the cause is still unclear.

Refs #9480
Refs #12618

comment:29 Changed 4 months ago by warpme@…

dekarl, if tests are OK then it think problem isn't in implementation but rather lays in design. As Your's commit is not only 1:1 backport of original ticket but it is also re-desing of it, I think maybe it is worth to carefully compare original ticket working logic with Your's implementation logic. There must be difference as original patch works well.

comment:30 Changed 4 months ago by Karl Dietz <dekarl@…>

In 67f13feb41e1928718eb3461634a8ee366b49449/mythtv:

Revert "remove unneeded casts"

Refs #9480
Refs #12618

This reverts commit a11d8cb3c74c0340d4d76dbf34d8ccb98075b4e4.

comment:31 Changed 4 months ago by Karl Dietz <dekarl@…>

In e60471d5ef3008882fd65cf694b2f09fe706e4e1/mythtv:

Revert "extend unit tests to catch issue with c4fe5210fa"

Refs #9480
Refs #12618

This reverts commit 770f9ebbb23d171f7c7a01b567c4171c564987f1.

comment:32 Changed 4 months ago by Karl Dietz <dekarl@…>

In 05159b0a12238e8f4ed525ae50943a8eb2ba6355/mythtv:

Revert "extend character encoding fixups for Cyfra+ and Polsat"

Refs #9480
Refs #12618

This reverts commit c4fe5210fa6db3879e0d941b50826c9b82af646a.

comment:33 Changed 4 months ago by dekarl

I'm moving work to a branch at https://github.com/MythTV/mythtv/tree/devel/eit-encoding-fixes so master is now back in the state that it was before I started working on this.

Add Comment

Modify Ticket

Action
as closed The owner will remain dekarl.
The resolution will be deleted. Next status will be 'new'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.