Modify
Warning Please read the Ticket HowTo before creating or commenting on a ticket. Failure to do so may cause your ticket to be rejected or result in a slower response.

Opened 4 years ago

Closed 3 years ago

Last modified 3 years ago

#8707 closed defect (Fixed)

ASCII NULL (\0) in metadata can cause Live TV to fail and recordings to disappear from mythfrontend

Reported by: glemsom@… Owned by: stuarta
Priority: major Milestone: 0.23-fixes
Component: MythTV - EIT Version: 0.23-fixes
Severity: medium Keywords: EPG eit \0
Cc: Ticket locked: no

Description

The EPG might fail if cells in the "program" table contains '\0'.

If the "program" table contains '\0', this might happen:

1: EPG in liveTV is unable to shuffle between channels - when it hits a channel containing \0 in subtitle or description - it will exit with "irrecoverable recorder error"

2: "Manage Recordings" displays a question-mark, if it hits \0.

Example to determine if your database contains \0 in subtitle, in the "program" table:

echo -e "USE mythconverg\nSELECT chanid,title,subtitle FROM program WHERE subtitle LIKE '\\\0';" | /usr/bin/mysql -umythtv -p

The \0 can be removed like:

echo -e "USE mythconverg\nUPDATE program SET subtitle = 'none' WHERE subtitle LIKE '\\\0';" | /usr/bin/mysql -umythtv -p

In my case, my database gets filled with '\0' when using EIT for EPG.
Maybe the EIT scanner is actually the problem here?

This is tested using mythtv-0.23-fixes@25423

Attachments (1)

eit_strip_null_chars.diff (911 bytes) - added by kenni 3 years ago.

Download all attachments as: .zip

Change History (16)

comment:1 Changed 4 years ago by robertm

  • Component changed from MythTV - General to MythTV - EIT
  • Owner set to stuarta

Changing component to EIT. Guess we should probably account for this.

comment:2 Changed 4 years ago by robertm

  • Status changed from new to assigned

comment:3 Changed 3 years ago by stuartm

  • Milestone changed from unknown to 0.24
  • Owner changed from stuarta to janne

comment:4 Changed 3 years ago by stuartm

  • Milestone changed from 0.24 to 0.24.1

comment:5 Changed 3 years ago by kenni

  • Owner changed from janne to stuarta
  • Summary changed from EPG can fail if database contains \0 to ASCII NULL (\0) in metadata can cause Live TV to fail and recordings to disappear from mythfrontend

Besides failing Live TV; If a program containing the ASCII NUL character in its metadata gets recorded, the list of recordings on the frontend can result in "No recordings found" without any obvious error messages. Workaround:

UPDATE recorded SET title=REPLACE(title, '\0', '');
UPDATE recorded SET subtitle=REPLACE(subtitle, '\0', '');
UPDATE recorded SET description=REPLACE(description, '\0', '');

These chars should be stripped out from EIT before we pass them to the DB.

comment:6 Changed 3 years ago by waisted@…

Im using EIT in Denmark. The above-mentioned symptoms happened to me just last week (early april). I guess the EIT-format in DK has changed (at least for my TV-provider ("Yousee Cable")).
Apart from "No recordings found" in Watch Recordings and "irrecoverable recorder error" during Live TV I also experience buggy behaviour when trying to schedule recordings in EPG. Pressing R does nothing visual but exiting the EPG and reentering the EPG suddenly shows the program scheduled for recording. I have to exit and reenter the EPG for every schedule-change I make but only the first attempt will become "green" in the EPG indicating a scheduled recording. It is perhaps notable that this is not a problem when using MythWeb.

Might I suggest this workaround to users like myself, until the stripping of "bad chars" from the EIT is implemented.

For your mythconverg-DB inside the mysql-prompt do this:

delimiter //
CREATE TRIGGER no_zeros_rec BEFORE UPDATE ON recorded
FOR EACH ROW
BEGIN
 IF NEW.title ='\0' THEN SET NEW.title = 'none';
 ELSEIF NEW.subtitle ='\0' THEN SET NEW.subtitle = 'none';
 ELSEIF NEW.description ='\0' THEN SET NEW.description = 'none';
 END IF;
END;//

CREATE TRIGGER no_zeros_pro BEFORE UPDATE ON program
FOR EACH ROW
BEGIN
 ELSEIF NEW.subtitle ='\0' THEN SET NEW.subtitle = 'none';
 ELSEIF NEW.description ='\0' THEN SET NEW.description = 'none';
 END IF;
END;//
delimiter ;

I've only had trouble with "subtitle"-values when it comes to the program-table, but I figure that "description" might be prone to the same \0 as subtitle from the EIT-data.

In my opinion this is somewhat severe as it rendered my box useless as a recorder unless going to a PC and using mythweb. I could not see previous recordings and could not schedule new recordings leaving my box a crippled music/video-player.

Please let me know if I can help debugging in any way!

Changed 3 years ago by kenni

comment:7 Changed 3 years ago by kenni

waisted, please test if the attached patch solves the issue and report back. The patch will not clean out existing NULL-chars in the database, so you'll need to remove these manually. If the patch works as expected, no NULL-chars will get inserted into program table. To ensure a clean test, please truncate your program table before running EIT with the patch. I don't have access to Yousee DVB-C, so I can't test this myself.

comment:8 Changed 3 years ago by kenni

  • Status changed from assigned to infoneeded

comment:9 Changed 3 years ago by Github

Filter out NULL-characters from EIT before we save the data in the database.

Refs #8707.

Changeset: 43bef4efbe3214d3e4c7eb14d2098d0e38fd89e4

comment:10 Changed 3 years ago by Github

Filter out NULL-characters from EIT before we save the data in the database.

Refs #8707.

Changeset: c1c7af0900b357c2e2b811c6affd31fb8a0600ca

comment:11 Changed 3 years ago by Github

Filter out NULL-characters from EIT before we save the data in the database.

Refs #8707.

Changeset: 6802d58a6d4b6dddb51d80995e058e1695605bad

comment:12 Changed 3 years ago by kenni

  • Milestone changed from 0.24.1 to 0.23-fixes
  • Resolution set to Fixed
  • Status changed from infoneeded to closed

Hopefully fixed in 0.23-fixes, 0.24-fixes and master. The fix does not clean up already broken data in the database, so existing NULL-chars will need to be deleted manually.

Please remember to delete any custom SQL triggers to avoid future database problems when upgrading MythTV.

comment:13 Changed 3 years ago by waisted@…

I have deleted the triggers from DB and I have build MythTV with your patch. I have yet to see whether or not it works (I think it works beautifully). My EPG has lots of (old) data fetched when my db-triggers were active so I'm wondering if theres a way to trigger a re-read from the EIT-handler. Perhaps clearing "programs" and then somehow trigger EIT to fill the database again ?

My apologies if this is out of scope for this bug, I just want to be able to test and verify so that you guys can get the patch validated from "average joe"-me :)

comment:14 Changed 3 years ago by kenni

Yep, as I wrote; truncate your program table - "truncate program;". It will remove your existing program guide data. Startup your backend and your frontend, start live tv, open up the EPG and watch the program data slowly getting populated from EIT. Hopefully the new data should be added to the DB without the NULL characters.

comment:15 Changed 3 years ago by waisted@…

Thankyou for explaining it an extra time and sorry for the overhead in this thread.
Program table is truncated, triggers dropped earlier.
The first 30-40 programs have started to appear in the program-guide during Live TV.
The shows/channels that used to cause trouble when pressing "R" now seems to be responding correctly when trying to schedule in EPG.
In mysql I've tried selecting the program that used to contain \0 in the "subtitle"-field, - it now appears blank.
Succes! :)
In a few hours one of the troublesome programs will be recorded and I'll try opening "Watch Recordings" to verify that all is working well. I'll only report back if it does not work as expected (and this'll be within 4 hours).

Thankyou for all your help and support.

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'new'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.