Opened 14 years ago

Closed 13 years ago

Last modified 13 years ago

#8707 closed defect (Fixed)

ASCII NULL (\0) in metadata can cause Live TV to fail and recordings to disappear from mythfrontend

Reported by: glemsom@… Owned by: Stuart Auchterlonie
Priority: major Milestone: 0.23-fixes
Component: MythTV - EIT Version: 0.23-fixes
Severity: medium Keywords: EPG eit \0
Cc: Ticket locked: no

Description

The EPG might fail if cells in the "program" table contains '\0'.

If the "program" table contains '\0', this might happen:

1: EPG in liveTV is unable to shuffle between channels - when it hits a channel containing \0 in subtitle or description - it will exit with "irrecoverable recorder error"

2: "Manage Recordings" displays a question-mark, if it hits \0.

Example to determine if your database contains \0 in subtitle, in the "program" table:

echo -e "USE mythconverg\nSELECT chanid,title,subtitle FROM program WHERE subtitle LIKE '\\\0';" | /usr/bin/mysql -umythtv -p

The \0 can be removed like:

echo -e "USE mythconverg\nUPDATE program SET subtitle = 'none' WHERE subtitle LIKE '\\\0';" | /usr/bin/mysql -umythtv -p

In my case, my database gets filled with '\0' when using EIT for EPG.
Maybe the EIT scanner is actually the problem here?

This is tested using mythtv-0.23-fixes@25423

Attachments (1)

eit_strip_null_chars.diff (911 bytes) - added by Kenni Lund [kenni a kelu dot dk] 13 years ago.

Download all attachments as: .zip

Change History (16)

comment:1 Changed 14 years ago by robertm

Component: MythTV - GeneralMythTV - EIT
Owner: set to Stuart Auchterlonie

Changing component to EIT. Guess we should probably account for this.

comment:2 Changed 14 years ago by robertm

Status: newassigned

comment:3 Changed 13 years ago by stuartm

Milestone: unknown0.24
Owner: changed from Stuart Auchterlonie to Janne Grunau

comment:4 Changed 13 years ago by stuartm

Milestone: 0.240.24.1

comment:5 Changed 13 years ago by Kenni Lund [kenni a kelu dot dk]

Owner: changed from Janne Grunau to Stuart Auchterlonie
Summary: EPG can fail if database contains \0ASCII NULL (\0) in metadata can cause Live TV to fail and recordings to disappear from mythfrontend

Besides failing Live TV; If a program containing the ASCII NUL character in its metadata gets recorded, the list of recordings on the frontend can result in "No recordings found" without any obvious error messages. Workaround:

UPDATE recorded SET title=REPLACE(title, '\0', '');
UPDATE recorded SET subtitle=REPLACE(subtitle, '\0', '');
UPDATE recorded SET description=REPLACE(description, '\0', '');

These chars should be stripped out from EIT before we pass them to the DB.

comment:6 Changed 13 years ago by waisted@…

Im using EIT in Denmark. The above-mentioned symptoms happened to me just last week (early april). I guess the EIT-format in DK has changed (at least for my TV-provider ("Yousee Cable")). Apart from "No recordings found" in Watch Recordings and "irrecoverable recorder error" during Live TV I also experience buggy behaviour when trying to schedule recordings in EPG. Pressing R does nothing visual but exiting the EPG and reentering the EPG suddenly shows the program scheduled for recording. I have to exit and reenter the EPG for every schedule-change I make but only the first attempt will become "green" in the EPG indicating a scheduled recording. It is perhaps notable that this is not a problem when using MythWeb.

Might I suggest this workaround to users like myself, until the stripping of "bad chars" from the EIT is implemented.

For your mythconverg-DB inside the mysql-prompt do this:

delimiter //
CREATE TRIGGER no_zeros_rec BEFORE UPDATE ON recorded
FOR EACH ROW
BEGIN
 IF NEW.title ='\0' THEN SET NEW.title = 'none';
 ELSEIF NEW.subtitle ='\0' THEN SET NEW.subtitle = 'none';
 ELSEIF NEW.description ='\0' THEN SET NEW.description = 'none';
 END IF;
END;//

CREATE TRIGGER no_zeros_pro BEFORE UPDATE ON program
FOR EACH ROW
BEGIN
 ELSEIF NEW.subtitle ='\0' THEN SET NEW.subtitle = 'none';
 ELSEIF NEW.description ='\0' THEN SET NEW.description = 'none';
 END IF;
END;//
delimiter ;

I've only had trouble with "subtitle"-values when it comes to the program-table, but I figure that "description" might be prone to the same \0 as subtitle from the EIT-data.

In my opinion this is somewhat severe as it rendered my box useless as a recorder unless going to a PC and using mythweb. I could not see previous recordings and could not schedule new recordings leaving my box a crippled music/video-player.

Please let me know if I can help debugging in any way!

Changed 13 years ago by Kenni Lund [kenni a kelu dot dk]

Attachment: eit_strip_null_chars.diff added

comment:7 Changed 13 years ago by Kenni Lund [kenni a kelu dot dk]

waisted, please test if the attached patch solves the issue and report back. The patch will not clean out existing NULL-chars in the database, so you'll need to remove these manually. If the patch works as expected, no NULL-chars will get inserted into program table. To ensure a clean test, please truncate your program table before running EIT with the patch. I don't have access to Yousee DVB-C, so I can't test this myself.

comment:8 Changed 13 years ago by Kenni Lund [kenni a kelu dot dk]

Status: assignedinfoneeded

comment:9 Changed 13 years ago by Github

Filter out NULL-characters from EIT before we save the data in the database.

Refs #8707.

Changeset: 43bef4efbe3214d3e4c7eb14d2098d0e38fd89e4

comment:10 Changed 13 years ago by Github

Filter out NULL-characters from EIT before we save the data in the database.

Refs #8707.

Changeset: c1c7af0900b357c2e2b811c6affd31fb8a0600ca

comment:11 Changed 13 years ago by Github

Filter out NULL-characters from EIT before we save the data in the database.

Refs #8707.

Changeset: 6802d58a6d4b6dddb51d80995e058e1695605bad

comment:12 Changed 13 years ago by Kenni Lund [kenni a kelu dot dk]

Milestone: 0.24.10.23-fixes
Resolution: Fixed
Status: infoneededclosed

Hopefully fixed in 0.23-fixes, 0.24-fixes and master. The fix does not clean up already broken data in the database, so existing NULL-chars will need to be deleted manually.

Please remember to delete any custom SQL triggers to avoid future database problems when upgrading MythTV.

comment:13 Changed 13 years ago by waisted@…

I have deleted the triggers from DB and I have build MythTV with your patch. I have yet to see whether or not it works (I think it works beautifully). My EPG has lots of (old) data fetched when my db-triggers were active so I'm wondering if theres a way to trigger a re-read from the EIT-handler. Perhaps clearing "programs" and then somehow trigger EIT to fill the database again ?

My apologies if this is out of scope for this bug, I just want to be able to test and verify so that you guys can get the patch validated from "average joe"-me :)

comment:14 Changed 13 years ago by Kenni Lund [kenni a kelu dot dk]

Yep, as I wrote; truncate your program table - "truncate program;". It will remove your existing program guide data. Startup your backend and your frontend, start live tv, open up the EPG and watch the program data slowly getting populated from EIT. Hopefully the new data should be added to the DB without the NULL characters.

comment:15 Changed 13 years ago by waisted@…

Thankyou for explaining it an extra time and sorry for the overhead in this thread. Program table is truncated, triggers dropped earlier. The first 30-40 programs have started to appear in the program-guide during Live TV. The shows/channels that used to cause trouble when pressing "R" now seems to be responding correctly when trying to schedule in EPG. In mysql I've tried selecting the program that used to contain \0 in the "subtitle"-field, - it now appears blank. Succes! :) In a few hours one of the troublesome programs will be recorded and I'll try opening "Watch Recordings" to verify that all is working well. I'll only report back if it does not work as expected (and this'll be within 4 hours).

Thankyou for all your help and support.

Note: See TracTickets for help on using tickets.