Opened 7 years ago

Closed 6 years ago

Last modified 6 years ago

#9801 closed Bug Report - General (fixed)

mythtranscode produces unusable output when transcoding mpeg4->mpeg4 with cutlists

Reported by: mythtv@… Owned by: beirdo
Priority: minor Milestone:
Component: MythTV - Mythtranscode Version: 0.24-fixes
Severity: medium Keywords: mythtranscode cutlist
Cc: Ticket locked: no

Description (last modified by robertm)

I am having the following problem with the latest 0.24-fixes version of mythtranscode:

  1. Transcoding mpeg2 to mpeg4 with or without cutlist works fine
  2. (Re)Transcoding mpeg4 to mpeg4 WITHOUT cutlist works fine
  3. (Re)Transcoding mpeg4 to mpeg4 WITH cutlist terminates (without relevant error message) after a few seconds with an output file of about 100-150KB independent of the actual cutlist. If mythfrontend is used then this useless, truncated recording replaces and TRASHES the original recording.

This seems to occur regardless of the actual mpeg4->mpeg4 recording profile parameters and occurs even when doing a 'lossless' version (note in all cases I use mp3 for audio).

Here is an example running mythtranscode from the CLI:

#mythtranscode --chanid 1021 --starttime 20101120163000 --outfile 1021_20101120 163000.nuv-OUT2 --honorcutlist --profile 3

(where profile 3 = High Quality mpeg4/mp3)

Gives the following output:

2011-05-23 21:52:57.355 Using runtime prefix = /usr

2011-05-23 21:52:57.355 Using configuration directory = /home/use/.mythtv

2011-05-23 21:52:57.372 Empty LocalHostName?.

2011-05-23 21:52:57.426 New DB connection, total: 1

2011-05-23 21:52:57.447 Closing DB connection named 'DBManager0'

2011-05-23 21:52:57.495 Enabled verbose msgs: important

2011-05-23 21:52:57.787 Found video height of 1088. This is unusual and more than likely the video is actually 1080 so mythtranscode will treat it as such.

2011-05-23 21:52:57.788 New DB connection, total: 2

2011-05-23 21:52:57.816 New DB connection, total: 3

2011-05-23 21:52:57.849 Using protocol version 63

Error in my_thread_global_end(): 1 threads didn't exit

Note that the last error line is a MySQL error that at least in other past reported mythtv contexts may be benign and I don't believe it has anything to do with this bug report.

The full verbose output from adding '-v all' is too long to post so I have posted it to

Attachments (1)

transcode.log (68.2 KB) - added by sphery 7 years ago.
transcode log file, from pastebin, so it's available after pastebin expiration

Download all attachments as: .zip

Change History (24)

comment:1 Changed 7 years ago by robertm

Priority: criticalminor
Severity: highmedium
Summary: mythtranscode TRASHES recordings when transcoding mpeg4->mpeg4 with cutlistsmythtranscode produces unusable output when transcoding mpeg4->mpeg4 with cutlists

Changing ticket title to something less frantic. Also, please see the ticket howto (as you were prompted to read when opening the ticket. Severity and priority are for developer use.

Please attach your log as a file, not as plain text or a pastebin.

comment:2 Changed 7 years ago by robertm

Component: MythTV - GeneralMythTV - Mythtranscode

comment:3 Changed 7 years ago by robertm

Description: modified (diff)

Removing opinion/unsolicited development advice.

comment:4 Changed 7 years ago by mythtv@…

The problem is not that the output is "unusable" or even that there is no output -- if that were the case, then it would be just a case of trying it again or ignoring the output.

The problem is that mythtranscode when used via the default mythfrontend settings overwrites and thus TRASHES AKA DESTROYS AKA RUINS AKA DELETES your original usable recording. Mythfrontend thinks that the transcoding has worked properly and merrily goes on to erase the good original and replace it with a worthless truncated version. Even worse, there is no indication that something went wrong so one could go on for months unwittingly destroying all your precious recordings.

Anybody who does development professional recognizes the difference between a program that DESTROYS original data vs. a program that merely creates UNUSABLE output. This all happens without any warning so months could go by before an innocent user realizes that all his/her recordings have been destroyed.

Feel free to change the title to misrepresent the gravity of a data-destroying bug or to change the severity to something not reflective of such irreversible data deletion. The point is that this bug is by any honest evaluation much more serious than anything that simply makes a random feature flakey, ugly or even unusable since this bug introduces irreversible damage to potentially priceless recordings.

Please explain how such a bug in a core feature can be priority=minor and severity=medium. I truly challenge you to find a bug that is more *severe* than this (though there are doubtless equally severe bugs that prevent or destroy recordings). If you don't want to fix the bug, then at least disable the feature or insert a warning or change the default so as not to overwrite originals so that recordings aren't inadvertently destroyed (though apparently such suggestions are considered to be opinion/unsolicited development advice).

comment:5 Changed 7 years ago by robertm

Ticket locked: set

Calm down. Most users don't use nuv to nuv transcoding, and I daresay none of the developers do. If you'd like to have a rational discussion we'd be happy to address your concerns on the users mailing list. Until then, this ticket is locked and we will address it when it is both convenient and fun, as befits a hobby. We are not slaves. You have been told how to avoid having mythtranscode delete the old recording on the mailing list.

comment:6 Changed 7 years ago by sphery

Likely a dup of #8864

Changed 7 years ago by sphery

Attachment: transcode.log added

transcode log file, from pastebin, so it's available after pastebin expiration

comment:7 Changed 7 years ago by sphery

Status: newinfoneeded_new

What distro is this? There's an issue that's been observed on some distros (specifically some versions of *buntu) where the exit status is not properly reported to applications, so MythTV doesn't realize that an application died--which would explain why it continued without realizing there was a failure.

comment:8 Changed 7 years ago by JYA

Ticket locked: unset

comment:9 Changed 7 years ago by mythtv@…

I am running using the atrpms version (0.24-272) of MythTV under Fedora.

The attached log above shows all the verbose log information and should capture any of the messaged errors (though there don't seem to be any related to this bug).

Note that the behavior can be replicated by running mythtranscode directly from the CLI (see the original post). I did not however check the shell exit code at the time. If that would be helpful, I could run it again and check it.

Regarding your mention of MythTV checking exit codes before overwriting the original, one suggestion for future versions of the code may be to also sanity check the file size to avoid obvious destructive overwriting of original recordings. For example, if the output size is less than a few megabytes then either don't overwrite (and signal an error) or overwrite but rename original rather than deleting it.

Note such an added sanity check would have caught this bug where the output files are 100-150KB. It also would have caught rare unexplained cases I have observed in the past (unrelated to this bug) where the output file is zero length.

comment:10 Changed 7 years ago by Raymond Wagner

Status: infoneeded_newnew

comment:11 Changed 7 years ago by sphery

Resolution: Duplicate
Status: newclosed

There's a setting, "Save original files after transcoding (globally)", that allows users to choose to keep the old transcoded file in case of problems like this. Anyone who is transcoding MythTV recordings should enable this setting (and clean up after it). The setting is disabled by default so that users don't end up with multiple gigabytes of "orphaned" files--a user who doesn't explicitly enable this setting is unlikely to know that he or she needs to clean up the ".old" files after the transcode is proven successful. Therefore, we won't change the default value of the setting.

Future changes, still in the works, to MythTV will allow us to always keep the old file without it becoming and orphaned file. This will provide the safety net of keeping the original for all users, but will still allow management of files within MythTV. Therefore, I do not want to add in a "hack" that chooses some arbitrary size and refuses to believe a transcode was successful (regardless of what the process's reported exit status was) if the resulting file is smaller than said value. If we do, it will fail to detect some types of failures that result in a partial transcode and there will likely be arguments over what file size constitutes success. So, I'm going to consider that part of the ticket, "Won't Fix" (but the future changes will provide the same benefit--and more--without the disadvantages of the "try to outsmart the OS" approach).

Note, also, if you have any non-default value for the setting (anything other than "" or "mythtranscode"):

Transcoder command The program used to transcode recordings. The default is 'mythtranscode' if this setting is empty.

(including an absolute path to the mythtranscode binary, such as /usr/bin/mythtranscode) it's quite possible that is causing the inability to recognize the failed transcode--especially if the transcode command is a script, which may not provide proper exit status to indicate failures.

And, again, anyone who is transcoding MythTV recordings, please enable the setting, "Save original files after transcoding (globally)".

Therefore, all that remains of this ticket is the failing MPEG4->MPEG4 transcode, which is a dup of #8864.

comment:12 Changed 7 years ago by mythtv@…

Not sure why you classify this as a duplicate of #8864 since the context and symptoms seem quite different -- the only thing in common seems to be an issue with transcoding NUV->NUV.

#8864 says: "When running mythtranscode on a HD recording using the "Low Quality" profile and MPEG4, I have had a number of corrupted recordings. The output file sizes are approximately 1/3 of their normal size for typical recordings. The audio appears fine, however all but a few sporadic frames of video are gone/corrupted."

This bug report is *specific* to cutlists and generates files with *neither* audio nor video -- the entire file length is about 150KB which is nothing like 1/3 the file length.

I suspect that the bugs are different and am not sure why one would think that they are the same except at the highest possible level of there being various issues with mythtranscode when transcoding NUV->NUV.

Again to be clear, #8864 is about data corruption/loss with transcoding, this bug #9801 is about cutlists causing transcoding to abort without doing any transcoding -- I have not encountered *any* issues with pure NUV->NUV transcoding with no cutlist.

Please either unmark this as a duplicate or generalize #8864 to include these very different additional symptoms. Otherwise, I fear that one may fix #8864 and have no clue about this issue with cutlists.

comment:13 Changed 7 years ago by mythtv@…

Also, IMO, the description in #8864 is rather vague and seems to imply that the error is sporadic and also doesn't give any clear way to replicate the problem - nor does it give any logging information.

This bug report gives a clear, 100% repeatable way of causing the bug and gives documented logging output. This should facilitate both identifying the cause of the bug and enable a test to confirm that it has been fixed.

comment:14 Changed 6 years ago by bugfixed@…

Well I found the bug and have a simple working fix -- but since the devs persist on maintaining this is merely a duplicate of #8864 (which I can assure everybody it is not) and since the devs seem to think the bug itself is of low priority and not worthy of any attention, I assume that these same devs are probably not interested in my fix. The bug by the way is in the core libmythtv code.

I will be happy to share the bug fix when the devs stop acting so defensively about their precious project and stop trying to hide and minimize the presence and importance of bugs, but instead start acting maturely and helpfully to help users (particularly those who are willing and able to contribute) to resolve critical bug (especially those that destroy recordings or break important existing functionality)

comment:15 Changed 6 years ago by beirdo

If you have a fix, please attach it as a patch. There is no point in pontificating and trying to alienate the developers. We have far too many things on our plates, and some tickets may get misfiled. I appreciate that you've taken the time to find a fix. If you can provide the fix, it will be applied.

comment:16 Changed 6 years ago by mythtv@…

OK here is the fix and patch - it's trivial.... (the problem was due to subtracting 1 from frame 0 which resulted in something like frame 232 - 1 due to unsigned long long type of stuff).

This explains why it only happens in transcoding nuv (since the problem is only in the nuppeldecoder). Also, by tracing through the code you can see why it only takes affect when there is a cutlist.

Assuming you accept this patch, this bug can now be finally closed (note that you might want to edit the ERRONEOUS marking of this bug as a duplicate of bug #8864 since as I first noted 4 months ago this bug has absolutely NOTHING to do with that other long outstanding mythtranscoding bug).

--- nuppeldecoder.cpp.jorig     2011-10-28 10:53:21.000000000 -0400
+++ nuppeldecoder.cpp   2011-11-23 23:02:24.288482191 -0500
@@ -1124,7 +1124,7 @@
             else if (frameheader.comptype == 'V')
                 lastKey = frameheader.timecode;
-                framesPlayed = frameheader.timecode - 1;
+                framesPlayed = (frameheader.timecode > 0 ? frameheader.timecode - 1 : 0);

                 if (!hasFullPositionMap)

comment:17 Changed 6 years ago by beirdo

Resolution: Duplicate
Status: closednew

comment:18 Changed 6 years ago by beirdo

Owner: set to beirdo
Status: newaccepted

comment:19 Changed 6 years ago by beirdo

Ignoring the pointless shouting about "ERRONEOUS" closing... Thanks for the patch. We have seen these little gotchas before, and it's not too surprising. I'll get that in shortly.

comment:20 Changed 6 years ago by Github

Milestone: unknown0.25
Resolution: fixed
Status: acceptedclosed

Catch first frame corrupting unsigned count

Fixes #9801

Signed-off-by: Gavin Hurlbut <ghurlbut@…> (cherry picked from commit c4ee599818c8cfd6f8e9b69df3011ca527d39295)

Branch: master Changeset: 03cff208162704985d1dfe36f8a302da4f2b4251

comment:21 Changed 6 years ago by beirdo

Also on fixes/0.24 at c4ee599818c8cfd

comment:22 Changed 6 years ago by beirdo


comment:23 Changed 6 years ago by stuartm

Milestone: 0.24.2

Milestone 0.24.2 deleted

Note: See TracTickets for help on using tickets.