Opened 18 years ago

Closed 9 years ago

#2077 closed Bug Report - General (Fixed)

transcoding fails with "Deadlock detected. One buffer is full when ..." for some DVB recordings

Reported by: seb Owned by: beirdo
Priority: minor Milestone: 0.26.2
Component: MythTV - Mythtranscode Version: Master Head
Severity: medium Keywords: mythtranscode
Cc: Ticket locked: no

Description

Since the memory leak fixes (ticket 10507) transcoding works very well for most of my recordings but fails on a few with "Deadlock detected. One buffer is full when the other is empty! Aborting". This is lossless transcoding of DVB-T recordings. If sample mpg files are required I will need some advice on how (the originals are around 1.4G). I'm currently running SVN trunk rev 10584.

Attachments (8)

transcode-crash.txt (4.2 KB) - added by seb 18 years ago.
transcode.log (229.0 KB) - added by seb 18 years ago.
mythtranscode -i ../1002_20060703213000.mpg -o foo.mpg -v all --mpeg2 > transcode.log
transcode.log.gz (95.4 KB) - added by rob@… 17 years ago.
mythtranscode -i ./7_20070118003000.mpg -o test.mpg -v all --mpeg2 > transcode.log
adeffs.mythtv-transcode01.tar.gz (185 bytes) - added by adeffs.mythtv@… 17 years ago.
one attempt at a firewire mpeg2 transcode
mt-replexsync-clean.diff (163.9 KB) - added by robert.mcnamara@… 15 years ago.
Initial work on replex sync
mythtranscode-replexsync-trunk-r18536.diff (173.1 KB) - added by robert.mcnamara@… 15 years ago.
makes replex sync functional
mythtranscode-replexsync-trunk-r18755.diff (173.2 KB) - added by robert.mcnamara@… 15 years ago.
Updated version of replex resync with some bug fixes.
mythtransode-ac3.diff (3.1 KB) - added by Nicolas Pöhlmann <nicolas.poehlmann@…> 11 years ago.
AC3 deadlock patch

Download all attachments as: .zip

Change History (66)

Changed 18 years ago by seb

Attachment: transcode-crash.txt added

comment:1 Changed 18 years ago by anonymous

I reverted 10507 to see if it was relevant but it isn't. Memory leak came back but this problem didn't go away.

comment:2 Changed 18 years ago by xris

Owner: changed from cpinkham to Geoffrey Hausheer

Changed 18 years ago by seb

Attachment: transcode.log added

mythtranscode -i ../1002_20060703213000.mpg -o foo.mpg -v all --mpeg2 > transcode.log

comment:3 Changed 17 years ago by Geoffrey Hausheer

Status: newassigned

I'm not usre this can be fixed, to be honest. It appears that at 10.5 MB into the file, the audio on stream #3 disappears. The transcoder can not handle this case. About the only thing I can do is to add a command-line switch which would force the transcoder to only preserve a specific (in this case the first) audio stream. I will work on adding such an option in the mean-time.

If you still have this test case, I'd be happy if you could make the 1st 15MB of it available to me.

comment:4 Changed 17 years ago by laurent.devilliers@…

Is the feature still being worked on to specify an audio stream? It would be very usefull on DVB streams that have multiple languages. When doing a transcode to mpg4 it always selects the first audio stream.

comment:5 Changed 17 years ago by ylee@…

I have recently seen a spate of "deadlock detected"/232 errors when losslessly transcoding many (*not* all) recordings. I am using 0.20-fixes with ATrpms-147. I have a EM64T system, and am pretty sure these started occurring when I moved from a Fedora Core 4-based system running in 32-bit to Fedora Core 6 on 64-bit (but same hardware otherwise).

comment:6 Changed 17 years ago by rob@…

Milestone: 0.20
Version: head0.20

Hi, I'm seeing the same, with 0.20-fixes from the ubuntu archive. As with the previous comments, I only see this error on a small number of UK-based DVB-T channels - mainly BBC 3.

Interesting, I also receive this channel on a DVB-C card, and this error does *not* occur when using this source, so this would appear to be something particular to the DVB-T broadcast.

Making an educated assumption here - Could it be the hard-of-hearing audio description stream that is coming and going? This would make sense, as this would be the only other audio stream this channel ever broadcasts, and bits on the DVB-T multiplexes are highly contented. If so, I'm not sure how you could tell which is which, as they are both tagged as "English".

What about an option to pre-scan the TS and disregard any streams that are not present throughout?

Attached a log for reference. Can also supply an offending TS file if needs be.

Changed 17 years ago by rob@…

Attachment: transcode.log.gz added

mythtranscode -i ./7_20070118003000.mpg -o test.mpg -v all --mpeg2 > transcode.log

comment:7 Changed 17 years ago by anonymous

Is there any progress with this as I am seeing the same problem with a few programs on BBC3?

comment:8 Changed 17 years ago by adeffs.mythtv@…

I'm seeing this too on firewire recordings on InHD and MTVHD. mplayer's identify feature shows only one audio PID for these files (ac3, 2 channel).

I've attached the log output (-v all) for one of my attempts.

Changed 17 years ago by adeffs.mythtv@…

one attempt at a firewire mpeg2 transcode

comment:9 Changed 17 years ago by Janne Grunau

Milestone: 0.20unknown
Version: 0.20head

comment:10 Changed 17 years ago by anonymous

I'm seeing this as well on firewire recordings of MHD.

comment:11 Changed 17 years ago by cbrabandt@…

This ticket is over a year old. Does anyone have a fix or workaround yet? The bug is still present in the current Mythdora 4 distro and every version of SVN I've ever compiled.

I find that mythtranscode completes successfully, if I don't use the -l cutlist option:

[mythtv@mythbackend1 recordings]$ mythtranscode -c 1061 -s "2007-06-14 23:34:00" -o test.mpg --showprogress --mpeg2 2007-07-07 11:48:24.117 Using runtime prefix = /usr 2007-07-07 11:48:24.218 New DB connection, total: 1 2007-07-07 11:48:24.253 Enabled verbose msgs: important 2007-07-07 11:48:24.268 New DB connection, total: 2 Mux rate: 20.28 Mbit/s 2007-07-07 11:48:29.368 1.8% complete ... ... 2007-07-07 11:54:44.362 99.7% complete [mythtv@mythbackend1 recordings]$

However, with a cutlist, it fails:

[mythtv@mythbackend1 recordings]$ mythtranscode -c 1061 -s "2007-06-14 23:34:00" -o test.mpg -l --showprogress --mpeg2 2007-07-07 11:43:46.070 Using runtime prefix = /usr 2007-07-07 11:43:46.114 New DB connection, total: 1 2007-07-07 11:43:46.128 Enabled verbose msgs: important 2007-07-07 11:43:46.197 New DB connection, total: 2 2007-07-07 11:43:51.873 3.1% complete ... ... ... 2007-07-07 11:46:02.206 90.6% complete 2007-07-07 11:46:07.089 Deadlock detected. One buffer is full when

the other is empty! Aborting

The recordings are HD mpeg2 transport streams captured from KWorld 115 and Dvico Fusion cards, and firewire.

Is there anything I can do to help the devs resolve this bug (video samples, etc.)?

comment:12 Changed 17 years ago by paulh

Component: mythtvmythtranscode

comment:13 Changed 16 years ago by anonymous

This is the same bug as 3274. I run into this bug on *all* transcodes on material captured via firewire. I have added a message to ticket 3274, and provided two short mpeg excerpts that cause this crash, at:

http://idisk.mac.com/r.mcnamara-Public/?view=web

This bug has been around for over a year? Can we get an update?

comment:14 Changed 16 years ago by anonymous

yeah--way over a year! It was a year when I bumped into it. I've given up on mythtranscode. I use mythweb to save the thumbnail--except I don't save it but rather just copy the filename text from the save dialog box so I may paste the .mpg filename into HDTVtoMPEG2.exe. It runs under wine but I needed to get a special version from the developer on www.avsforum.com that would recognize transport stream files with .mpg extensions.

Well--that's my workaround anyway. Not too convenient and no batch mode--though you could probably use avidemux or some other ap too but avidemux doesn't handle transport streams very well and tends to lose A/V sync.

-Cal (c b r a b a n d t AT y a h o o PERIOD c o m)

comment:15 Changed 16 years ago by anonymous

I'm very sorry that you've been having problems with mythtranscode. If you contact customer services we can arrange a refund.

comment:16 Changed 16 years ago by TugBoat

Besides affecting MythArchive?, I've bumped into this problem with attempts at lossless transcoding to get rid of ads and save space. This bug makes MythArchive? useless with DVB (MPEG-2 TS) recordings, which represent the majority on our HD system, because MythArchive? can't take the cuts out and nor fix a/v sync before it tries encoding.

It happens on both our FC6 and FC7 slave backends. If mythtranscode uses the other encoders, it works fine.

It is a serious bug.

comment:17 Changed 16 years ago by robert.mcnamara@…

While we (hopefully) work towards a more permanent solution, I have found a workaround that basically works. I have written a script that demuxes and remuxes the broken recordings, resulting in a file that does, finally, work with myth's lossless transcode. I've put it on my talk page at the mythtv wiki:

http://www.mythtv.org/wiki/index.php/User_talk:Iamlindoro

This is the first solution that I know of that doesn't use a Windows program through Wine. ProjectX has never worked for me, but this appears to work consistently. Read the comments on the script carefully. You will need to compile one tiny prerequisite, and will also need ffmpeg. The script should be set up as a user job, and I personally run in on all recordings from my Firewire source. There is no quality loss involved as there is no real transcoding, just demux/remux. I use mythtranscode to rebuild the index of the file (mythcommflag --rebuild doesn't work properly) but no quality is lost there. The resultant file can be marked and losslessly transcoded successfully. Obviously the elegant solution is to get mythtranscode fixed, but this is working for me in the meantime.

Geoff-- Any chance you might be able to look at this soon?

comment:18 Changed 16 years ago by J.Pilk@…

See also my post on Myth-users; it includes a less-polished script for stripping off all but one audio channel that I have used to pre-process recordings before creating the cutlist. It may be quicker than the one above. http://www.gossamer-threads.com/lists/mythtv/users/320076

comment:19 Changed 16 years ago by jarb1@…

I ran into this problem recently with some DVB recordings from BBC multiplex on freeview.

Rather than run scripts separately, I wanted mytharchive just to work and found that projectX when installed and enabled in mytharchive settings worked a treat.

Use the guide for Mythburn http://www.mythtv.org/wiki/index.php/MythBurnInstallation#Java

I could see no other way in Myth0.21 to run additional commands such as mencoder to "clean" the mpeg file before it hit mythtranscode.

comment:20 Changed 15 years ago by robert.mcnamara@…

Here is a long (3 Minute, 235 MB) sample of a channel that exhibits this problem. In living with this bug over the past year or so, I've learned some things anecdotally. For me, this problem manifests on certain channels without fail. For example, Universal HD, Discovery HD, HBO HD, etc. *always* fails lossless transcode. Further, this problem seems to be most evident on HD channels.

Sample: http://www.fecitfacta.com/1739_20080928094700.mpg

There is a Windows Tool called MPEG2Cut2 that lossless cuts affected files properly. It's GPL and runs well in WINE, so it's a workaround for now. Unfortunately, it only cuts at GOP borders and not frame-level, but it's close enough in most cases. There are occasional issues with audio sync, but it's the best option I've found thus far. I wonder whether any of the stream correction code could be leveraged in mythtranscode to overcome this.

The executable is here: http://download.videohelp.com/download/mpg2cut2_7701_libmmd_mpalib.zip

and the source is here: http://www.dvbsupport.net/download/index.php?act=download&id=98

The source is fairly nicely commented, and the file "Fault_Tolerance_SUBALLOC.txt" may be of some use/inspiration here.

Changed 15 years ago by robert.mcnamara@…

Attachment: mt-replexsync-clean.diff added

Initial work on replex sync

comment:21 Changed 15 years ago by robert.mcnamara@…

I have added a diff that brings the version of replex included in mythtranscode up to the newest version. This bug appears to spring from a problem with the current version in trunk. This updates mythreplex to the newest version and makes some of the changes necessary for mythtranscode to hook into it. Someone with better knowledge of the way that mpeg2fix.cpp and replex interact will have to finish the work, though. This patch WILL break trunk compiling as the work is not finished, so don't go compiling it if you don't intend to help finish the work. Specifically, mpeg2fix.cpp will not compile because the arguments for the remuxing stuff have changed.

Changed 15 years ago by robert.mcnamara@…

makes replex sync functional

comment:22 Changed 15 years ago by robert.mcnamara@…

Thanks to some big help from Gnome42 in #mythtv, this syncs Myth's version of replex to the latest available. It unfortunately does *not* solve this bug (although it does appear to introduce some AC3 fixing routines which is kind of nice). For anyone looking seriously to fix this bug, the primary issue appears to be broken (negative) PTS's in the stream. I have found some references in ancient discussions about a possibly implementing a "big hammer" solution to this kind of problem where, after a certain number of PTS discrepancies, mythtranscode would give up and keep a running count instead. As it stands, mythtranscode simply gives up after 20 unfixable PTS discrepancies.

Geoff, any chance of your coming out of retirement and taking a quick look at this one? If you've moved on from myth, no worries, but any chance of touching base so we can see if someone else can take ownership of this bug?

Changed 15 years ago by robert.mcnamara@…

Updated version of replex resync with some bug fixes.

comment:23 Changed 15 years ago by wesattexas

mythtranscode causes mytharchive to fail with result:232[[BR]] ( mythtranscode also fails from command line )

ubuntu 8.04 Hardy Heron
Linux J8 2.6.24-23-generic #1 SMP Thu Nov 27 18:13:46 UTC 2008 x86_64 GNU/Linux
Silicon Dust HDTV capture device.
All My recordings are from HDTV.
Most all have the same problem as this one.

Saved one hour program from PBS
1131_20090108215900.mpg 5.9 GB MPEG video Thu 08 Jan 2009 11:00:01 PM CST

wes@J8:/video/longterm$ mythtranscode -m -c 1131 -s 2009-01-08T21:59:00 -o /video/mytharchivetemp/work/1/newfile.mpg
2009-01-30 02:18:50.752 Using runtime prefix = /usr, libdir = /usr/lib
2009-01-30 02:18:50.811 Empty LocalHostName?.
2009-01-30 02:18:51.040 New DB connection, total: 1
2009-01-30 02:18:51.105 Closing DB connection named 'DBManager0'
2009-01-30 02:18:51.105 Enabled verbose msgs: important
2009-01-30 02:18:51.122 New DB connection, total: 2
2009-01-30 02:18:51.716 Deadlock detected. One buffer is full when the other is empty! Aborting

on sudo apt-get install mythtv mytharchive,
mythtv and mytharchive both claim to be already the newest version.

I will need some help if any one needs mythtv and mythtranscode
version info or log info, but this doesn't look version specific.
I found only a few references and no info on a solution for this error,
but the errors seem go back over several years (2006)
and several flavors of MythTV/mythtranscode.
regards, Wes

comment:24 Changed 15 years ago by stuartm

Owner: Geoffrey Hausheer deleted
Status: acceptednew

comment:25 Changed 15 years ago by Janne Grunau

Milestone: unknown0.22
Owner: set to Janne Grunau
Status: newaccepted

comment:26 Changed 15 years ago by Janne Grunau

(In [21503]) syncs replex copy to replex-1.06. Refs #2077

patch by: Robert McNamara?

comment:27 Changed 15 years ago by Janne Grunau

(In [21517]) fix FreeBSD compilation after [21503] (O_LARGEFILE not defined). Fixes #6907, Refs #2077

patch by: Raymond Wagner < raymond wagnerrp com >

comment:28 Changed 14 years ago by stuartm

Milestone: 0.22unknown

comment:29 Changed 14 years ago by stuartm

Component: mythtranscodeMythTV - Mythtranscode

comment:30 Changed 14 years ago by Janne Grunau

(In [22154]) revert [21503]: replex sync to 0.1.6. Fixes #6981, Refs #2077

comment:31 Changed 13 years ago by beirdo

Owner: changed from Janne Grunau to beirdo
Status: acceptedassigned

This has sat dormant for nearly 2 years since the last comment. I am days away from committing a reworked mpeg2fix.cpp without Qt3Support structures being used. There is a good chance that these reported issues are gone with either the several ffmpeg syncs we have done, or with subsequent debugging. I will close this ticket a couple weeks after the new code has been put into master, and if the problem reoccurs with that code, please open a new ticket. If someone is up to debugging it with known bad recordings in the meantime, all the better.

comment:32 Changed 13 years ago by Github

Convert mythtranscode/mpeg2fix code from Qt3->Qt4 structures

As Qt3 is long past in our history, it was high time to remove the uses of Q3PtrList, Q3PtrQueue from mpeg2fix.

Also changed were some QMap and QString-related code that complained loudly when I removed qt3support from the .pro file.

This could well fix a ticket as well. Refs #2077.

Branch: master Changeset: ab988ce84c68039806e4ef0f64b25a1faca27f07

comment:33 Changed 12 years ago by beirdo

Status: assignedinfoneeded

After the recent rework, is this still an ongoing issue, or should we close this and open a new one if it reoccurs?

comment:34 Changed 12 years ago by J.Pilk@…

This used to be an issue for me with dvb-t recordings in the UK, but it was content dependent and not really predictable. I don't see it now because I use Project-X instead, on 0.24-fixes. The recent changes probably don't apply to that branch, but to be honest, even if they did, I don't think I'd feel motivated to switch back to mythtranscode. Not unless it really worked _and_ was frame-accurate; but testing for that would be a chore. HTH.

comment:35 Changed 12 years ago by stuartm

I can still reproduce the original problem and following the recent conversion to QT4 I'm also seeing corruption around cut boundaries that wasn't an issue before.

comment:36 Changed 12 years ago by tom@…

I'm still seeing the originally reported error. I can keep this recording for further testing, but it's 6 GB, so sharing will be difficult.

2011-11-08 21:39:38.386497 I  Input #0, mpegts, from '/var/lib/mythtv/recordings/1747_20111104190000.mpg':
2011-11-08 21:39:38.386505 I    Duration: 01:01:20.14, start: 92744.741222, bitrate: 14191 kb/s
2011-11-08 21:39:38.386530 I      Stream #0.0[0x18cb]: Video: mpeg2video, yuv420p, 1920x1080 [PAR 1:1 DAR 16:9], 38810 kb/s,  29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
2011-11-08 21:39:38.386543 I      Stream #0.1[0x18cc](eng): Audio: ac3, 48000 Hz, 5.1, s16, 384 kb/s
2011-11-08 21:39:38.390480 I  #0 PTS:25:45:45.480 Delta: 0.0ms queue: 24
2011-11-08 21:39:38.390498 I  #1 PTS:25:45:45.477 Delta: 3.65556ms queue: 2
2011-11-08 21:39:38.453371 E  Deadlock detected.  One buffer is full when the other is empty!  Aborting
2011-11-08 21:39:38.454578 E  Transcoding /var/lib/mythtv/recordings/1747_20111104190000.mpg failed

comment:37 Changed 12 years ago by beirdo

OK, if someone has an example file for this, please let me know (email to gjhurlbu@…), and we can discuss how to get me the file to do some testing with.

comment:38 Changed 12 years ago by stuartm

Type: defectBug Report - General
Version: headMaster Head

I'll include an example or two with those others samples I promised you before Christmas, just keep reminding me until I finally stick them in the post.

comment:39 Changed 11 years ago by Peter Bennett <pgbennett@…>

2013-01-20 16:23:02.734844 E  Deadlock detected.  One buffer is full when the other is empty!  Aborting
2013-01-20 16:23:02.739084 E  Transcoding /srv/mythtv/video3/recordings/2781_20130113210000.mpg failed

I am getting this with version 0.26. This is the first time I have tried mythtranscode, and it fails immediately. The output file is 0 bytes.

This is the command line

mythtranscode --chanid 2781 --starttime "2013-01-14 02:00:00" --mpeg2 --honorcutlist -o "/srv/cstorage/Video/Recordings/Star Trek Nemesis/nemesis.mpg"

This was recorded with firewire, so it is not a DVB related problem. I will keep the file around in case anybody want to take a look into it. Please feel free to email me.

Peter Bennett

pgbennett at comcast.net

comment:40 Changed 11 years ago by Frank Feuerbacher <fbacher@…>

I am having this problem as well for 1080i content captured from an InfinTV on a remote Windows box. Problem occurs whenever I use mythtranscode and --mpeg option. If I don't specify --mpeg then it fails with "No Video"

It fails quickly. I'll spend some time getting a test file.

fbacher@smeagol:~/ReadyToTrim$ mythtranscode --infile /var/lib/mythtv/1854_20130113130100*.mpg --outfile /tmp/test3.mpg 
2013-01-20 16:45:24.329743 C  mythtranscode version: fixes/0.26 [v0.26.0-82-g8f8274a] www.mythtv.org
2013-01-20 16:45:24.329753 C  Qt version: compile: 4.8.3, runtime: 4.8.3
2013-01-20 16:45:24.329757 N  Enabled verbose msgs:  general
2013-01-20 16:45:24.329763 N  Setting Log Level to LOG_INFO
2013-01-20 16:45:24.329923 I  Added logging to the console
2013-01-20 16:45:24.330131 I  Setup Interrupt handler
2013-01-20 16:45:24.330136 I  Setup Terminated handler
2013-01-20 16:45:24.330141 I  Setup Segmentation fault handler
2013-01-20 16:45:24.330145 I  Setup Aborted handler
2013-01-20 16:45:24.330148 I  Setup Bus error handler
2013-01-20 16:45:24.330152 I  Setup Floating point exception handler
2013-01-20 16:45:24.330156 I  Setup Illegal instruction handler
2013-01-20 16:45:24.330163 I  Setup Real-time signal 0 handler
2013-01-20 16:45:24.330186 N  Using runtime prefix = /usr
2013-01-20 16:45:24.330194 N  Using configuration directory = /home/fbacher/.mythtv
2013-01-20 16:45:24.330229 I  Assumed character encoding: en_CA.UTF-8
2013-01-20 16:45:24.330669 N  Empty LocalHostName.
2013-01-20 16:45:24.330674 I  Using localhost value of smeagol
2013-01-20 16:45:24.346230 N  Setting QT default locale to en_CA
2013-01-20 16:45:24.346277 I  Current locale en_CA
2013-01-20 16:45:24.346308 N  Reading locale defaults from /usr/share/mythtv//locales/en_ca.xml
2013-01-20 16:45:24.348372 I  Loading en_us translation for module mythfrontend
2013-01-20 16:45:24.349512 N  Transcoding from /var/lib/mythtv/1854_20130113130100.mpg to /tmp/test3.mpg
2013-01-20 16:45:24.553170 I  Added logging to mythlogserver at TCP:35327
2013-01-20 16:45:24.652409 I  MythCoreContext: Connecting to backend server: 192.168.1.118:6543 (try 1 of 1)
2013-01-20 16:45:24.706170 I  Using protocol version 75
2013-01-20 16:45:24.797522 I  AFD: codec AC3 has 6 channels
2013-01-20 16:45:24.814311 I  AFD: Opened codec 0x1916e20, id(AC3) type(Audio)
2013-01-20 16:45:24.868975 I  AFD: Opened codec 0x19191e0, id(MPEG2VIDEO) type(Video)
2013-01-20 16:45:24.869039 N  AudioPlayer: Enabling Audio
2013-01-20 16:45:24.886220 N  Found video height of 1088.  This is unusual and more than likely the video is actually 1080 so mythtranscode will treat it as such.
2013-01-20 16:45:24.886240 N  Transcode: Looking for autodetect profile: Autodetect from 1080i
2013-01-20 16:45:24.917185 N  Transcode: Using autodetect profile: MPEG2
2013-01-20 16:45:24.917272 E  No video information found!
2013-01-20 16:45:24.917275 E  Please ensure that recording profiles for the transcoder are set
2013-01-20 16:45:24.973397 I  Starting process manager
2013-01-20 16:45:24.973401 I  Starting process signal handler
2013-01-20 16:45:24.973989 I  Starting IO manager (write)
2013-01-20 16:45:24.974049 I  Starting IO manager (read)
2013-01-20 16:45:25.246612 I  Pulse: PulseAudio resume OK
2013-01-20 16:45:25.266397 E  Transcoding /var/lib/mythtv/1854_20130113130100.mpg failed
2013-01-20 16:45:25.266411 I  Waiting for threads to exit.
Last edited 11 years ago by Karl Egly (previous) (diff)

comment:41 Changed 11 years ago by Frank Feuerbacher <fbacher@…>

I created a 9 second 1080i mpg clip (13.5M) available at:http://feuerbacher.us/data/No%20Way%20Out9sec.mpg

I created the file using Avidemux in copy mode (audio & Video) from a .mpg file that was failing in the Mythtv library. After creating the file I used: mythtranscode --infile /var/lib/mythtv/1854_20130121110000.mpg --mpeg2 --buildindex --allkeys --showprogress To re-register the file into mythTV

Then I used:

mythtranscode --infile /var/lib/mythtv/1854_20130121110000.mpg --outfile /tmp/test3.mpg --mpeg2 --honorcutlist

To produce the error. Deadlock Detected.

I ran on a 4cpu Ubuntu 12.10 with mythtv 0.26

2013-01-21 20:18:47.520160 C mythtranscode version: fixes/0.26 [v0.26.0-86-g012ebe9] www.mythtv.org 2013-01-21 20:18:47.520180 C Qt version: compile: 4.8.3, runtime: 4.8.3

comment:42 Changed 11 years ago by nicolas.poehlmann@…

Status: infoneededassigned

I'm running in the same problem after upgrading from 0.25.3 to 0.26/fixes. After two days debugging I've found the problem which makes lossless transcode stop working if the record has one or more AC3-Stream(s). It's the same problem as described in the ffmpeg-Ticket at http://ffmpeg.org/trac/ffmpeg/ticket/1240 where coping audio wasn't working.

I'll upload an patch which can be run against 0.26/fixes. This will let lossless transcode work again and will make an improved log-output to detect such a problem in the future and a little performance improvement.

BUT to get rid of such a "wrong" error output the whole buffering/multiplex routine had to be rewritten nearly from scratch. The real problem is not that the buffer was running empty, it was never filled because the buffering AddFrame?-Routine isn't starting when the codec reports a frame_size of 0, but the multiplex-routine is running into a loop trying to (re-)fill the buffer which can't be filled. And ending up in a "Deadlock detected. One buffer is full when the other is empty! Aborting"-Error. To avoid such a loop-problem in the future, there had to be a second frame_size calculation if the mpx/ac3-header had real errors.

Changed 11 years ago by Nicolas Pöhlmann <nicolas.poehlmann@…>

Attachment: mythtransode-ac3.diff added

AC3 deadlock patch

comment:43 Changed 11 years ago by tom@…

Would you like the sample file I referenced on 2011-11-09? I trimmed it down to 19 MB so it can be emailed.

comment:44 in reply to:  43 Changed 11 years ago by Nicolas Pöhlmann <nicolas.poehlmann@…>

just upload the file to a webspace and place the link to it, but i don't think it's the same problem. the external ffmpeg version in 0.26 has more bugs that stops the transcode or ruin the output files which isn't directly related to mythtv. 0.27 has a new ffmpeg version implemented where most of these problems should be fixed.

comment:45 Changed 11 years ago by Karl Egly

Here is a PPA for testing the fix with fixes/0.26 on Ubuntu 12.04 LTS without installing all the development tools. https://launchpad.net/~dekarl/+archive/mythtv-transcode-ac3

comment:46 Changed 11 years ago by Robert Kulagowski

I've confirmed that with the PPA installed, 0.26 is now able to successfully cut programs generated by an HDHR-Prime.

comment:47 Changed 11 years ago by aaron@…

Just tested 2 different recordings from 2 sources and both had no errors on mythtranscode. I will be testing more but it would seem the patch fixed it for me.

comment:48 Changed 11 years ago by Karl Dietz <dekarl@…>

In a9b906528ec607400ee2427354bcd84a6a1ce51e/mythtv:

update mythtranscode to work with ac3 with newer ffmpeg

ffmpeg has moved the duration/size of the audio frame to another place, so look
there

links used to figure it out how to move forward
http://ffmpeg.org/trac/ffmpeg/ticket/1240
http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=364c71c80e9124a2624e3bfeb8e84c5cddeda222
http://code.mythtv.org/trac/ticket/2077#comment:42

thanks to Nicolas Pöhlmann for finding the commit that broke it

Refs: #2077

comment:49 Changed 11 years ago by Karl Dietz <dekarl@…>

In f04a8e42b94c68f5b96206c145ce38ff47a9e162/mythtv:

add NULL check missed in [a9b90652]

found by Jim Stichnoth

Refs #2077

comment:50 Changed 11 years ago by Karl Egly

I have updated the 0.26 PPA at https://launchpad.net/~dekarl/+archive/mythtv-transcode-ac3 with the fix that I made to master. I'd like to commit it to fixes/0.26 if it works for you, too.

comment:51 Changed 11 years ago by celpa.firl@…

Did this reach the fixes branch?

comment:52 Changed 11 years ago by Karl Dietz <dekarl@…>

In 61174b20ebba8f1ac595960fffc39f9dad7e4610/mythtv:

update mythtranscode to work with ac3 with newer ffmpeg

ffmpeg has moved the duration/size of the audio frame to another place, so look
there

links used to figure it out how to move forward
http://ffmpeg.org/trac/ffmpeg/ticket/1240
http://git.videolan.org/?p=ffmpeg.git;a=commitdiff;h=364c71c80e9124a2624e3bfeb8e84c5cddeda222
http://code.mythtv.org/trac/ticket/2077#comment:42

thanks to Nicolas Pöhlmann for finding the commit that broke it

Refs: #2077
(cherry picked from commit a9b906528)

comment:53 Changed 11 years ago by Karl Dietz <dekarl@…>

In da89ee585f417584d74cbaee42467dabdc758261/mythtv:

add NULL check missed in [a9b90652]

found by Jim Stichnoth

Refs #2077
(cherry picked from commit f04a8e42b)

comment:54 Changed 11 years ago by Karl Egly

yes, I just pushed it to fixes/0.26 after very light testing

comment:55 Changed 11 years ago by j@…

Running 0.26.0+fixes20130617 I can confirm that it no longer fails with this error message 100% of the time as it had been but now just some of the time.

In addition, when the transcoding process does work, sometimes the files are damaged in the process and FFmpeg will not read them (I try to use FFmpeg to encode them into WebM after the commercials have been deleted.) In other cases where the transcoding process works, FFmpeg reads the files just fine.

In those cases where FFmpeg doesn't like the files it just keeps discarding frame after frame. The total number of frames encoded remain zero or one while the number of discarded frames just keeps going up and up and up.

I tried to use FFmpeg's -acodec copy and -vcodec copy to remux it into a Matroska file but FFmpeg died in the process. However, I was able to use mkvmerge to remux it into a Matroska file just fine, and FFmpeg read and encoded that file into a WebM file. So there seens to be a workaround for the files getting damaged during the transcoding process.

So, there's certainly been progress on this bug but it seems to not be totally fixed just yet.

I have a 10MB sample here in which the transcoding still failed with the error "Deadlock detected. One buffer is full when the other is empty! Aborting", in case it's helpful. I produced it using dd if=/backup/tvrecordings/1647_20130624015800.mpg of=~/Desktop/home.mpg bs=1024 count=1000

It lives at http://aws.bluehome.net/home.mpg

comment:56 Changed 11 years ago by billstuff2001@…

I found that in some cases index_vrbuf was overflowing before vrbuf, and this was causing the "Deadlock detected" failure. This patch increases the default INDEX_BUF size and lets more recordings succeed instead of failing.

--- mythbuild.orig/mythtv/programs/mythtranscode/mpeg2fix.cpp
+++ mythbuild/mythtv/programs/mythtranscode/mpeg2fix.cpp
@@ -590,7 +590,7 @@ void MPEG2replex::Start()  --- Hunk 1 mythbuild/mythtv/programs/mythtranscode/mpeg2fix.cpp
     }
 }
 
-#define INDEX_BUF (sizeof(index_unit) * 200)
+#define INDEX_BUF (sizeof(index_unit) * 2000)
 void MPEG2fixup::InitReplex()
 {
     // index_vrbuf contains index_units which describe a video frame

comment:57 Changed 10 years ago by tobias.powalowski@…

I'm pretty sure this is fixed with latest fixes from: #12153, #12147 and #11639

comment:58 in reply to:  57 Changed 9 years ago by Karl Egly

Milestone: unknown0.26.2
Resolution: Fixed
Status: assignedclosed

Replying to tobias.powalowski@…:

I'm pretty sure this is fixed with latest fixes from: #12153, #12147 and #11639

I think the original issue has been fixed in the meantime. If some of the uncovered other issues are still present in master or fixes/0.27 they should go to new tickets. One issue per ticket.

Note: See TracTickets for help on using tickets.