Opened 4 years ago

Last modified 8 days ago

#12010 assigned Bug Report - General

mythcommflag --rebuild does not re-create good seektables with h264 recordings

Reported by: J.Pilk@… Owned by: pbennett
Priority: minor Milestone: unknown
Component: MythTV - Video Decoding Version: Master Head
Severity: medium Keywords: mythcommflag --rebuild h264
Cc: Ticket locked: no


This looks like a re-run of #11435 and #6243. Initial report here, quoted below.

I have just found that for recordings from DVB-T2 FreeviewHD in the UK, (h264), a listing of the markup data generated after mythcommflag --rebuild contains only a small subset of the lines in a similar listing made immediately after recording.

In the 'original' table, keyframe separation is typically around 24 frames. In the 'rebuilt' table the spacing is variable, sometimes in the hundreds of frames. Points listed there are also in the original table.

hexdump shows that all keyframes in the original list for this recording begin 47 40 65 3x, and to me look genuine. The problem does not affect DVB-T SD recordings, which are mpeg2 format.

In my tests I have usually been working with recordings having seektables that have been rebuilt, often several times. This may explain why editing has sometimes been difficult.

mythutil --chanid 1102 --starttime 20140109182600 --getmarkup Haworth_orig.xml mythcommflag --rebuild --chanid 1102 --starttime 20140109182600 mythutil --chanid 1102 --starttime 20140109182600 --getmarkup Haworth_rebuilt.xml

ls -l Haw* -rw-rw-r--. 1 john john 290708 Jan 9 19:13 Haworth_orig.xml -rw-rw-r--. 1 john john 73286 Jan 9 19:16 Haworth_rebuilt.xml

Attachments (3)

Simp_rebuilt.xml (1.2 KB) - added by J.Pilk@… 4 years ago.
Simp_orig.xml (6.6 KB) - added by J.Pilk@… 4 years ago.
mythtv-master-seektable.patch (9.5 KB) - added by J.Pilk@… 3 years ago.

Download all attachments as: .zip

Change History (21)

Changed 4 years ago by J.Pilk@…

Changed 4 years ago by J.Pilk@…

comment:1 Changed 4 years ago by J.Pilk@…

Short clip from 'The Simpsons', Channel4HD

The link is being rejected as potential spam. I'll try adding it to the email thread quoted above.

comment:2 Changed 4 years ago by stichnot

  • Component changed from MythTV - General to MythTV - Video Decoding
  • Owner set to stichnot
  • Status changed from new to accepted

The situation here is that the recorder has captured keyframes as a combination of IDR keyframes and non-IDR I-frames, but mythcommflag is only registering IDR keyframes.

The fundamental problem is that the particular recorder decides whether non-IDR I-frames are allowed to act as keyframes, but the recorder leaves no clues in the recording itself about which recorder was used. This leaves mythcommflag to guess whether or not to register a keyframe when it sees a non-IDR I-frame.

The default behavior is to assume only IDR keyframes, but then you find e.g. DVB-S transmissions that completely lack IDR keyframes, hence the logic in 802e32ba4dc8852da56db170893e95cbc28637e3 that rewinds and retries with the more lenient setting when it doesn't find an IDR keyframe within the first 1000 or so frames.

Unless ffmpeg can somehow figure this out for us automatically, the only solution I can think of is to add an extra flag to "mythcommflag --rebuild" that allows the user to manually override the IDR-vs-I-frame decision.

comment:3 Changed 4 years ago by J.Pilk@…

Thanks, Jim, for the explanation. An extra flag, or a different option name, would be fine for me, and the default could be left as it is so that anyone who doesn't have problems at present wouldn't be affected. I suppose that at a particular site the setting might be associated with chanid via a config table - but I doubt that that would be worth the complication.

comment:4 Changed 4 years ago by J.Pilk@…

I rebuilt 0.27-fixes today with the patch posted here:

With a single short test recording (3436 frames) the before-and-after getmarkup files are almost identical, with just one of the original keyframes not found. hexdump looks as if it is real. Editor responsiveness is fine. Keyframe numbers displayed by the editor are +1 frame relative to those listed. Stepping forward by single frames, a sudden scene change occurs when the editor frame-number coincides with the 'missed' keyframe, but this isn't true for the 'Braz' clip that I posted in the thread above, in which 2 of the original keyframes were missed.

On this showing I would be happy to see this patch committed. Thank you!

comment:5 Changed 4 years ago by blm-ubunet@…

AFAIU The patch from

has a mistake.

-        if (!usingIframes &&
-            (GetEof() != kEofStateNone || (frames > 1000 && frames < 1100)) &&
+        if (usingIframes &&
+            (GetEof() != kEofStateNone || (frames > 100 && frames < 1100)) &&

The test "frames >100" should have just been left as original code "frames > 1000"


comment:6 Changed 3 years ago by J.Pilk@…

I have v0.28-pre-2255-ge3d17b6 built with a minimally-updated version of this patch. In a test recording from BBC TWO HD DVB-T, duration almost 42 minutes, the original seektable includes only 7 frames that are not in the rebuild. Each getmarkup file is 288 kB. I haven't tested a recent build without the patch, but I suppose it would have the original problem. Is a commit still not possible?

Changed 3 years ago by J.Pilk@…

comment:7 Changed 3 years ago by J.Pilk@…

For convenience I have attached my update for master of the original patch by 'blm-ubunet'. It applies OK for me during rpmbuild although some chunks get 1-line shift.

comment:8 Changed 3 years ago by Justin Alcorn <justin@…>

Mythcommflag v0.28-pre-2291-gd97aeef

Running mythcommflag --rebuild results in an empty recordedseek table for that recording. Recordings are H.264 HD recordings from HD-PVR. Would this issue apply, and has this patch been applied to the 0.28 branch?

comment:9 Changed 3 years ago by John Pilkington <J.Pilk@…>

When I opened this ticket, the problem was that mythcommflag --rebuild in fixes did not find most of the original keyframes. Recently I've been running master with the patch attached earlier; I make few HD recordings, but it has seemed to do the job.

I now find that in current builds of 0.27-fixes almost all of the original keyframes are found, at the same byte-offsets, but the frame-numbers and timestamps drift gradually away from their original values; over a 1-hour recording the rebuilt table shows 48 fewer frames and a duration 1920 ms less.

wc -l *.xml shows 6487 lines originally, 6478 after the rebuild. It looks as if keyframe detection has been fixed somehow, but frame counting is - literally - adrift.

This was a DVB-T2 recording from BBC FOUR HD

comment:10 Changed 3 years ago by J.Pilk@…

It seemed worth running a similar test with the patched current-master. With a 4-minute recording, "diff XX_orig.xml XX_rebuilt.xml" showed that *that* rebuild omitted only the 0 ms timestamp and was otherwise the same as the original.

For me, this is better than I saw with current-fixes, but current-fixes without the frame-count-drift would be OK.

comment:11 Changed 3 years ago by J.Pilk@…

Another datapoint, this time with today's unpatched build of master for ubuntu trusty, [v0.28-pre-2593-gea542a9]. Recording DVB-T2 cbbc HD, wc -l gives 4707 for original, 4700 for rebuild. It's very similar to the results for unpatched -fixes at Comment 9; byte offsets are identical, rebuilt frame count this time is 17 greater than the original.

comment:12 Changed 2 months ago by pbennett

  • Owner changed from stichnot to pbennett
  • Status changed from accepted to assigned

comment:13 Changed 2 months ago by pbennett

  • Version changed from 0.27-fixes to Master Head

comment:14 Changed 2 months ago by pbennett

I have a similar issue with building seektables for h264 video, that has irked me for some time. I am looking into that related problem. See

comment:15 Changed 2 months ago by blm-ubunet@…

Do you have a sample recording avail ? None of my international collection or broadcast H264 causes any problems with incorrect keyframe/seektable generation. But I have some H264Parser patches dating back some years now.

comment:16 Changed 2 months ago by pbennett

The problem was with h264 mkv files, created using handbrake from recordings. See

comment:17 Changed 8 days ago by J.Pilk@…

Peter: You suggested this might be associated with #13242, and asked for a sample file. I'll try to send something useful. Regenerating a seektable from my latest raw sample, and some others, is done at around 120 fps, while I usually see something much faster. It's faster without the extra streams, too. I find that repeated rebuilds are consistent and believe the recorder may miss a few.

I tried vidcutter a few weeks ago and found an ffprobe keyframe command line, which I now see was being used with mkv, that I hadn't seen before. It's here, but I haven't tried timing it:

comment:18 Changed 8 days ago by J.Pilk@…

Add Comment

Modify Ticket

as assigned The owner will remain pbennett.

E-mail address and user name can be saved in the Preferences.

Note: See TracTickets for help on using tickets.