Improved Video timing and Synchronization

[Peter Bennett]

Background

The MythTV frontend AVSync process assumes that video uses a fixed frame rate that is provided by the video header. It uses a variety of methods to show frames at that rate. Then it uses the audio and video timestamps to either halve or double the frame rate if the audio is more than three frames out of sync.

A recent recording that I looked at in detail gave 29.97 at its frame rate, but based on the time stamps of the frames it switched between 25 and 29.97 frames per second several times and in fact the main movie was 25 frames per second. MythTV AVSync does a good job of making playback work in spite of this, although in this case it involved dropping a fair number of frames. If there was no audio, this video would be running at the wrong rate most of the time.

AVSync uses the VideoSync class to synchronize based on video scan intervals, real time clock, or two ways of using usleep. It chooses whichever it can get to work.

Built on top of AVSync is a frame drop predictor that drops frames if it determines the frames are coming faster than the video refresh and an audio predictor that predicts over 4 frames whether to drop or extend a frame interval.

New Method

Since modern videos have audio and video timestamps, it makes more sense to use the timestamp to display each frame at the required time rather than assuming a frame rate that may be wrong. A new synchronization routine (AVSync2) works this way. It uses a real time clock via a system function and shows each frame at the calculated correct time. The frame is actually shown at the next refresh time, since the renderers use a dual buffering scheme. AVSync2 adjusts audio synchronization in a different way, by changing the video time base. There is a setting that can be adjusted to improve audio sync when the audio card does not provide the application with good data on the amount of sound buffered.

With this patch, the old sync routine is still the default. To use AVSync2, you need to select it in frontend settings.

The various predictors and the VideoSync class are no longer used with this approach.

Playing of avi files still works, because ffmpeg creates a timestamp for these, since the frames in avi files do not carry timestamps.

Settings

Enable the new sync method as follows. frontend -> Setup -> Video -> Playback -> Advanced Playback Settings -> Enable new timestamp based playback speed (AVSync2). Zoom into that setting to get to AVSync2 audio correction (ms). The audio correction should be set smaller for smoother audio correction or bigger for faster audio correction.

[Peter Bennett]

AVSync2 patch

[Peter Bennett]

updated patch

[jpoet]

I tried AVSync2 on my nVidia GTX-1050 mythfrontend. At 1.0x playback speed, it looks great.

At faster playback speeds, it does look/behave just slightly different than the old AVSync, but I cannot say if it is better or worse -- just different. When skipping/jumping around, it seems to take just slightly longer for the video to "get back up to speed", but only slightly.

On non-interlaced material such as FS1 or ESPN, even at high playback speeds it is smooth. With interlaced content, there is a little bit of jitter, but the old AVSync has that as well. The jitter may have a slightly different quality to it, or that could be my imagination -- I cannot say it is any worse.

This does not fix the issue with it flickering back and forth with new/old frames, when skipping/jumping around within the show, but I assume it was not expected to.

In summary: nice work.

[Peter Bennett]

Support for private decoders (e.g. Raspberry Pi OpenMax?) fixed.

[Peter Bennett]

Replying to jpoet:

At faster playback speeds, it does look/behave just slightly different than the old AVSync, but I cannot say if it is better or worse -- just different. When skipping/jumping around, it seems to take just slightly longer for the video to "get back up to speed", but only slightly.

You can change how quickly it "gets back up to speed" by changing the AVsync2 audio correction (ms) (right arrow from the Enable new timestamp...). A higher value will get back up to speed quicker but at the cost of the video going extra fast or extra slowly.

[warpme]

Peter, I played a bit with this nice improvement. So far works perfect. I have only one remark: jiterometer (OSD info about codeces, CPU usage, A/V sync etc) has some fields empty when I select AVSync2. In my case empty fields are: CPU usage, fps, a/v sync. This is minor thing but I would be nice to have them back :-p

[warpme]

Ah - I forgot to mention: maybe it will be more nicely to have a/v sync method selection via list with 2 entries?

i.e.

"frame predictions playback A/V sync"

"timestamps playback A/V sync"

instead of tick box for AVsync2 + default AVsync ?

after all AVsync2 ins't additional variation of AVsync - but rather alternative. IMHO, in good UI design alternatives should be presented via list of equal choices (like list). It seems to be more intuitive than default + option. Of course old AVsync should be default highlighted choice on list.

just my 0.02$ :-p

[Peter Bennett]

Fix jitterometer. This is a full replacement patch

[Peter Bennett]

With avsync2 the jitterometer displays the av sync in milliseconds. With avsync it is in number of frames. Since frame rates can be variable, I want to work in time basis and not in number of frames. Also the frame rate can be doubled by the deinterlacer and there will be confusion over whether the jitterometer will take that into account.

I am leaving avsync2 setup with a check box. My intention is that avsync2 will replace avsync if it is satisfactory and then the checkbox can be removed.

[jpilk]

I have been using master 1af2764 today, mainly in Fedora 28 with vdpau advanced, and mainly on UK DVB-T .ts recordings or edited recordings in mplex -f 9 (DVD profile without nav packets). I've also viewed some processed DVB-T2 HD content.

Playback of all of these was excellent, and I tried some with modest speedup. Here are a few minor comments.

In the online editor, playback from an editpoint starts with 'AV Sync audio ahead by' around -500 ms, after which the number falls rapidly. I didn't see or hear any artefacts.

Stepping backwards in the editor by single frames doesn't really work. This may not be a direct result of this commit. Stepping forwards is ok. For years I have been editing DVB content at keyframes and feel no need to change.

Sometimes playback does not exit cleanly at EoF; the picture stays on screen and the 'audio ahead' number gets more and more negative.

And playing the .iso image taken from my favourite defective DVD (#13326) showed no obvious event at the point where playback of the actual DVD still fails...

[jpilk]

Some of my comments above probably applied to earlier builds. I don't usually run with -v playback.

Playback restart after a step within the editor has a larger negative AV Sync offset, sometimes more than 1 sec, when not using vdpau. Lipsync is clearly bad initially but is quickly recovered.

[Peter Bennett]

commit 1af27643ac60e6a392c1ab04130aa999ac40b6dd (HEAD -> master, origin/master, origin/HEAD)

Author: Peter Bennett <pbennett@mythtv.org>
Date:   Sat Oct 27 18:43:09 2018 -0400

    Playback: Improved Video timing and Synchronization (avsync2)
    
    Since modern videos have audio and video timestamps, it makes more
    sense to use the timestamp to display each frame at the required time
    rather than assuming a frame rate that may be wrong. A new synchronization
    routine (AVSync2) works this way. It uses a real time clock via a
    system function and shows each frame at the calculated correct time.
    The frame is actually shown at the next refresh time, since the renderers
    use a dual buffering scheme. AVSync2 adjusts audio synchronization in a
    different way, by changing the video time base. There is a setting that
    can be adjusted to improve audio sync when the audio card does not provide
    the application with good data on the amount of sound buffered.
    
    With this patch, the old sync routine is still the default. To use AVSync2,
    you need to select it in frontend settings.
    
    Fixes #13334

[Peter Bennett]

commit c459f3089f0d56b46b94133a65580e0d2a072b94 (HEAD -> master, origin/master, origin/HEAD) Author: Peter Bennett <pbennett@…> Date: Tue Oct 30 10:08:31 2018 -0400

    Playback: Fix AVSync2 failure to compile on MACOS
    
    MACOS does not support Linux functions clock_gettime and clock_nanosleep.
    Change to use QT timing methods instead, which will be more portable.
    
    Refs #13334

[Peter Bennett]

avsync2 fix for where you are stuck on one fame for extended time when decoding is slow

[Peter Bennett]

Replace the prior patch, which has a bug.

[Peter Bennett]

commit 7a74d6a845b80e1223bdcefe669a079bbf7c7762 Author: Peter Bennett <pbennett@…> Date: Wed Oct 31 18:39:22 2018 -0400

    Playback: Avsync2: Fix for when player shows one frame for a long time
    
    If decoding is slow, the player drops frames to catch up. However
    if the decoding is slower than playback, it never catches up, leaving
    a single frame shown. The fix limits the number of sequential
    dropped frames and also pauses audio so that video can catch up
    in extreme situations
    
    Refs #13334

[jpilk]

Unfortunately that commit broke playback of DVB radio, where I normally see a blank screen. I haven't tried it with the 'interactive' OSD or other visuals.

The audio playback is chopped into brief chunks between longer gaps. el7 with old intel hardware gives this, while my f28 vdpau-box currently reports an even larger 'audio ahead' value of almost a day.

I may be able to build with patches now, although I haven't done it yet.

2018-11-03 11:19:18.893017 I  Player(1): AV Sync, audio ahead by 28473771 ms
2018-11-03 11:19:18.922569 I  Player(1): AV Sync, audio ahead by 28473761 ms
2018-11-03 11:19:18.953193 I  Player(1): AV Sync, audio ahead by 28473751 ms
2018-11-03 11:19:18.982027 I  Player(1): Waiting for video buffers...
2018-11-03 11:19:19.093990 N  Player(1): Waited 111ms for video buffers AUUUUUUUUUUUAAAAAAAAAAAAAAAAAAAP
2018-11-03 11:19:19.201245 N  Player(1): Waited 219ms for video buffers AUUUUUUUUUUUAAAAAAAAAAAAAAAAAAAP

....

2018-11-03 11:19:19.844902 N  Player(1): Waited 862ms for video buffers AUUUUUUUUUUUAAAAAAAAAAAAAAAAAAAP
2018-11-03 11:19:19.952255 N  Player(1): Waited 970ms for video buffers AUUUUUUUUUUUAAAAAAAAAAAAAAAAAAAP
2018-11-03 11:19:20.059534 N  Player(1): Waited 1077ms for video buffers AUUUUUUUUUUUAAAAAAAAAAAAAAAAAAAP
2018-11-03 11:19:20.166880 N  Player(1): Waited 1184ms for video buffers AUUUUUUUUUUUAAAAAAAAAAAAAAAAAAAP
2018-11-03 11:19:20.228758 I  Player(1): Resetting AV Sync2, lateness = 1218000
2018-11-03 11:19:20.228773 I  Player(1): AV Sync, audio ahead by 28477747 ms
2018-11-03 11:19:20.270954 I  Player(1): AV Sync, audio ahead by 28477749 ms

[Peter Bennett]

commit eafa7d3aa0951e567db6d309d6c37c290664324a (HEAD -> master, origin/master)

Author: Peter Bennett <pbennett@mythtv.org>
Date:   Sat Nov 3 14:22:18 2018 -0400

    Playback: AVSync2: Fix DVB Radio playback
    
    Cater for case where video has no timestamps.
    
    Refs #13334

[jpilk]

Peter. Thanks again. Simple playback of DVB radio is now fine, but navigation and editing are not.

Jump Fwd, Jump Back, and Skip Fwd seem ok. Skip Back, and quitting at the start of 'Cut to End' are broken. Going to that point in the Editor, moving back 2 x 20 s and restarting plays for about 3 s. Here's -v playback,timestamp for that:

2018-11-04 13:45:20.471573 I  Player(3): A/V timecodes audio=80060743 video=80060743 frameinterval=40000 audioadj=0 tcoffset=0 unow=1541339120471000 udue=1541339120463000
2018-11-04 13:45:20.474106 I  Player(3): A/V timecodes audio=80060746 video=80060746 frameinterval=40000 audioadj=0 tcoffset=0 unow=1541339120474000 udue=1541339120466000
2018-11-04 13:45:20.479342 I  AFD: audio timecode 7206187148 7206187148 80068746 80068770
2018-11-04 13:45:20.480973 I  Player(3): A/V timecodes audio=80060753 video=80060753 frameinterval=40000 audioadj=0 tcoffset=0 unow=1541339120480000 udue=1541339120473000
2018-11-04 13:45:20.483418 I  Player(3): A/V timecodes audio=80060755 video=80060755 frameinterval=40000 audioadj=0 tcoffset=0 unow=1541339120483000 udue=1541339120475000
2018-11-04 13:45:20.485792 I  Player(3): A/V timecodes audio=80060758 video=80060758 frameinterval=40000 audioadj=0 tcoffset=0 unow=1541339120485000 udue=1541339120478000
2018-11-04 13:45:20.488016 I  Player(3): A/V timecodes audio=80060760 video=80060760 frameinterval=40000 audioadj=0 tcoffset=0 unow=1541339120487000 udue=1541339120480000
2018-11-04 13:45:20.490693 I  HasReachedEof() at framesPlayed=47265 totalFrames=66672
2018-11-04 13:45:20.537958 I  TV::HandleStateChange(): Attempting to change from WatchingPreRecorded to None
2018-11-04 13:45:20.538015 I  Player(3): StopPlaying - begin
2018-11-04 13:45:20.539578 I  Player(3): Decoder thread exiting.
2018-11-04 13:45:20.539654 I  Player(3): Exited decoder loop.
2018-11-04 13:45:20.542007 I  VideoBuffers::DiscardFrames(1): AUUUUUUUUUUUUUUUUUUUUUUUUUUAAAAP
2018-11-04 13:45:20.542049 I  VideoBuffers::DiscardFrames(1): AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP -- done
2018-11-04 13:45:21.066249 I  Player(3): StopPlaying - end
2018-11-04 13:45:21.066263 I  TV::StopStuff(): Stopping ring buffer
2018-11-04 13:45:21.066284 I  TV::StopStuff(): Stopping player

[jpilk]

DVB Radio: I placed editpoints using the non-AVSync2 setting. They worked, as expected.

With AVSync2, playback in EditMode? from those points (with cutlist inverted when appropriate) starts immediately, but from a position several seconds later ( ~15 to ~30 s).

The time-and-frame display at the editpoint shows the values as intended. After start and immediate stop the values shown are updated to those for the sound being played.

For these recordings the seektable shows a frame duration of 40 ms and keyframes separated by 8 frames. TTBOMK mythcommflag --rebuild will not work with a copied file. I suppose the timecodes in use here are embedded in the data.

Non-AVSync2 skipping is unreliable for skip durations of 1 frame, 1 keyframe and 0.5 s, but otherwise it works.

I don't understand the magic numbers (133466 etc) in MythPlayer::UpdateFFRewSkip(void)

[Peter Bennett]

I tried the actions in comment 13 and I cannot see your problem. It appears to work the same with or without AVSync2.

Please provide detailed steps. Let me know what setting you use for "Automatically Skip Commercials", what you do, what happens and what is expected.

I am unable to test edit with DVB radio. The only DVB radio recordings I have were recorded by others and I have no seek table for them.

There is nothing in AVSync or AVSync2 specific to editing so I would not expect any differences.

[jpilk]

"Automatically Skip Commercials" is off, but cutlists created by the Editor are acted on, and I would not want to lose that facility. It doesn't work for me with AVSync2 and DVB radio.

But I haven't personally had bad experiences with AVSync, so I'd be quite happy to stay with that. It would be nice to know why some skip values don't work properly with radio; maybe MythPlayer::OpenDummy?, with a keyframe distance of 15, is involved. I believe AVSync WfM. Let's put this on hold. And thanks...

[Peter Bennett <pbennett@…>]

In eafa7d3aa0/mythtv:

Playback: AVSync2: Fix DVB Radio playback

Cater for case where video has no timestamps.

Refs #13334

[Peter Bennett <pbennett@…>]

In eafa7d3aa0/mythtv:

Playback: AVSync2: Fix DVB Radio playback

Cater for case where video has no timestamps.

Refs #13334

[Peter Bennett <pbennett@…>]

In 7a74d6a84/mythtv:

Playback: Avsync2: Fix for when player shows one frame for a long time

If decoding is slow, the player drops frames to catch up. However
if the decoding is slower than playback, it never catches up, leaving
a single frame shown. The fix limits the number of sequential
dropped frames and also pauses audio so that video can catch up
in extreme situations

Refs #13334

[Peter Bennett <pbennett@…>]

In 7a74d6a84/mythtv:

Playback: Avsync2: Fix for when player shows one frame for a long time

If decoding is slow, the player drops frames to catch up. However
if the decoding is slower than playback, it never catches up, leaving
a single frame shown. The fix limits the number of sequential
dropped frames and also pauses audio so that video can catch up
in extreme situations

Refs #13334

[Peter Bennett <pbennett@…>]

In c459f3089/mythtv:

Playback: Fix AVSync2 failure to compile on MACOS

MACOS does not support Linux functions clock_gettime and clock_nanosleep.
Change to use QT timing methods instead, which will be more portable.

Refs #13334

[Peter Bennett <pbennett@…>]

In c459f3089/mythtv:

Playback: Fix AVSync2 failure to compile on MACOS

MACOS does not support Linux functions clock_gettime and clock_nanosleep.
Change to use QT timing methods instead, which will be more portable.

Refs #13334

[Peter Bennett <pbennett@…>]

In 1af27643ac/mythtv:

Playback: Improved Video timing and Synchronization (avsync2)

Since modern videos have audio and video timestamps, it makes more
sense to use the timestamp to display each frame at the required time
rather than assuming a frame rate that may be wrong. A new synchronization
routine (AVSync2) works this way. It uses a real time clock via a
system function and shows each frame at the calculated correct time.
The frame is actually shown at the next refresh time, since the renderers
use a dual buffering scheme. AVSync2 adjusts audio synchronization in a
different way, by changing the video time base. There is a setting that
can be adjusted to improve audio sync when the audio card does not provide
the application with good data on the amount of sound buffered.

With this patch, the old sync routine is still the default. To use AVSync2,
you need to select it in frontend settings.

Fixes #13334

[Peter Bennett <pbennett@…>]

In 1af27643ac/mythtv:

Playback: Improved Video timing and Synchronization (avsync2)

Since modern videos have audio and video timestamps, it makes more
sense to use the timestamp to display each frame at the required time
rather than assuming a frame rate that may be wrong. A new synchronization
routine (AVSync2) works this way. It uses a real time clock via a
system function and shows each frame at the calculated correct time.
The frame is actually shown at the next refresh time, since the renderers
use a dual buffering scheme. AVSync2 adjusts audio synchronization in a
different way, by changing the video time base. There is a setting that
can be adjusted to improve audio sync when the audio card does not provide
the application with good data on the amount of sound buffered.

With this patch, the old sync routine is still the default. To use AVSync2,
you need to select it in frontend settings.

Fixes #13334

[jpilk]

Following from Comment 16.

I'm posting this here because it seems relevant, not necessarily to spur more urgent action :-)

I have uploaded a 4-minute DVB clip from BBC Radio 4 and the xml file from running:

mythutil --chanid 1704 --starttime 20181109125800 --getmarkup worldatone2018Nov09.xml

I believe that overwriting the file of a local recording, and running:

mythutil --chanid xxxx --starttime yyyymmddhhmmss00.ts --setmarkup worldatone2018Nov09.xml

will create a playable recording with its seektable. That means it should be editable. It worked for me.

The recording includes a time signal, 2 minutes in. For me, running the old AVSync, that coincides with the 'mythical bookmark' and the start of normal playback. It appears in the editor's audiograph display about 10 seconds earlier. Some time ago I tried to make that display longer and in sync, but gave up because it seemed more likely that everything would freeze if the patch wasn't quite right. Tickets #10848 and #12901 refer.

​https://www.mediafire.com/file/01evtm6e0iv8k6o/1704_20181109125800.ts/file

​https://www.mediafire.com/file/u7ywe953tqod5gi/worldatone2018Nov09.xml/file

[jpilk]

Oops: the --starttime in the --setmarkup line shouldn't have the .ts suffix

[Peter Bennett]

commit d61abd28df40ea2fd9c7a3f3d857a4293bdaab24 (HEAD -> master, origin/master, origin/HEAD)

Author: Peter Bennett <pbennett@mythtv.org>
Date:   Mon Nov 12 12:19:19 2018 -0500

    Playback: AVSync2 fix DVB Radio time display
    
    Info screen for DVB radio will display the same elapsed times
    as fo AVSync
    
    Refs #13334

[Peter Bennett]

On my system here in the USA, playing the DVB recording and pressing I shows an incorrect elapsed time in the OSD.

With v29 or master with AVSync2 disabled, after 1 minute timed by stopwatch, the info OSD shows 1:14. This is because DVB radio involves the decoder generating "dummy" video frames, required because all elapsed timing in MythTV is done by frame number. Some parts of the code assume 25fps and other parts assume 30fps, hence the timestamp is out by a factor of 30/25.

It may be that the calculation is related to the refresh interval of my screen, which is 60. If you had a refresh rate of 50 it may work correctly.

AVSync2 was doing worse, displaying 2:26 after 1 minute by stopwatch. The above commit restores the behavior of AVSync2 to what AVSync is doing (i.e. displaying 1:14 after 1 minute).

Fixing the timing problem is out of scope of this ticket.

[jpilk]

Yes, the time displays now agree and, for me with a quoted 25 fps, seem to be correct. Thank you!

But there is now a problem with start of playback, both initially and from the edit screen. It hangs until it gets another control event. The details aren't clear yet. It doesn't continue on the far side of cuts, and in-play skips and jumps die after a fraction of a second.

When it's playing it sounds good. Sorry about the uninspiring test sample!

[Peter Bennett <pbennett@…>]

In d61abd28d/mythtv:

Playback: AVSync2 fix DVB Radio time display

Info screen for DVB radio will display the same elapsed times
as fo AVSync

Refs #13334

[Peter Bennett <pbennett@…>]

In d61abd28d/mythtv:

Playback: AVSync2 fix DVB Radio time display

Info screen for DVB radio will display the same elapsed times
as fo AVSync

Refs #13334

[Peter Bennett]

Closing ticket as complete. Further issues can be raised in new tickets.