Opened 10 years ago

Closed 9 years ago

Last modified 9 years ago

#7964 closed enhancement (fixed)

Predictively skip frames to smooth out timestretch

Reported by: jppoet@… Owned by: tralph
Priority: minor Milestone: 0.25
Component: MythTV - Video Playback Version: head
Severity: medium Keywords: timestretch playback
Cc: Ticket locked: no

Description

Timestretch can be a little jerky because Myth just tries to decode/display the frames faster, but they still have to fit within the refresh.

This patch tries to predictively drop frames at a steady rate, to match the timestretch speed. While it does not make playback silky smooth, it does make it much less jerky.

Attachments (21)

timestretch-predictive-frame-skip.patch (9.5 KB) - added by jppoet@… 10 years ago.
Predictively drop frames based on playback speed
frame-skip-v3.patch (13.9 KB) - added by jpoet 10 years ago.
Use a table for show:skip values
frame-skip-v1.4.patch (14.8 KB) - added by jpoet 10 years ago.
Much smoother at non 1.25x,1.5x speeds
mythtv_smoothsync.3.patch (9.1 KB) - added by Mark Spieth 9 years ago.
vsync fix + predictive skip at any rate
mythtv_smoothsync.4.patch (18.2 KB) - added by Mark Spieth 9 years ago.
smoothsync with better start and unpause
mythtv_smoothsync.5.patch (51.1 KB) - added by Mark Spieth 9 years ago.
mythtv_smoothsync.6.patch (51.1 KB) - added by Mark Spieth 9 years ago.
same as 5 but made from lower in patch stack
mythtv_smoothsync.8.patch (54.4 KB) - added by Mark Spieth 9 years ago.
for post hdaudio
mythtv_smoothsync.11.patch (53.2 KB) - added by Mark Spieth 9 years ago.
updated for r25575
mythtv_smoothsync.12.patch (12.8 KB) - added by Mark Spieth 9 years ago.
smoother-vsync-audio.11.patch (1.3 KB) - added by Mark Spieth 9 years ago.
smoother-vsync-vsync-opt.11.patch (12.1 KB) - added by Mark Spieth 9 years ago.
mythtv_smoothsync.13.patch (12.9 KB) - added by Mark Spieth 9 years ago.
update after r26895
mythtv_smoothsync.13-24fixes-p0.patch (12.4 KB) - added by jpoet 9 years ago.
version 13 against 0.24-fixes r27341
smoother-vsync-vsync-opt.11-24fixes-p0.patch (11.8 KB) - added by jpoet 9 years ago.
Version 11 against 0.24-fixes r27341
smoother-vsync-audio.11-24fixes-p0.patch (466 bytes) - added by jpoet 9 years ago.
Version 11 against 0.24-fixes r27341
mythtv_smoothsync.13-trunk.patch (13.1 KB) - added by jpoet 9 years ago.
Updated for 206bde055d2d440a730270bf15c34dcf42992d8f
smoother-vsync-vsync-opt.11-trunk.patch (11.9 KB) - added by jpoet 9 years ago.
Updated for 206bde055d2d440a730270bf15c34dcf42992d8f
mythtv-smoothsync-extra-trunk-01.patch (14.5 KB) - added by tralph 9 years ago.
Remaining extra code for smooth sync feature - needs review - updated for 49065221b40b3bc928bb372f57bb63b26077a871
mythtv-smoothsync-extra-trunk-02.patch (10.0 KB) - added by tralph 9 years ago.
updated for 956b4f277fa23d8d8fc98b82f0911da642b18305
mythtv_smoothsync-24fixes-p0.patch (24.6 KB) - added by tralph 9 years ago.
combinded and refeshed patch for 0.24-fixes users

Download all attachments as: .zip

Change History (54)

Changed 10 years ago by jppoet@…

Predictively drop frames based on playback speed

comment:1 Changed 10 years ago by Stuart Auchterlonie

Component: MythTV - GeneralMythTV - Video Playback
Milestone: unknown0.24
Owner: changed from Isaac Richards to markk
Status: newassigned
Version: unknownhead

comment:2 Changed 10 years ago by markk

Status: assignedaccepted

Changed 10 years ago by jpoet

Attachment: frame-skip-v3.patch added

Use a table for show:skip values

comment:3 Changed 10 years ago by jpoet

Update patch for r23788.

Uses a table to define the ratio of shown frames to skipped frames. Using math does not produce expected results.

New version ignores interlaced content, since 30 frame per second material can be display at 2.0x speed on a 60Hz display smoothly without skipping frames.

Still not perfectly smooth past 1.5x play speed, but much better than without the patch.

Changed 10 years ago by jpoet

Attachment: frame-skip-v1.4.patch added

Much smoother at non 1.25x,1.5x speeds

comment:4 Changed 10 years ago by jpoet

New version is much smoother at non 1.25x/1.5x speeds. 1.25x and 1.5x are still the smoothest.

Changed 9 years ago by Mark Spieth

Attachment: mythtv_smoothsync.3.patch added

vsync fix + predictive skip at any rate

comment:5 Changed 9 years ago by Mark Spieth

there are 2 problems with existing smooth playback of mismatched framerate vs refreshrate.

  1. vsync for gl and drm have an incorrect equation in WaitForFrame?
  1. if frameinterval < refreshrate, it will never work correctly. this si the problem addressed by johns patches though it was only for the case where source material rate = refresh rate, otherwise it introduces blockyness.

the latest patch makes playback really smooth for frameinterval >= refreshinterval. tested with playback,timestmap. divergence is now very stable at any play speed. vsync.cpp part only.

the frame skip is untested (NuppelVideoPlayer?.*) but should work. needs extra testing as my systems cant exceed this limit due to fast refresh speeds.

Changed 9 years ago by Mark Spieth

Attachment: mythtv_smoothsync.4.patch added

smoothsync with better start and unpause

comment:6 Changed 9 years ago by Mark Spieth

patch 4 has the previous improvements plus:

  1. better unpause and start playing response. This almost eliminates the frame doubling required at start of play or after seek/unpause. Reduces the zero write latency when audio is paused, and discards any audio packets which come before the first video packet.
  1. faster avsync convergence. convergence is now adjusted by a frame interval period.

item 1 can be split from the patch if desired. audiooutputbase.cpp and avformatdecoder.cpp,h

the vsync patch can also be split as it is also independent on the other changes in NVP.

Changed 9 years ago by Mark Spieth

Attachment: mythtv_smoothsync.5.patch added

comment:7 Changed 9 years ago by Mark Spieth

latest patch has accurate avsync estimation algorithm and doesnt require avsync_avg because of it.

fixed audio timecode generation. wont be relevant as done differently in hdaudio branch but important for avsync.

result is very smooth until the cpu has trouble keeping up.

feedback appreciated.

comment:8 Changed 9 years ago by managementboy@…

Tried to apply, but I get some hunks that don't. Used with trunk 24479...

comment:9 Changed 9 years ago by robertm

Please keep questions about use of patches off of trac... Do not want to have to lock this ticket.

Changed 9 years ago by Mark Spieth

Attachment: mythtv_smoothsync.6.patch added

same as 5 but made from lower in patch stack

Changed 9 years ago by Mark Spieth

Attachment: mythtv_smoothsync.8.patch added

for post hdaudio

comment:10 Changed 9 years ago by Mark Spieth

new patch 8 with #8500 patch included for hdaudio. may fix delayed seek issue.

Changed 9 years ago by Mark Spieth

Attachment: mythtv_smoothsync.11.patch added

updated for r25575

comment:11 Changed 9 years ago by Mark Spieth

updated patch for r25575.

comment:12 Changed 9 years ago by jpoet

Been using version 11 of Mark Spieth's patch for a few days now. It works very well.

I am not qualified to review the actual code, though. Mark Kendall do you still want to own this, or does someone like tralph want to take it?

Changed 9 years ago by Mark Spieth

Attachment: mythtv_smoothsync.12.patch added

Changed 9 years ago by Mark Spieth

Changed 9 years ago by Mark Spieth

comment:13 Changed 9 years ago by Mark Spieth

split logic for smoothsync to simplify approval.

audio (11) is a logging fix and a line I think is important for locking purposes. most went in #8500.

vsync-opt (11) is the vsync stuff which the player uses.

main (12) one uses both audio and vsync opt.

comment:14 Changed 9 years ago by markk

Milestone: 0.240.25
Owner: changed from markk to elmojo
Status: acceptedassigned

comment:15 Changed 9 years ago by stuartm

Milestone: 0.25

Milestone 0.25 deleted

Changed 9 years ago by Mark Spieth

Attachment: mythtv_smoothsync.13.patch added

update after r26895

comment:16 Changed 9 years ago by Mark Spieth

updated patch after r26895 which had a conflict.

Changed 9 years ago by jpoet

version 13 against 0.24-fixes r27341

Changed 9 years ago by jpoet

Version 11 against 0.24-fixes r27341

Changed 9 years ago by jpoet

Version 11 against 0.24-fixes r27341

comment:17 Changed 9 years ago by jpoet

New patches added for 0.24-fixes r27341

comment:18 Changed 9 years ago by tralph

Mark, could you please explain why the audio patch is needed?

comment:19 Changed 9 years ago by Mark Spieth

from memory, when upmix state is changed, reset is necessary so reset_active lock is used. independent of the smoothsync patch. there were other parts which were committed by jean-yves a while back I think.

the other is a logging change as the args were bogus.

comment:20 Changed 9 years ago by Mark Spieth

the main purpose of audio patch was for accurate audio timestamps. see patch 8 for historical purposes.

comment:21 Changed 9 years ago by tralph

Owner: changed from elmojo to tralph

comment:22 in reply to:  19 Changed 9 years ago by JYA

Replying to markspieth:

from memory, when upmix state is changed, reset is necessary so reset_active lock is used. independent of the smoothsync patch. there were other parts which were committed by jean-yves a while back I think.

the other is a logging change as the args were bogus.

Mark, ToggleUpmix? calls Reconfigure, which calls reset_active.Clear() very early on. So calling reset_active.Ref() as well would be equivalent to doing: reset_active.head++; immediately followed by: reset_active.head = 0;

I can't imagine why this audio patch would make a difference whatsoever.

Changed 9 years ago by jpoet

Changed 9 years ago by jpoet

comment:23 Changed 9 years ago by jpoet

Updated patches for trunk 206bde055d2d440a730270bf15c34dcf42992d8f

Changed 9 years ago by tralph

Remaining extra code for smooth sync feature - needs review - updated for 49065221b40b3bc928bb372f57bb63b26077a871

comment:24 Changed 9 years ago by tralph

Mark, I've committed the bulk of the changes and removed some dead code. I'd like to get everything wrapped up but need some questions answered.

vsync change questions

  1. For the hardware sync methods you changed the value of 'n' to a new equation. Could you explain this? I have no way of testing these changes.
  1. Also you are returning an 'int' for the WaitForFrame? but it's only effective for the hardware vsync methods. Could also explain why this is needed?
  1. Which hardware vsync methods have you tested? Only DRM works anymore.
  1. Why do you avoid the negative trigger accumulation when doing predictive skip in vsync.cpp. Specifically the line, "if (ret_val < -m_frame_interval && m_frame_interval >= m_refresh_interval)"

mythplayer questions

  1. We used to allow +/- 1.5 frame intervals of AV-sync slip before adding compensation to the next trigger. Why did you change it +/- 1 refreshrate interval?
  1. Why do you use the avsync_used variable instead of avsync_avg for the slip adjustment check?
  1. You removed the 'lastsync = true' from the +/- 1 refreshrate interval block. Is this because you want the changes to occur on every new frame. It seems the old code would only make adjustments every other frame because of this and maybe that shouldn't have been the case.
  1. Why is the prevtc != 0 check needed?
  1. It seems you've added a new function to retrieve currentaudiotime and you update the variable in more instances. What's the reason for this?

I believe that covers everything. Thanks again for the patch. The bulk that has been committed makes a substantial difference for time stretch of 60fps material.

Changed 9 years ago by tralph

comment:25 Changed 9 years ago by tralph

Resolution: Fixed
Status: assignedclosed

Closed by the following commits:

49510a8d421e22b2dab6

8fd44c8f6b0c7c2053a6

2fe9124b538524c3399e

956b4f277fa23d8d8fc9

3b102319c7ffa89fc116

Changed 9 years ago by tralph

combinded and refeshed patch for 0.24-fixes users

comment:26 Changed 9 years ago by Taylor Ralph

vsync: refactor the time of day and nexttrigger calculations to simplify the code.

Patch supplied by Mark Spieth.

Refs #7964.

Changeset: 45b8f89034549ef64522bdac2d22097dde342b7f

comment:27 Changed 9 years ago by Taylor Ralph

mythplayer: add LOC to VERBOSE calls for improved logging and make formatting improvements.

Refs #7964.

Changeset: 0db106f66ffb6fa238e9bd990cf871571ae40ce8

comment:28 Changed 9 years ago by Taylor Ralph

mythplayer: add most of the smooth sync code.

Adds most of the smooth sync code which provides predictive frames drops when time stretch needs to playback faster than the displays refesh rate. Some limited testing shows playback at 1.5x for 60fps material is much smoother now.

Many smaller changes have been left out until further review.

Patch by Mark Spieth.

Refs #7964.

Changeset: 49065221b40b3bc928bb372f57bb63b26077a871

comment:29 Changed 9 years ago by Taylor Ralph

Add AVSyncGetAudiotime() helper function to address video only playback.

Also update currentaudiotime in a few additional places.

Patch by Mark Spieth.

Refs #7964.

Changeset: 49510a8d421e22b2dab6c216fbfaa268e990557b

comment:30 Changed 9 years ago by Taylor Ralph

Change avsync_adjustment to use frame_interval instead of refreshrate when audio is way behind.

Patch by Mark Spieth.

Refs #7964.

Changeset: 8fd44c8f6b0c7c2053a61c2590c781961023c8bb

comment:31 Changed 9 years ago by Taylor Ralph

Avoid performing an avsync_adjustment on the very first frame when prevtc == 0.

Patch Mark Spieth.

Refs #7964.

Changeset: 2fe9124b538524c3399eefc99e23009aff818a42

comment:32 Changed 9 years ago by Taylor Ralph

Comments by Mark Spieth: "Due to the fact that refresh rate is the fundamental output timing rate, not the input rate which can be anything. The av_sync adjustment needs to be with respect to the resfresh rate and not frame interval. However, no real proof that this is any better or worse but just IMO more logically correct and consistent."

"avsync_used is the minimum of the delayed estimate and the current value of avsync (with the delay from last vsync event included). This improves lock time as the smallest value is the better one to use just in case the instantaneous value deviates from the steady state. Also speeds up initial lock time as avdel is 0 initially."

Refs #7964.

Changeset: 956b4f277fa23d8d8fc98b82f0911da642b18305

comment:33 Changed 9 years ago by Taylor Ralph

Resolution: Fixedfixed

Add vsync_delay_clock method to compensate for hardware vsync phase issues.

This also deprecates the use of KeepPhase? in favor of handling this in AVSync.

Patch by Mark Spieth.

Closes #7964.

Changeset: 3b102319c7ffa89fc1169079151ac6670909f79c

Note: See TracTickets for help on using tickets.