Opened 8 months ago

Last modified 8 months ago

#13376 new Developer Task

Refresh of video and UI rendering code and hardware video acceleration

Reported by: mark-kendall Owned by:
Priority: major Milestone: 31.0
Component: MythTV - General Version: Master Head
Severity: medium Keywords:
Cc: Ticket locked: no

Description

Key tasks:-

  • remove deprecated libmythui classes
  • migrate to using the QOpenGLxx classes (QGL classes are deprecated)
  • minimise LOC by using new QOpenGL functionality where possible
  • remove OpenGL1 support
  • remove legacy private hardware decoders - VDA and CrystalHD (subject to agreement)
  • convert existing use of FFmpeg hardware accelerations to new API and utilise FFmpeg code where possible (VAAPI and VDPAU)
  • add support for currently unsupported hardware accelerations - CUDA, D3D11
  • optimise AND simplify rendering paths for all hardware accelerations
  • remove VideOutputNullVAAPI and VideoOutputNullVDPAU - equivalent functionality is now available in AVFormatDecoder
  • add OpenMax? EGL render path
  • support higher bit depth video formats - decoder and render. Render with simple downsampling, dithered downsampling and lossless where available.
  • enforce compatible and consistent render methods between the UI and video
    • OpenGL end to end
    • D3D end to end
    • Qt/Xlib? end to end as fallback/legacy

Change History (68)

comment:1 Changed 8 months ago by ctreleaven

What does this mean for the macOS port? OpenGL is now deprecated although it will probably be a long time before it is actually removed:

https://developer.apple.com/macos/whats-new/

https://appleinsider.com/articles/18/06/04/opengl-opencl-deprecated-in-favor-of-metal-2-in-macos-1014-mojave

Last edited 8 months ago by ctreleaven (previous) (diff)

comment:2 Changed 8 months ago by mark-kendall

re macOS port

This doesn't really change anything - although I forgot to add adding support for VideoToolBox? - which is the replacement for VDA.

As you say, OpenGL will be around for a while. In the meantime, we will hopefully get some Vulkan support alongside OpenGL (Qt 5.12 started adding Vulkan code) and then make use of something like MoltenVK - which is a Vulkan wrapper around Metal. Without Qt support for Metal, I can't see how we can use it directly atm - or it would be a big ask of someone to write code from the ground up.

comment:3 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 2b783da11/mythtv:

Remove use of deprecated QGLXXX classes

  • remove the USE_OPENGL_QT5 define and any code that wasn't encapsulated

by that define.

comment:4 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In f738ae10dc/mythtv:

Create the native window in the root widget

comment:5 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 4cb41f24fc/mythtv:

Remove remaining QGLWidget use

  • one remaining use case in MythOpenGLPainter which can go when we

convert to QOpenGLTexture

comment:6 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 35d2b4fde/mythtv:

Remove deprecated libmythui code

  • MythDialogs? has not been removed - just effectively disabled - so some

screens may not work

comment:7 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In e17b0d3dff/mythtv:

VideoOutputOpenGL: Cleanup up some shared painter handling

  • no longer needed - or at least shouldn't be and will expose any

remaining issues

comment:8 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 2b783da11/mythtv:

Remove use of deprecated QGLXXX classes

  • remove the USE_OPENGL_QT5 define and any code that wasn't encapsulated

by that define.

comment:9 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In f738ae10dc/mythtv:

Create the native window in the root widget

comment:10 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 4cb41f24fc/mythtv:

Remove remaining QGLWidget use

  • one remaining use case in MythOpenGLPainter which can go when we

convert to QOpenGLTexture

comment:11 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 35d2b4fde/mythtv:

Remove deprecated libmythui code

  • MythDialogs? has not been removed - just effectively disabled - so some

screens may not work

comment:12 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In e17b0d3dff/mythtv:

VideoOutputOpenGL: Cleanup up some shared painter handling

  • no longer needed - or at least shouldn't be and will expose any

remaining issues

comment:13 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 2b783da11/mythtv:

Remove use of deprecated QGLXXX classes

  • remove the USE_OPENGL_QT5 define and any code that wasn't encapsulated

by that define.

comment:14 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In f738ae10dc/mythtv:

Create the native window in the root widget

comment:15 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 4cb41f24fc/mythtv:

Remove remaining QGLWidget use

  • one remaining use case in MythOpenGLPainter which can go when we

convert to QOpenGLTexture

comment:16 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 35d2b4fde/mythtv:

Remove deprecated libmythui code

  • MythDialogs? has not been removed - just effectively disabled - so some

screens may not work

comment:17 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In e17b0d3dff/mythtv:

VideoOutputOpenGL: Cleanup up some shared painter handling

  • no longer needed - or at least shouldn't be and will expose any

remaining issues

comment:18 in reply to:  2 Changed 8 months ago by Gary Buhrmaster

Replying to mark-kendall:

As you say, OpenGL will be around for a while.

With Apple, there is no way to know if OpenGL/OpenCL will be removed in 10.15 (and iOS 13) or not until 10.25. Every WWDC will have some people listening intently to see if this is the year.

Without Qt support for Metal

In addition to some of the experimental MoltenVK support in Qt 5.12, there was some initial Vulkan support in Qt 5.10. There is also a mesa based project to support OpenGL on Vulkan (called Zink), so one possible path forward might be OpenGL via Zink using MoltenVK on Metal. Most of that code is in various stages of a work in progress rather than considered production ready, but it does seem that there may be a way.

comment:19 Changed 8 months ago by mark-kendall

comment:20 Changed 8 months ago by mark-kendall

d4434fbf3cd00dbf6c4ebf5561db3eefbbf2466b

Convert MythRenderOpenGL to use QOpenGLFunctions

comment:21 Changed 8 months ago by mark-kendall

cb31439dd0a41f3c03d1e0ea080380958abc1369

MythRenderOpenGL: Use QOpenGLShaderProgram for ... shader programs

comment:22 Changed 8 months ago by mark-kendall

fba17a396c186e706eb15abcaf89206d62bd7400

Fix VideoOutputOpenGLVAAPI for recent changes to OpenGL code

comment:23 Changed 8 months ago by mark-kendall

13e6b85b90cc3506040dbec16d53d8bdd132d275

Explicitly define precision in all shaders

comment:24 Changed 8 months ago by mark-kendall

c4bab190b8baa58398820ce3e5885bc441a1439c

MythMainWindow?: Cleanup properly if OpenGL init fails

comment:25 Changed 8 months ago by mark-kendall

4ed4f84d4c640cde51bba51db06cb73ef449c197

OpenGLVideo: Default to standard samplers and textures in GLSL

comment:26 Changed 8 months ago by mark-kendall

62e586da5f9c11132ff4058120907391806f48f2

OpenGLVideo: Move video shaders into a seperate file

comment:27 Changed 8 months ago by mark-kendall

a39c8a9165e7e463391d2a90a59def7ec07e8b81 MythRenderOpenGL: Move shaders into a seperate file

comment:28 Changed 8 months ago by mark-kendall

71901de33b32434eb6e657ec85e5325457421fb0 OpenGLVideo: Cleanup shader customisation

comment:29 Changed 8 months ago by mark-kendall

2577dbbf31167771cb4f6ac4c95250f389252a3c OpenGLVideo: Minor simplification

comment:30 Changed 8 months ago by mark-kendall

ec22d9b9236368436e672bcd0fde88e3487d679c OpenGLVideo: Try and improve shader precision

comment:31 Changed 8 months ago by mark-kendall

247f77faae883db7dd7a9a70389ac3121c40aa24 VideoOutputOpenGL: Drop support for software bob deinterlacer

comment:32 Changed 8 months ago by mark-kendall

8d6ef36799f518b30d54550029c848bc569b92c5 OpenGL: Clean up buffer mapping

comment:33 Changed 8 months ago by mark-kendall

13ada7235f4dea16db4d5eb72d3de6d19c4ee473 MythRenderOpenGL: Use QOpenGLFramebufferObject for ... Framebuffer objects

comment:34 Changed 8 months ago by mark-kendall

c9514dc713c755b2b3917dec0ef6c6518afdac3e Add OpenGL debug logging

comment:35 Changed 8 months ago by mark-kendall

660b88d2b442bbea65ea9a316f3c0d7c5fef079c MythRenderOpenGL: Cleanup glCheck

comment:36 Changed 8 months ago by mark-kendall

c656076d0cc5b126de43a02fa3b9ab3ae5729196 Remove some unneeded OpenGL defines

comment:37 Changed 8 months ago by mark-kendall

b07e39a3c3334a6c107be7f4e20ece1702f2b9c7 Ensure consistent framebuffer targets in MythOpenGLPainter

comment:38 Changed 8 months ago by mark-kendall

29d14f0e274f1e89f5320c5e754fd1106eab21cc OpenGLVideo: Fix texture filtering

comment:39 Changed 8 months ago by mark-kendall

378de555d8ff56007bc93db185079984ddd4eac6 VideoOutputOpenGL: Remove redundant frame cropping

comment:40 Changed 8 months ago by mark-kendall

a45098681d57cec9179352f94d433544f9624396 MythRenderOpenGL: Improve GPU debug output

comment:41 Changed 8 months ago by mark-kendall

22e6f31de1597e2673d48425242d23ff9e004db3 Fixes for MythRenderEGL (Pi)

comment:42 Changed 8 months ago by mark-kendall

1ad06e5b1abedef9d34c696c5858b77b5ce351a1 mythavtest: Extend functionality

comment:43 Changed 8 months ago by mark-kendall

741eeb435c367394201ff5509213d330228c4377 MythRenderOpenGL: Add initial support for framebuffer discards

comment:44 Changed 8 months ago by mark-kendall

27248e5b8356cbbdd75df73c6024661ab8831b05 MythRenderOpenGL: Fix discard define for GL ES

comment:45 Changed 8 months ago by mark-kendall

77d58cfaea5f4187e83f7f59bd9384c884b4eac0 VideoOutputOpenGL: Minor cleanup of gpu debug

comment:46 Changed 8 months ago by mark-kendall

f4ac9f71b6c96f78761200b1b78d53574d27a6e4 OpenGLVideo: Cleanup YV12 shaders

comment:47 Changed 8 months ago by mark-kendall

2daa11ea5afb7ccd7e691283b8d9d3b6c1a2f37f OpenGLVideo: Fix shader clamping

comment:48 Changed 8 months ago by mark-kendall

ee45569748d4407bc89fe86de48b5d2d6be67876 MythRenderOpenGL: Convert to using QOpenGLBuffer for Pixel Buffers

comment:49 Changed 8 months ago by mark-kendall

016a14f6a41dbdf1d3121c91a1e24561b4de15dc Remove QOpenGLWidget use

comment:50 Changed 8 months ago by mark-kendall

87aacc72d33776a3841fd61a4e2303cdeb369723 MythRenderOpenGL: Use QOpenGLBuffer for Vertex Buffer objects

comment:51 Changed 8 months ago by mark-kendall

c4d6ba3f6b9ff737b576f4b427aac1b74b4d9bba MythRenderOpenGL: Save some VBO memory

comment:52 Changed 8 months ago by mark-kendall

54ed51ca8f861a237ce4259143cf785dceda8c5b Mythplugins: Disable gallery plugin

comment:53 Changed 8 months ago by mark-kendall

f677902d302d4c0dafcc7ad5e29bf8f8c8fb3f08 OpenGLVideo: Add discard framebuffer setting on GLES

comment:54 Changed 8 months ago by mark-kendall

comment:55 Changed 8 months ago by mark-kendall

ead954b4d978e1e1a28e14e26c438250a4613bb8 MythRenderOpenGL: Use QOpenGLTexture

comment:56 Changed 8 months ago by mark-kendall

dd57fb3f18df0a9ebd8f37792bdd6fd5aa9ed1e1 MythRenderOpenGL: Remove GLhalf usage

comment:57 Changed 8 months ago by mark-kendall

b40cb1d18c36f8d71ff3a81b6d0f85d6885a50d4 MythRenderOpenGL: Minor refactor to help with breakpoints

comment:58 Changed 8 months ago by mark-kendall

dfd6a13b9ad3bcdd9fa069cf80b1bcd50e7c2cf9 Remove QtOpenGL requirement

comment:59 Changed 8 months ago by mark-kendall

a7e979bb65f096d8c9049195e9aee7bbc51d449d Bump the minimum Qt version to 5.5

comment:60 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In e785b3f96/mythtv:

MythRenderOpenGL: Avoid unnecessary vertex data updates

  • retain the current values and don't update the vertices if the

incoming source and destination are unchanged.

refs #13376

comment:61 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 8419dac97/mythtv:

OpenGLVideo: Remove the OpenGL bobdeint presentation adjustment

  • this clearly isn't working very well and will be replaced in the

shaders

refs #13376

comment:62 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 44a096388/mythtv:

OpenGLVideo: Improve video vertex handling

  • fixes a long standing off-by-one error in the texture coordinates
  • moves video texture/vertex data to QRect for pixel accuracy and

consistency with other MythRenderOpenGL clients

  • combine vertices update code for UI and video elements, providing

cacheing of video vertex data at the same time

  • greatly simplified frame flipping (into OpenGL coordinate space) by

using the new MythGLTexture m_flip flag

refs #13376

comment:63 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 8cdaab886/mythtv:

VideoColourSpace?: Convert to QMatrix4x4

  • for direct compatibilty with the QOpenGL classes and removal of custom

matrix code

  • this MAY BREAK VDPAU colour rendition - not yet tested

refs #13376

comment:64 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 73d700047/mythtv:

Remove VideoOutputNULLVAAPI and VideOutputNULLVDPAU

  • null vaapi functionality has been replicated in the decoder and the

same can be implemented easily for VDPAU.

refs #13376

comment:65 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In a273edf08/mythtv:

Try and 'fix' the Pi build

refs #13376

comment:66 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In d843e1b544/mythtv:

VideoColourSpace?: Fix the raw matrix debug output

Refs #13376

comment:67 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 55ad2e347/mythtv:

OpenGLVideo: Cleanup use of QOpenGLFunctions::Features and extra
features

Refs #13376

comment:68 Changed 8 months ago by Mark Kendall <mark.kendall@…>

In 1d39f1a8da/mythtv:

VideoColourSpace?: Pass the colourspace from FFmpeg to the player

  • create new colourspace defines in VideoFrame?
  • translate FFmpeg colorspaces that we support and pass on through the

video buffers

  • check colourspace at the start of each frame presentation in

VideoOutput? classes (as before - only OpenGL and VDPAU support
colourspaces).

  • add an RGBA colorspace that will be used in the near future
  • the UHD/rec 2020 colourspace hasn't been added as I can't find any

material that explicitly reports it is using it. I suspect we just need
to assume any 10bit pixel format is using a 10bit colorspace.

Refs #13376

Note: See TracTickets for help on using tickets.