vaapi / vdpau crash in display profile code

[Stuart Auchterlonie]

My system uses only the Intel graphics from the cpu and therefore uses VAAPI for acceleration.

I hit a situation where the frontend coredumped.

It looks like it's failed to create the vaapi context (happens occasionaly), and we noticed this and coped.

The problem then starts when the new display profile code then goes an selects a vdpau profile, which is wrong for this system. The frontend then tried to load the vdpau libraries and failed and segfaulted.

I'll attach some logs

[Stuart Auchterlonie]

gdb backtrace

[Stuart Auchterlonie]

You can see starting at ​https://code.mythtv.org/trac/attachment/ticket/13091/frontend-vaapi-fail-vdpau-fail-crash-with-verbose.log#L223 that it is clearly deciding (correctly) that vaapi is the way to go (lines 223-225) and then it randomly chooses to try vdpau and fails (lines 226-227)

223	2017-08-04 16:28:38.341663 I  VDP: Accepting: cmp(> 0 0) w() h() framerate() codecs() dec(vaapi) cpus(2) skiploop(enabled) rend(openglvaapi) osd(opengl2) osdfade(enabled) deint(vaapionefield,vaapionefield) filt()
224	2017-08-04 16:28:38.341699 I  VDP: LoadBestPreferences(1920x1080, 25.000, h264)
225	2017-08-04 16:28:38.341758 I  VDP: LoadBestPreferences Result cmp0:> 0 0, cmp1:, prio:1, w:, h:, fps:, codecs:, decoder:vaapi, renderer:openglvaapi, deint:vaapionefield
226	2017-08-04 16:28:38.341797 I  VideoOutput: Trying video renderer: 'vdpau'
227	Failed to open VDPAU backend libvdpau_nvidia.so: cannot open shared object file: 
 No such file or directory

The other log file frontend-vaapi-fail-vdpau-fail-crash.log.1​ captures the crash, but without the --verbose playback. The two taken together should allow us to piece together what is going wrong.

Looking at ​https://code.mythtv.org/trac/attachment/ticket/13091/frontend-vaapi-fail-vdpau-fail-crash.log.1#L74 the stack trace goes via videodisplayprofile.cpp:585 which is right around the area of the new code.

Regards Stuart

[Peter Bennett <pbennett@…>]

In 48111f8d6707c5a53cbb66c796046bb87c38192a/mythtv:

Fixes #13091 - Seg fault in video playback when using an alternate video output method

Fix premature deletion of VideoDisplayProfile? object.

[Peter Bennett <pbennett@…>]

In 433f5d17312e6ffce428de959d87ae4b60a2aebf/mythtv:

Fixes #13091 - Seg fault in video playback when using an alternate video output method

Fix premature deletion of VideoDisplayProfile? object.

(cherry picked from commit 48111f8d6707c5a53cbb66c796046bb87c38192a)