vaapi fails to initialize under wayland

[Stuart Auchterlonie]

When starting playback on Fedora 26 on VAAPI capable hardware, the vaapi libraries fail to initialize properly, and we fall back to software decode.

Switching back to Xorg, rather than Wayland and vaapi works correctly again.

Under Wayland:

libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns -1
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
...
2017-08-15 21:52:56.123025 E  VAAPI: Error at vaapicontext.cpp:162 (#-1, unknown libva error)
2017-08-15 21:52:56.123160 I  VAAPI: Deleting VAAPI display.
2017-08-15 21:52:56.123169 E  VAAPI: Invalid display
2017-08-15 21:52:56.123179 I  VAAPI: Deleted context

Under Xorg:

libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
...
2017-08-15 21:46:59.445622 I  VAAPI: Version: 0.40
2017-08-15 21:46:59.445632 I  VAAPI: Driver : Intel i965 driver for Intel(R) Sandybridge Mobile - 1.8.3
2017-08-15 21:46:59.445637 I  VAAPI: Created VAAPI X11 display
2017-08-15 21:46:59.445656 I  VAAPI: Profile: MPEG2Simple Entrypoints: VLD 
2017-08-15 21:46:59.445667 I  VAAPI: Profile: MPEG2Main Entrypoints: VLD 
2017-08-15 21:46:59.445678 I  VAAPI: Profile: H264ConstrainedBase Entrypoints: VLD EncSlice (UNSUPPORTED) 
2017-08-15 21:46:59.445700 I  VAAPI: Profile: H264Main Entrypoints: VLD EncSlice (UNSUPPORTED) 
2017-08-15 21:46:59.445712 I  VAAPI: Profile: H264High Entrypoints: VLD EncSlice (UNSUPPORTED) 
2017-08-15 21:46:59.445722 I  VAAPI: Profile: H264StereoHigh Entrypoints: VLD 
2017-08-15 21:46:59.445731 I  VAAPI: Profile: VC1Simple Entrypoints: VLD 
2017-08-15 21:46:59.445741 I  VAAPI: Profile: VC1Main Entrypoints: VLD 
2017-08-15 21:46:59.445756 I  VAAPI: Profile: VC1Advanced Entrypoints: VLD 
2017-08-15 21:46:59.445762 I  VAAPI: Profile: None Entrypoints: VideoProc (UNSUPPORTED) 
2017-08-15 21:46:59.445768 I  VAAPI: Desired profile for 'MPEG2 VAAPI': MPEG2Main
2017-08-15 21:46:59.445772 I  VAAPI: Found profile MPEG2Main with entry VLD 
2017-08-15 21:46:59.445780 I  VAAPI: Created context (544x576->544x576)
2017-08-15 21:46:59.446083 I  VAAPI: Deleting VAAPI display.
2017-08-15 21:46:59.446090 I  VAAPI: Deleted context

[Stuart Auchterlonie]

For reference, vainfo runs just fine under wayland

$ vainfo 
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib64/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.40 (libva )
vainfo: Driver version: Intel i965 driver for Intel(R) Sandybridge Mobile - 1.8.3
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileH264StereoHigh         :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileNone                   :	VAEntrypointVideoProc

[Stuart Auchterlonie]

Reported upstream with a simplified reproducer.

​https://bugzilla.redhat.com/show_bug.cgi?id=1491186

[Peter Bennett]

Please check if the patch in #13311 fixes the problem. Note that there is now a place in video setup to enter the VAAPI device path.

[Peter Bennett]

This is fixed in ff7a1478f4406d318bb3c01da3f25a26425847d3. Use vaapi2 instead of vaapi and fill in the device under mythfrontend -> Setup -> Video -> Playback -> General -> Decoder Device for VAAPI2