Android Hardware accelerated decode

[Peter Bennett]

Android devices support mediacodec. Add support to MythTV for accelerated decoding.

This may be a lot of work and a long term project.

[Peter Bennett]

Preliminary code with mediacodec support

[Peter Bennett]

Attached is preliminary in-progress code for MediaCodec. It still needs to be cleaned up. Not ready for commit.

Select MediaCodec Normal playback profile. MediaCodec hardware assistance is working. Leave deinterlace as None since the Shield already deinterlaces.

Not working:

• ATSC subtitles not working.
• Video resolution change during playback not working.

To be done:

• Aman suggested a change to help with interlaced content. Not yet done, although deinterlacing seems to be perfect anyway.
• Add a limit counter on retry of sending packet so as to avoid a possible infinite loop.

To Be Decided: For MediaCodec and software decoding (standard) it is now bypassing existing convoluted timing code and now is relying on ffmpeg to do timing calculations. I am not sure whether that change should apply to all decoders, MediaCodec only, decoders that create multiple frames from 1 packet, standard decoders, private decoders, or some combination. If it applies to all, the convoluted code can be deleted.

[Peter Bennett]

Added fix for resolution changes and Aman's deinterlace improvement

[Peter Bennett]

A new preliminary in-progress patch. (20180617_mediacodec_fix_reschange.patch​). It includes the previous patch and adds to it.

Fixes:

• Video resolution change during playback
• Change suggested by Aman for interlaced video
• Added limit counter to prevent endless loop.

[Peter Bennett]

New version of patch. Cleaned up unnecessary changes and fixed raspberry pi openmax

[Peter Bennett]

Latest patch cleans up some unneeded changes. Also the previous patches had broken private decoders including raspberry pi. This fixes that as well. I have removed the ffmpeg change that I got from Aman, waiting for it to be in the official FFmpeg release. It did not seem to make any difference anyway.

[Peter Bennett]

Aman's not yet released ffmpeg fix. Apply separately from other patch.

[Peter Bennett]

Apply this patch with git apply -v --dir mythtv/external/FFmpeg filename.patch

[Peter Bennett]

Playback Pause fixed. Still need to fix jump, FF, REW

[Peter Bennett]

Latest patch

Fix the PAUSE deadlock by allowing decoder pause during processing of a packet. Packet processing is successfully resumed at end of pause.

This does not yet address FF, REW or jump.

[Peter Bennett]

Updated patch fixes mediacodec jump and skip

[Peter Bennett]

20180623_1115_mediacodec_fixed_jump_but_time_bug.patch fixes jump and skip.

There is a bug where the elapsed time progresses at double speed with interlaced content on Nvidia shield, because of the type of deinterlace the shield performs.

Does not yet address FF and REW.

[Peter Bennett]

Fixed playback elapsed time and associated things

[Peter Bennett]

Patch 20180628_1445_timing_fixed.patch​ fixes the displayed elapsed time, and along with that, the forward and back skip will use correct positioning.

What the code now does is

1. Detect double frame rate from the pts timings coming out of the decoder.
2. If there is double frame rate, then increment the frame number only on every second frame.

[Peter Bennett]

Fixes Fast Forward and Rewind with Mediacodec

[Peter Bennett]

This patch, 20180629_ff_rew_fixed.patch​ , includes a fix for fast forward and rewind.

Note that all patches are cumulative, i.e. contain all prior patches and are applied to a clean copy of master source.

[Peter Bennett]

Fixes segfault, FF and REW and reduces buffers to help with 4K playback

[Peter Bennett]

A patch to ffmpeg fixes the segfault in Oreo with 1080 playback. It is likely that this patch cases a segfault in Nougat, it seems the interface may have changed.

Also fixes FF and REW by treating the illegalstate exception as it is was a EAGAIN return.

Reduces frame buffers from 31 to 4, thus reducing the memory required to play 4K content by 320MB. This may cause jerky playback, we may need to check and tune this further.

[Peter Bennett]

A couple of fixes to jump and speed

[Peter Bennett]

Details of patch 20180710_1734_jump_speed.patch :

1. Fix the corrupt frame during jump by dropping 4 frames after the avcodec_flush_buffers, if there is an fps doubling in effect (i.e. deinterlacing via mediacodec)
2. I found that if you disable the audio the interlaced video played at half speed. Fixed that by fixing the frame interval calculation.
3. I found that the "Generic error in an external library" still happens during FF. Changed so that the video codec is re-initialized when that happens. This causes a pause in the FF operation of a few seconds, but is better than the long wait and failure message that happens otherwise.

[Peter Bennett]

Smooth playing with speedup

[Peter Bennett]

Patch 20180716_1139_smoooooth.patch:

Smooth playback with speedup by changing the calculation of frame_interval to take into account fps doubling.

Setting of vbuffers when using mediacodec to 8 buffers.

[Peter Bennett]

Fix LiveTV crash

[Peter Bennett]

Patch: 20180717_1042_partial_livetv_fix.patch

Fixes LiveTV crash. LiveTV still has a problem: it is basically a slide show until you press pause for a few seconds and play again.

[Peter Bennett]

Fixes Live TV

[Peter Bennett]

Patch 20180717_1551_livetv.patch​

Fixes LiveTV glitching at start of play

[Peter Bennett]

These changes (latest patch) have been pushed to git.

[Peter Bennett]

A remaining problem is that Fast Forward and Rewind experience frequent failures (IllegalStateException? from the mediacodec system), which show as pauses during the operation. These seem to get worse with time. A possible solution is to use software decoding for fast forward and rewind. A better solution would be a fix for the IllegalStateException? that is thrown.

[Peter Bennett <pbennett@…>]

In 8e34e6e55/mythtv:

Playback: Prerequisites to support for mediacodec

Mediacodec is used for hardware decoding on android. It works differently
from other decoding methods and needs several changes to support the different flow.

1. On NVidia Shield, mediacodec provides built in deinterlace with frame doubling. This causes a discrepancy between reported framerate and actual framerate. Changes are made to support detecting and using the double framerate.
2. Up to now there was always a maximum of one frame released after a packet was provided to the decoder. Mediacodec releases many frames at a time, typically 15 frames at once after 15 or more packets were sent. Many changes are needed to support this.
3. In conjunction with the previous change, FFmpeg recommend that the code always read all frames available before sending the next packet. Previously it would read one frame before sending the next packet. Change to support this.
4. The existing code uses arcane methods to calculate frame timing from the packet sent. This causes havoc with mediacodec because of the multiple frames per packet and the frame doubling. FFmpeg alreay does this work for you, correctly. Change to use ffmpeg frame timing for ffmeg and mediacodec. Other decoding methods probably should change as well, but that is not done here.
5. When skipping forward or back with doubled framerate from mediacodec, one frame is presented that is a combination of old and new frame. Change to drop the first 4 frames after skipping with doubled framerate to avoid this.
6. With mediacodec there is sometimes a situation where the decoder will neither accept a packet nor provide a frame. According to FFmpeg documentation this will never occur, but it does occur. A short wait and retry in this case solves the problem.

Refs #13233

[Peter Bennett <pbennett@…>]

In 88eb4ef87/mythtv:

FFmpeg avcodec: Fix segmentation fault with decoding on android oreo

avcodec_receive_frame consistently causes a seg fault when decoding 1080i mpeg2
on android version oreo. When copying the frame, the second plane in the buffer
follows on immediately after 1080 lines of the first plane, but the code assumes
it is after 1088 lines of the first plane, based on slice_height. It crashes on
copying data for the second plane when it hits the actual end of the data and
starts accessing addresses beyond that.

Instead of using slice_height here, change to use use height. slice_height is
used at other places in this module and I do not know if they also need to be
changed. I have confirmed that with this change, decoding works correctly
on android oreo as well as on the prior version, android nougat.

Refs #13233

[Peter Bennett <pbennett@…>]

In 11d426bd49/mythtv:

Playback: Support for mediacodec hardware decoding on android

Mediacodec is used for hardware decoding on android. Add support for
Mediacodect during playback, using FFmpeg, if enabled in configure.
Default to enabled when building for android.

Refs #13233

[Peter Bennett <pbennett@…>]

In 8e34e6e55/mythtv:

Playback: Prerequisites to support for mediacodec

Mediacodec is used for hardware decoding on android. It works differently
from other decoding methods and needs several changes to support the different flow.

1. On NVidia Shield, mediacodec provides built in deinterlace with frame doubling. This causes a discrepancy between reported framerate and actual framerate. Changes are made to support detecting and using the double framerate.
2. Up to now there was always a maximum of one frame released after a packet was provided to the decoder. Mediacodec releases many frames at a time, typically 15 frames at once after 15 or more packets were sent. Many changes are needed to support this.
3. In conjunction with the previous change, FFmpeg recommend that the code always read all frames available before sending the next packet. Previously it would read one frame before sending the next packet. Change to support this.
4. The existing code uses arcane methods to calculate frame timing from the packet sent. This causes havoc with mediacodec because of the multiple frames per packet and the frame doubling. FFmpeg alreay does this work for you, correctly. Change to use ffmpeg frame timing for ffmeg and mediacodec. Other decoding methods probably should change as well, but that is not done here.
5. When skipping forward or back with doubled framerate from mediacodec, one frame is presented that is a combination of old and new frame. Change to drop the first 4 frames after skipping with doubled framerate to avoid this.
6. With mediacodec there is sometimes a situation where the decoder will neither accept a packet nor provide a frame. According to FFmpeg documentation this will never occur, but it does occur. A short wait and retry in this case solves the problem.

Refs #13233

[Peter Bennett <pbennett@…>]

In 88eb4ef87/mythtv:

FFmpeg avcodec: Fix segmentation fault with decoding on android oreo

avcodec_receive_frame consistently causes a seg fault when decoding 1080i mpeg2
on android version oreo. When copying the frame, the second plane in the buffer
follows on immediately after 1080 lines of the first plane, but the code assumes
it is after 1088 lines of the first plane, based on slice_height. It crashes on
copying data for the second plane when it hits the actual end of the data and
starts accessing addresses beyond that.

Instead of using slice_height here, change to use use height. slice_height is
used at other places in this module and I do not know if they also need to be
changed. I have confirmed that with this change, decoding works correctly
on android oreo as well as on the prior version, android nougat.

Refs #13233

[Peter Bennett <pbennett@…>]

In 11d426bd49/mythtv:

Playback: Support for mediacodec hardware decoding on android

Mediacodec is used for hardware decoding on android. Add support for
Mediacodect during playback, using FFmpeg, if enabled in configure.
Default to enabled when building for android.

Refs #13233

[Peter Bennett <pbennett@…>]

In 8e34e6e55/mythtv:

Playback: Prerequisites to support for mediacodec

Mediacodec is used for hardware decoding on android. It works differently
from other decoding methods and needs several changes to support the different flow.

1. On NVidia Shield, mediacodec provides built in deinterlace with frame doubling. This causes a discrepancy between reported framerate and actual framerate. Changes are made to support detecting and using the double framerate.
2. Up to now there was always a maximum of one frame released after a packet was provided to the decoder. Mediacodec releases many frames at a time, typically 15 frames at once after 15 or more packets were sent. Many changes are needed to support this.
3. In conjunction with the previous change, FFmpeg recommend that the code always read all frames available before sending the next packet. Previously it would read one frame before sending the next packet. Change to support this.
4. The existing code uses arcane methods to calculate frame timing from the packet sent. This causes havoc with mediacodec because of the multiple frames per packet and the frame doubling. FFmpeg alreay does this work for you, correctly. Change to use ffmpeg frame timing for ffmeg and mediacodec. Other decoding methods probably should change as well, but that is not done here.
5. When skipping forward or back with doubled framerate from mediacodec, one frame is presented that is a combination of old and new frame. Change to drop the first 4 frames after skipping with doubled framerate to avoid this.
6. With mediacodec there is sometimes a situation where the decoder will neither accept a packet nor provide a frame. According to FFmpeg documentation this will never occur, but it does occur. A short wait and retry in this case solves the problem.

Refs #13233

[Peter Bennett <pbennett@…>]

In 88eb4ef87/mythtv:

FFmpeg avcodec: Fix segmentation fault with decoding on android oreo

avcodec_receive_frame consistently causes a seg fault when decoding 1080i mpeg2
on android version oreo. When copying the frame, the second plane in the buffer
follows on immediately after 1080 lines of the first plane, but the code assumes
it is after 1088 lines of the first plane, based on slice_height. It crashes on
copying data for the second plane when it hits the actual end of the data and
starts accessing addresses beyond that.

Instead of using slice_height here, change to use use height. slice_height is
used at other places in this module and I do not know if they also need to be
changed. I have confirmed that with this change, decoding works correctly
on android oreo as well as on the prior version, android nougat.

Refs #13233

[Peter Bennett <pbennett@…>]

In 11d426bd49/mythtv:

Playback: Support for mediacodec hardware decoding on android

Mediacodec is used for hardware decoding on android. Add support for
Mediacodect during playback, using FFmpeg, if enabled in configure.
Default to enabled when building for android.

Refs #13233

[Peter Bennett <pbennett@…>]

In 8e34e6e55/mythtv:

Playback: Prerequisites to support for mediacodec

Mediacodec is used for hardware decoding on android. It works differently
from other decoding methods and needs several changes to support the different flow.

1. On NVidia Shield, mediacodec provides built in deinterlace with frame doubling. This causes a discrepancy between reported framerate and actual framerate. Changes are made to support detecting and using the double framerate.
2. Up to now there was always a maximum of one frame released after a packet was provided to the decoder. Mediacodec releases many frames at a time, typically 15 frames at once after 15 or more packets were sent. Many changes are needed to support this.
3. In conjunction with the previous change, FFmpeg recommend that the code always read all frames available before sending the next packet. Previously it would read one frame before sending the next packet. Change to support this.
4. The existing code uses arcane methods to calculate frame timing from the packet sent. This causes havoc with mediacodec because of the multiple frames per packet and the frame doubling. FFmpeg alreay does this work for you, correctly. Change to use ffmpeg frame timing for ffmeg and mediacodec. Other decoding methods probably should change as well, but that is not done here.
5. When skipping forward or back with doubled framerate from mediacodec, one frame is presented that is a combination of old and new frame. Change to drop the first 4 frames after skipping with doubled framerate to avoid this.
6. With mediacodec there is sometimes a situation where the decoder will neither accept a packet nor provide a frame. According to FFmpeg documentation this will never occur, but it does occur. A short wait and retry in this case solves the problem.

Refs #13233

[Peter Bennett <pbennett@…>]

In 88eb4ef87/mythtv:

FFmpeg avcodec: Fix segmentation fault with decoding on android oreo

avcodec_receive_frame consistently causes a seg fault when decoding 1080i mpeg2
on android version oreo. When copying the frame, the second plane in the buffer
follows on immediately after 1080 lines of the first plane, but the code assumes
it is after 1088 lines of the first plane, based on slice_height. It crashes on
copying data for the second plane when it hits the actual end of the data and
starts accessing addresses beyond that.

Instead of using slice_height here, change to use use height. slice_height is
used at other places in this module and I do not know if they also need to be
changed. I have confirmed that with this change, decoding works correctly
on android oreo as well as on the prior version, android nougat.

Refs #13233

[Peter Bennett <pbennett@…>]

In 11d426bd49/mythtv:

Playback: Support for mediacodec hardware decoding on android

Mediacodec is used for hardware decoding on android. Add support for
Mediacodect during playback, using FFmpeg, if enabled in configure.
Default to enabled when building for android.

Refs #13233

[Peter Bennett <pbennett@…>]

In 4287f45fd6/mythtv:

Mediacodec: Remove workaround that does not solve issue

During forward and back skip, sometimes one corrupted frame causes
a flash. The workaround I previously put in place (drop 4 frames)
does not prevent this so this commit removes that workaround.

Refs #13233