Opened 22 months ago

Closed 15 months ago

#13233 closed Developer Task (Fixed)

Android Hardware accelerated decode

Reported by: Peter Bennett Owned by: Peter Bennett
Priority: minor Milestone: unknown
Component: Ports - Android Version: Master Head
Severity: medium Keywords:
Cc: Ticket locked: no

Description

Android devices support mediacodec. Add support to MythTV for accelerated decoding.

This may be a lot of work and a long term project.

Attachments (13)

20180614_1724_mediacodec_and_ffmpeg.patch (41.6 KB) - added by Peter Bennett 18 months ago.
Preliminary code with mediacodec support
20180617_mediacodec_fix_reschange.patch (42.8 KB) - added by Peter Bennett 18 months ago.
Added fix for resolution changes and Aman's deinterlace improvement
20180617_1712_medaicodec_fixed_openmax.patch (39.3 KB) - added by Peter Bennett 18 months ago.
New version of patch. Cleaned up unnecessary changes and fixed raspberry pi openmax
20180614_1707_ffmpeg_aman_revert.patch (1.7 KB) - added by Peter Bennett 18 months ago.
Aman's not yet released ffmpeg fix. Apply separately from other patch.
20180620_1715_mediacodec_pause_fixed.patch (40.1 KB) - added by Peter Bennett 18 months ago.
Playback Pause fixed. Still need to fix jump, FF, REW
20180623_1115_mediacodec_fixed_jump_but_time_bug.patch (40.2 KB) - added by Peter Bennett 18 months ago.
Updated patch fixes mediacodec jump and skip
20180628_1445_timing_fixed.patch (43.0 KB) - added by Peter Bennett 18 months ago.
Fixed playback elapsed time and associated things
20180629_ff_rew_fixed.patch (43.6 KB) - added by Peter Bennett 18 months ago.
Fixes Fast Forward and Rewind with Mediacodec
20180705_1535_segfault_fix.patch (46.7 KB) - added by Peter Bennett 17 months ago.
Fixes segfault, FF and REW and reduces buffers to help with 4K playback
20180710_1734_jump_speed.patch (48.6 KB) - added by Peter Bennett 17 months ago.
A couple of fixes to jump and speed
20180716_1139_smoooooth.patch (49.8 KB) - added by Peter Bennett 17 months ago.
Smooth playing with speedup
20180717_1042_partial_livetv_fix.patch (51.2 KB) - added by Peter Bennett 17 months ago.
Fix LiveTV crash
20180717_1551_livetv.patch (52.1 KB) - added by Peter Bennett 17 months ago.
Fixes Live TV

Download all attachments as: .zip

Change History (43)

comment:1 Changed 20 months ago by Peter Bennett

Owner: changed from Peter Bennett to Peter Bennett
Status: newassigned

Changed 18 months ago by Peter Bennett

Preliminary code with mediacodec support

comment:2 Changed 18 months ago by Peter Bennett

Attached is preliminary in-progress code for MediaCodec. It still needs to be cleaned up. Not ready for commit.

Select MediaCodec Normal playback profile. MediaCodec hardware assistance is working. Leave deinterlace as None since the Shield already deinterlaces.

Not working:

  • ATSC subtitles not working.
  • Video resolution change during playback not working.

To be done:

  • Aman suggested a change to help with interlaced content. Not yet done, although deinterlacing seems to be perfect anyway.
  • Add a limit counter on retry of sending packet so as to avoid a possible infinite loop.

To Be Decided: For MediaCodec and software decoding (standard) it is now bypassing existing convoluted timing code and now is relying on ffmpeg to do timing calculations. I am not sure whether that change should apply to all decoders, MediaCodec only, decoders that create multiple frames from 1 packet, standard decoders, private decoders, or some combination. If it applies to all, the convoluted code can be deleted.

Last edited 18 months ago by Peter Bennett (previous) (diff)

Changed 18 months ago by Peter Bennett

Added fix for resolution changes and Aman's deinterlace improvement

comment:3 Changed 18 months ago by Peter Bennett

A new preliminary in-progress patch. (20180617_mediacodec_fix_reschange.patch​). It includes the previous patch and adds to it.

Fixes:

  • Video resolution change during playback
  • Change suggested by Aman for interlaced video
  • Added limit counter to prevent endless loop.

Changed 18 months ago by Peter Bennett

New version of patch. Cleaned up unnecessary changes and fixed raspberry pi openmax

comment:4 Changed 18 months ago by Peter Bennett

Latest patch cleans up some unneeded changes. Also the previous patches had broken private decoders including raspberry pi. This fixes that as well. I have removed the ffmpeg change that I got from Aman, waiting for it to be in the official FFmpeg release. It did not seem to make any difference anyway.

Changed 18 months ago by Peter Bennett

Aman's not yet released ffmpeg fix. Apply separately from other patch.

comment:5 Changed 18 months ago by Peter Bennett

Apply this patch with git apply -v --dir mythtv/external/FFmpeg filename.patch

Changed 18 months ago by Peter Bennett

Playback Pause fixed. Still need to fix jump, FF, REW

comment:6 Changed 18 months ago by Peter Bennett

Latest patch

Fix the PAUSE deadlock by allowing decoder pause during processing of a packet. Packet processing is successfully resumed at end of pause.

This does not yet address FF, REW or jump.

Changed 18 months ago by Peter Bennett

Updated patch fixes mediacodec jump and skip

comment:7 Changed 18 months ago by Peter Bennett

20180623_1115_mediacodec_fixed_jump_but_time_bug.patch fixes jump and skip.

There is a bug where the elapsed time progresses at double speed with interlaced content on Nvidia shield, because of the type of deinterlace the shield performs.

Does not yet address FF and REW.

Changed 18 months ago by Peter Bennett

Fixed playback elapsed time and associated things

comment:8 Changed 18 months ago by Peter Bennett

Patch 20180628_1445_timing_fixed.patch​ fixes the displayed elapsed time, and along with that, the forward and back skip will use correct positioning.

What the code now does is

  1. Detect double frame rate from the pts timings coming out of the decoder.
  2. If there is double frame rate, then increment the frame number only on every second frame.

Changed 18 months ago by Peter Bennett

Attachment: 20180629_ff_rew_fixed.patch added

Fixes Fast Forward and Rewind with Mediacodec

comment:9 Changed 18 months ago by Peter Bennett

This patch, 20180629_ff_rew_fixed.patch​ , includes a fix for fast forward and rewind.

Note that all patches are cumulative, i.e. contain all prior patches and are applied to a clean copy of master source.

Changed 17 months ago by Peter Bennett

Fixes segfault, FF and REW and reduces buffers to help with 4K playback

comment:10 Changed 17 months ago by Peter Bennett

A patch to ffmpeg fixes the segfault in Oreo with 1080 playback. It is likely that this patch causes a segfault in Nougat, it seems the interface may have changed.

Also fixes FF and REW by treating the illegalstate exception as it is was a EAGAIN return.

Reduces frame buffers from 31 to 4, thus reducing the memory required to play 4K content by 320MB. This may cause jerky playback, we may need to check and tune this further.

Last edited 17 months ago by Peter Bennett (previous) (diff)

Changed 17 months ago by Peter Bennett

A couple of fixes to jump and speed

comment:11 Changed 17 months ago by Peter Bennett

Details of patch 20180710_1734_jump_speed.patch :

  1. Fix the corrupt frame during jump by dropping 4 frames after the avcodec_flush_buffers, if there is an fps doubling in effect (i.e. deinterlacing via mediacodec)
  2. I found that if you disable the audio the interlaced video played at half speed. Fixed that by fixing the frame interval calculation.
  3. I found that the "Generic error in an external library" still happens during FF. Changed so that the video codec is re-initialized when that happens. This causes a pause in the FF operation of a few seconds, but is better than the long wait and failure message that happens otherwise.

Changed 17 months ago by Peter Bennett

Smooth playing with speedup

comment:12 Changed 17 months ago by Peter Bennett

Patch 20180716_1139_smoooooth.patch:

Smooth playback with speedup by changing the calculation of frame_interval to take into account fps doubling.

Setting of vbuffers when using mediacodec to 8 buffers.

Changed 17 months ago by Peter Bennett

Fix LiveTV crash

comment:13 Changed 17 months ago by Peter Bennett

Patch: 20180717_1042_partial_livetv_fix.patch

Fixes LiveTV crash. LiveTV still has a problem: it is basically a slide show until you press pause for a few seconds and play again.

Changed 17 months ago by Peter Bennett

Attachment: 20180717_1551_livetv.patch added

Fixes Live TV

comment:14 Changed 17 months ago by Peter Bennett

Patch 20180717_1551_livetv.patch​

Fixes LiveTV glitching at start of play

comment:15 Changed 17 months ago by Peter Bennett

These changes (latest patch) have been pushed to git.

Last edited 17 months ago by Peter Bennett (previous) (diff)

comment:16 Changed 17 months ago by Peter Bennett

A remaining problem is that Fast Forward and Rewind experience frequent failures (IllegalStateException? from the mediacodec system), which show as pauses during the operation. These seem to get worse with time. A possible solution is to use software decoding for fast forward and rewind. A better solution would be a fix for the IllegalStateException? that is thrown.

comment:17 Changed 17 months ago by Peter Bennett <pbennett@…>

In 8e34e6e55/mythtv:

Playback: Prerequisites to support for mediacodec

Mediacodec is used for hardware decoding on android. It works differently
from other decoding methods and needs several changes to support the different flow.

  1. On NVidia Shield, mediacodec provides built in deinterlace with frame doubling. This causes a discrepancy between reported framerate and actual framerate. Changes are made to support detecting and using the double framerate.
  2. Up to now there was always a maximum of one frame released after a packet was provided to the decoder. Mediacodec releases many frames at a time, typically 15 frames at once after 15 or more packets were sent. Many changes are needed to support this.
  3. In conjunction with the previous change, FFmpeg recommend that the code always read all frames available before sending the next packet. Previously it would read one frame before sending the next packet. Change to support this.
  4. The existing code uses arcane methods to calculate frame timing from the packet sent. This causes havoc with mediacodec because of the multiple frames per packet and the frame doubling. FFmpeg alreay does this work for you, correctly. Change to use ffmpeg frame timing for ffmeg and mediacodec. Other decoding methods probably should change as well, but that is not done here.
  5. When skipping forward or back with doubled framerate from mediacodec, one frame is presented that is a combination of old and new frame. Change to drop the first 4 frames after skipping with doubled framerate to avoid this.
  6. With mediacodec there is sometimes a situation where the decoder will neither accept a packet nor provide a frame. According to FFmpeg documentation this will never occur, but it does occur. A short wait and retry in this case solves the problem.

Refs #13233

comment:18 Changed 17 months ago by Peter Bennett <pbennett@…>

In 88eb4ef87/mythtv:

FFmpeg avcodec: Fix segmentation fault with decoding on android oreo

avcodec_receive_frame consistently causes a seg fault when decoding 1080i mpeg2
on android version oreo. When copying the frame, the second plane in the buffer
follows on immediately after 1080 lines of the first plane, but the code assumes
it is after 1088 lines of the first plane, based on slice_height. It crashes on
copying data for the second plane when it hits the actual end of the data and
starts accessing addresses beyond that.

Instead of using slice_height here, change to use use height. slice_height is
used at other places in this module and I do not know if they also need to be
changed. I have confirmed that with this change, decoding works correctly
on android oreo as well as on the prior version, android nougat.

Refs #13233

comment:19 Changed 17 months ago by Peter Bennett <pbennett@…>

In 11d426bd49/mythtv:

Playback: Support for mediacodec hardware decoding on android

Mediacodec is used for hardware decoding on android. Add support for
Mediacodect during playback, using FFmpeg, if enabled in configure.
Default to enabled when building for android.

Refs #13233

comment:20 Changed 17 months ago by Peter Bennett <pbennett@…>

In 8e34e6e55/mythtv:

Playback: Prerequisites to support for mediacodec

Mediacodec is used for hardware decoding on android. It works differently
from other decoding methods and needs several changes to support the different flow.

  1. On NVidia Shield, mediacodec provides built in deinterlace with frame doubling. This causes a discrepancy between reported framerate and actual framerate. Changes are made to support detecting and using the double framerate.
  2. Up to now there was always a maximum of one frame released after a packet was provided to the decoder. Mediacodec releases many frames at a time, typically 15 frames at once after 15 or more packets were sent. Many changes are needed to support this.
  3. In conjunction with the previous change, FFmpeg recommend that the code always read all frames available before sending the next packet. Previously it would read one frame before sending the next packet. Change to support this.
  4. The existing code uses arcane methods to calculate frame timing from the packet sent. This causes havoc with mediacodec because of the multiple frames per packet and the frame doubling. FFmpeg alreay does this work for you, correctly. Change to use ffmpeg frame timing for ffmeg and mediacodec. Other decoding methods probably should change as well, but that is not done here.
  5. When skipping forward or back with doubled framerate from mediacodec, one frame is presented that is a combination of old and new frame. Change to drop the first 4 frames after skipping with doubled framerate to avoid this.
  6. With mediacodec there is sometimes a situation where the decoder will neither accept a packet nor provide a frame. According to FFmpeg documentation this will never occur, but it does occur. A short wait and retry in this case solves the problem.

Refs #13233

comment:21 Changed 17 months ago by Peter Bennett <pbennett@…>

In 88eb4ef87/mythtv:

FFmpeg avcodec: Fix segmentation fault with decoding on android oreo

avcodec_receive_frame consistently causes a seg fault when decoding 1080i mpeg2
on android version oreo. When copying the frame, the second plane in the buffer
follows on immediately after 1080 lines of the first plane, but the code assumes
it is after 1088 lines of the first plane, based on slice_height. It crashes on
copying data for the second plane when it hits the actual end of the data and
starts accessing addresses beyond that.

Instead of using slice_height here, change to use use height. slice_height is
used at other places in this module and I do not know if they also need to be
changed. I have confirmed that with this change, decoding works correctly
on android oreo as well as on the prior version, android nougat.

Refs #13233

comment:22 Changed 17 months ago by Peter Bennett <pbennett@…>

In 11d426bd49/mythtv:

Playback: Support for mediacodec hardware decoding on android

Mediacodec is used for hardware decoding on android. Add support for
Mediacodect during playback, using FFmpeg, if enabled in configure.
Default to enabled when building for android.

Refs #13233

comment:23 Changed 17 months ago by Peter Bennett <pbennett@…>

In 8e34e6e55/mythtv:

Playback: Prerequisites to support for mediacodec

Mediacodec is used for hardware decoding on android. It works differently
from other decoding methods and needs several changes to support the different flow.

  1. On NVidia Shield, mediacodec provides built in deinterlace with frame doubling. This causes a discrepancy between reported framerate and actual framerate. Changes are made to support detecting and using the double framerate.
  2. Up to now there was always a maximum of one frame released after a packet was provided to the decoder. Mediacodec releases many frames at a time, typically 15 frames at once after 15 or more packets were sent. Many changes are needed to support this.
  3. In conjunction with the previous change, FFmpeg recommend that the code always read all frames available before sending the next packet. Previously it would read one frame before sending the next packet. Change to support this.
  4. The existing code uses arcane methods to calculate frame timing from the packet sent. This causes havoc with mediacodec because of the multiple frames per packet and the frame doubling. FFmpeg alreay does this work for you, correctly. Change to use ffmpeg frame timing for ffmeg and mediacodec. Other decoding methods probably should change as well, but that is not done here.
  5. When skipping forward or back with doubled framerate from mediacodec, one frame is presented that is a combination of old and new frame. Change to drop the first 4 frames after skipping with doubled framerate to avoid this.
  6. With mediacodec there is sometimes a situation where the decoder will neither accept a packet nor provide a frame. According to FFmpeg documentation this will never occur, but it does occur. A short wait and retry in this case solves the problem.

Refs #13233

comment:24 Changed 17 months ago by Peter Bennett <pbennett@…>

In 88eb4ef87/mythtv:

FFmpeg avcodec: Fix segmentation fault with decoding on android oreo

avcodec_receive_frame consistently causes a seg fault when decoding 1080i mpeg2
on android version oreo. When copying the frame, the second plane in the buffer
follows on immediately after 1080 lines of the first plane, but the code assumes
it is after 1088 lines of the first plane, based on slice_height. It crashes on
copying data for the second plane when it hits the actual end of the data and
starts accessing addresses beyond that.

Instead of using slice_height here, change to use use height. slice_height is
used at other places in this module and I do not know if they also need to be
changed. I have confirmed that with this change, decoding works correctly
on android oreo as well as on the prior version, android nougat.

Refs #13233

comment:25 Changed 17 months ago by Peter Bennett <pbennett@…>

In 11d426bd49/mythtv:

Playback: Support for mediacodec hardware decoding on android

Mediacodec is used for hardware decoding on android. Add support for
Mediacodect during playback, using FFmpeg, if enabled in configure.
Default to enabled when building for android.

Refs #13233

comment:26 Changed 17 months ago by Peter Bennett <pbennett@…>

In 8e34e6e55/mythtv:

Playback: Prerequisites to support for mediacodec

Mediacodec is used for hardware decoding on android. It works differently
from other decoding methods and needs several changes to support the different flow.

  1. On NVidia Shield, mediacodec provides built in deinterlace with frame doubling. This causes a discrepancy between reported framerate and actual framerate. Changes are made to support detecting and using the double framerate.
  2. Up to now there was always a maximum of one frame released after a packet was provided to the decoder. Mediacodec releases many frames at a time, typically 15 frames at once after 15 or more packets were sent. Many changes are needed to support this.
  3. In conjunction with the previous change, FFmpeg recommend that the code always read all frames available before sending the next packet. Previously it would read one frame before sending the next packet. Change to support this.
  4. The existing code uses arcane methods to calculate frame timing from the packet sent. This causes havoc with mediacodec because of the multiple frames per packet and the frame doubling. FFmpeg alreay does this work for you, correctly. Change to use ffmpeg frame timing for ffmeg and mediacodec. Other decoding methods probably should change as well, but that is not done here.
  5. When skipping forward or back with doubled framerate from mediacodec, one frame is presented that is a combination of old and new frame. Change to drop the first 4 frames after skipping with doubled framerate to avoid this.
  6. With mediacodec there is sometimes a situation where the decoder will neither accept a packet nor provide a frame. According to FFmpeg documentation this will never occur, but it does occur. A short wait and retry in this case solves the problem.

Refs #13233

comment:27 Changed 17 months ago by Peter Bennett <pbennett@…>

In 88eb4ef87/mythtv:

FFmpeg avcodec: Fix segmentation fault with decoding on android oreo

avcodec_receive_frame consistently causes a seg fault when decoding 1080i mpeg2
on android version oreo. When copying the frame, the second plane in the buffer
follows on immediately after 1080 lines of the first plane, but the code assumes
it is after 1088 lines of the first plane, based on slice_height. It crashes on
copying data for the second plane when it hits the actual end of the data and
starts accessing addresses beyond that.

Instead of using slice_height here, change to use use height. slice_height is
used at other places in this module and I do not know if they also need to be
changed. I have confirmed that with this change, decoding works correctly
on android oreo as well as on the prior version, android nougat.

Refs #13233

comment:28 Changed 17 months ago by Peter Bennett <pbennett@…>

In 11d426bd49/mythtv:

Playback: Support for mediacodec hardware decoding on android

Mediacodec is used for hardware decoding on android. Add support for
Mediacodect during playback, using FFmpeg, if enabled in configure.
Default to enabled when building for android.

Refs #13233

comment:29 Changed 17 months ago by Peter Bennett <pbennett@…>

In 4287f45fd6/mythtv:

Mediacodec: Remove workaround that does not solve issue

During forward and back skip, sometimes one corrupted frame causes
a flash. The workaround I previously put in place (drop 4 frames)
does not prevent this so this commit removes that workaround.

Refs #13233

comment:30 Changed 15 months ago by Peter Bennett

Resolution: Fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.