Opened 5 years ago

Closed 4 years ago

#13446 closed Developer Task (Fixed)

Android Digital Audio

Reported by: Peter Bennett Owned by: Peter Bennett
Priority: minor Milestone: 31.0
Component: Ports - Android Version: Master Head
Severity: medium Keywords:
Cc: Ticket locked: no

Description (last modified by Peter Bennett)

The Android port uses OpenSL ES for audio. However this only caters for stereo. AC3 passthrough seems to work intermittently on Shield for some sound systems, however it does not work with Fire stick and likely will not work with others.

I plan to implement a new audio method using android.media.AudioTrack. This allows audio with many encodings including AC3, DTS, E_AC3 to be decoded or passed through based on the capabilities of the system.

Attachments (3)

20190504_1902_audiotrack.patch (27.7 KB) - added by Peter Bennett 5 years ago.
Add "AudioTrack?" Audio playback capability for Android. Note that to build for android you need a change to the packaging repository as well
20190505_1106_audiotrack_packaging.patch (710 bytes) - added by Peter Bennett 5 years ago.
Patch to packaging repository to support building the audiotrack code.
20190512_1505_working_latency_cleaned.patch (29.3 KB) - added by Peter Bennett 5 years ago.
AudioTrack code fixed to handle device latency. This patch is for the mythtv repository. You also need the packaging repository patch above to build.

Download all attachments as: .zip

Change History (25)

comment:1 Changed 5 years ago by Peter Bennett

Description: modified (diff)

comment:2 Changed 5 years ago by Peter Bennett

Milestone: needs_triageunknown
Type: Bug Report - GeneralDeveloper Task

comment:3 Changed 5 years ago by Peter Bennett

I have AudioTrack audio working and playing digital passthrough, digital hardware AC3 decode and normal stereo using existing MythTV ffmpeg decode. However the hardware decode has a 2000 ms latency and the digital passthrough also a large latency. The existing MythTV code for handling the latency is not working, and audio sync is all over the place so this is proving to be a challenge.

comment:4 Changed 5 years ago by Peter Bennett

I am making slow progress. Ac3 is working now, but time stretch is behaving weirdly.

Latency was not the problem, there is not a 2000 ms latency.

Changed 5 years ago by Peter Bennett

Add "AudioTrack?" Audio playback capability for Android. Note that to build for android you need a change to the packaging repository as well

Changed 5 years ago by Peter Bennett

Patch to packaging repository to support building the audiotrack code.

comment:5 Changed 5 years ago by Peter Bennett

Add "AudioTrack" Audio playback capability for Android

Add a new audio output module for android devices. This supports digital passthrough as well as multi-channel LPCM. It also supports hardware accelerated decoding of media streams if your sound system does not support them.

At this time there is a restriction that time stretch will use stereo or multi-channel LPCM. It does not support AC3 output in time stretch mode.

Internal volume control is not supported via AudioTrack. If software volume control is used, AC3 output is disabled.

java code

I have added a java module in a new directory "android-package-source". This is merged during android packaging with the packaging directory "android-package-source". I propose that any java code that is by nature application code should be in the mythtv repository. I suggest that the other two java modules also be moved to the mythtv directory, for consistency to keep the code together.

To use AudioTrack on Android

In the Android setup, choose "Automatic" or "Best Available". There is no need to force Dolby Digital in Android, it will be used automatically if your hardware supports it.

Select "AudioTrack:" audio device. Enable all entries under "Digital Capabilities" and set the appropriate number of speakers. Android will detect whether your TV or sound system supports the audio format and will either pass it through or decode it within android hardware. So for example you can select Dolby Digital even if your sound system, TV or display monitor does not support it and android will decode it. If you do not select the checkbox, MythTV will decode it.

I have tested Dolby Digital (AC-3). I cannot test multi channel LPCM, DTS, E-AC3, etc. If anybody can test these it would be appreciated. In particular, testing multi-channel LPCM is needed to make sure the channel mapping is correct.

Let me know if there is any feedback.

comment:7 Changed 5 years ago by Peter Bennett

I have found that I have to set audio sync to -300 with AudioTrack (fire stick 4K), playing to a TV without surround sound system. With OpenSLES it was -150.

I need to add code to detect the latency and apply the appropriate adjustment automatically, as manually adding such a large sync adjustment should not be necessary.

comment:8 Changed 5 years ago by jpilk

I had understood that an a/v sync setting, once made, would apply to all playback by mythfrontend. Perhaps this has changed recently, and I am currently away from my firestick, so can't re-check. But an accurate auto-setting would certainly be good.

IIRC VLC needs it on a per-file basis.

Changed 5 years ago by Peter Bennett

AudioTrack code fixed to handle device latency. This patch is for the mythtv repository. You also need the packaging repository patch above to build.

comment:9 Changed 5 years ago by Peter Bennett

Updated patch fixes synchronization. You will need to revert to audio sync of 0 if you changed it for the previous patch. Also you may need to increase audio read ahead if the audio cuts out (Frontend->Setup->video->playback->advanced->audio read ahead). With Shield I had to set this to around 1000.

comment:11 Changed 5 years ago by Peter Bennett

The changes are now committed to master. Packages are avaialable at https://dl.bintray.com/bennettpeter/generic/

comment:12 Changed 5 years ago by mspieth

I have been testing audiotrack. A few observations on nvshield:

  • on playback and press back/escape to get the quit menu, myth locks up. Not sure why yet. It times out eventually.
  • In audio setup, selected 5.1 with upconvert best, then test speakers, surround left and right come out of front left and right, and lfe comes out of center. Also not sure why. Could be my setup (shield (hdmi) -> TV -> AVAmp. This setup works with a PC in passthrough mode (spdif).
  • on skip (e.g. ads), some audio from the part before skip gets played before the new audio is played. No sync problems when this occurs. Sounds like the part buffered in android audiotrack is not cleared when the rest is cleared.
  • After trying audiotrack, opensles no longer works (stereo in speaker test).

I'll have a look at (some of) these tomorrow.

Last edited 5 years ago by mspieth (previous) (diff)

comment:13 Changed 5 years ago by Peter Bennett

There appears to be an issue with AudioTrack with system hanging. It looks like there is an intermittent hang if you exit playback while paused. I recommend changing back to !OpenSLES until I can resolve this. !OpenSLES should still work as before even with the AudioTrack code present. If anybody notices these hangs while !OpenSLES is selected please let me know.

Regarding upconvert, we may need some code to rearrange the audio tracks appropriately. There is a feature to do that in android (setChannelIndexMask) but that requires api version 23. Not sure if we can use that.

Regarding speaker test, in my experience speaker test never worked for !OpenSLES. In my testing, switching from AudioTrack to !OpenSLES for playback is fine, but audio sync may need to be adjusted.

comment:14 Changed 5 years ago by Peter Bennett

I have committed a fix in master for a deadlock which occurred with AudioTrack? when exiting playback while paused.

comment:15 Changed 5 years ago by Peter Bennett

comment:16 Changed 5 years ago by mspieth

Standard channel order for 5.1 is L,R,C,LFE,RL,RR

Not sure why you used SL and SR. If you look at the constants in android 5.1 == 0xfc. I swapped them but it didnt make any difference. I think proper digital enc mode is required for all this to work.

There is no need for the DISABLE_AC3 if get options reports no digital support. Suggest AudioTrack:Digital and AudiTrack:PCM modes to allow selection. ALSA does this.

Also no LFE or rear with AudioTrack? for me currently but its ok in SLES mode due to passive matrix decoding. I suspect PCM mode with HDMI does not support multi channel.

comment:17 Changed 5 years ago by Peter Bennett <pbennett@…>

In f3c90896b/mythtv:

Add "AudioTrack?" Audio playback capability for Android

Add a new audio output module for android devices. This supports
digital passthrough as well as multi-channel LPCM. It also supports
hardware accelerated decoding of media streams if your sound system
does not support them.

At this time there is a restriction that time stretch will use
stereo or multi-channel LPCM. It does not support AC3 output
in time stretch mode.

Internal volume control is not supported via AudioTrack?. If software
volume control is used, AC3 output is disabled.

Refs #13446

comment:18 Changed 5 years ago by Peter Bennett <pbennett@…>

In af05a71bc/mythtv:

Android: Fix deadlock in AudioTrack? processing

When exiting during a pause, there was an infinite wait for the
output thread to end if there was data buffered. Now fixed.

Refs #13446

comment:19 Changed 5 years ago by Peter Bennett

Channels - You are right. I did not see the 5POINT1 setting. CHANNEL_OUT_5POINT1 consists of CHANNEL_OUT_BACK_LEFT, CHANNEL_OUT_BACK_RIGHT, CHANNEL_OUT_FRONT_CENTER, CHANNEL_OUT_FRONT_LEFT, CHANNEL_OUT_FRONT_RIGHT, CHANNEL_OUT_LOW_FREQUENCY.

I had thought that 5.1 had side speakers and 7.1 added the rear ones. I guess that is wrong. I will get that fixed.

Regarding DISABLE_AC3 I assume you are referring to the DISABLE_AC3_ENCODE macro. This is a temporary fix because the audio encode was not working during speedup. I hope to get it sorted out. The basic problem is that the SPDIF encoding is done in android so it needs to be bypassed in the AC3 encoding done during speedup and I did not succeed in getting that bypass working. That macro causes the AC3 encoding to by bypassed so that it reverts to stereo during speedup.

I think that multi-channel LPCM is not working correctly due to incorrect ordering of channels. This is another thing that needs to be fixed. This is likely the cause of your LFE not working. Unfortunately I do not have a sound system capable of multi-channel LPCM.

comment:20 Changed 5 years ago by mspieth

My understanding is that there is no digital passthrough working at all ATM since GetOutputSettings? does not report as such. I removed the DISABLE_AC3_ENCODE define and no difference in functionality as expected.

I can see how to do passthrough though. There are modes available such as ENCODING_AC3 which accept compressed digital streams, just set the bitrate correctly and pass the ac3 stream as encoded/reencoded and everything *SHOULD* just work.

I have not check the patches yet so you might have done this already and it didnt work.

comment:21 Changed 5 years ago by Peter Bennett <pbennett@…>

In b4cb676c94/mythtv:

android: Fix channel masks for AudioTrack?

Apply correct channel masks for the various surround sound options.

Refs #13446

comment:22 Changed 4 years ago by Peter Bennett

Milestone: unknown31.0
Resolution: Fixed
Status: assignedclosed
Version: UnspecifiedMaster Head
Note: See TracTickets for help on using tickets.