audio doesn't work with game-activity

[kyoobey]

i tried to get cpal working but couldn't do so

edit:
code: https://github.com/tshrpl/duel (run the game crate)

[rib]

Just to check - is cpal working ok with this app when run as a desktop app?

[kyoobey]

yes, i haven't done any extensive testing on the audio engine, but it works on desktop

[rib]

okey, yeah, curious. I haven't played with Oboe/cpal on Android but have been using cpal in a NES emulator recently so would be curious to try.

[kyoobey]

please, my game would never be complete without audio

[rib]

Okey, I've just pushed an Oboe based example that demonstrates playing a 440Hz sine wave on Android, running with GameActivity: https://github.com/rib/android-activity/tree/main/examples/agdk-oboe

Testing that on my phone it seems to work.

Since cpal uses Oboe then I would hope that cpal should also work but I've only tried using Opal directly so far.

It would be interesting to hear if the above example works for you.

[kyoobey]

the oboe example works, i guess i must have messed up somewhere in my code

[rib]

not sure but maybe the next step should be to try modifying the minimal oboe example to use cpal, based on the cpal android example: https://github.com/RustAudio/cpal/blob/master/examples/android.rs and if that works maybe experiment with playing back your wav file instead of playing a generated sine wave.

[kyoobey]

i copied the cpal rust example directly into audio.rs, just renamed the main function to try_start and inserted std::thread::spawn(|| audio::try_start()); in MainEvent::Resume in lib.rs. it builds and the logs dont show any signigicant error expect this

INFO [android_debugger] RustStdoutStderr: thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\ndk-glue-0.6.2\src\lib.rs:60:39

i dont hear any sound

logs

DEBUG ANDROID_SDK_ROOT = C:\Users\hitus\scoop\apps\android-clt\current
DEBUG ANDROID_NDK_HOME = C:\Users\hitus\scoop\apps\android-clt\current\ndk\25.0.8775105
DEBUG adb: "C:\\Users\\hitus\\scoop\\apps\\android-clt\\current\\platform-tools\\adb.exe"
DEBUG ndk addr2line: "C:\\Users\\hitus\\scoop\\apps\\android-clt\\current\\ndk\\25.0.8775105\\toolchains\\llvm\\prebuilt\\windows-x86_64\\bin\\llvm-addr2line.exe"
DEBUG Debug file to analyze: "E:\\duel\\target\\aarch64-linux-android\\debug\\libmain.so"
DEBUG Package process id: "23675"
WARN BaseMiuiPhoneWindowManager: keyCode:24 down:true eventTime:23675641 downTime:23675641 policyFlags:22000000 flags:8 deviceId:8 isScreenOn:true keyguardActive:false repeatCount:0
WARN BaseMiuiPhoneWindowManager: keyCode:24 down:false eventTime:23675790 downTime:23675641 policyFlags:22000000 flags:8 deviceId:8 isScreenOn:true keyguardActive:false repeatCount:0
DEBUG Zygote  : Forked child process 23675
DEBUG OemNetd : setPidForPackage: packageName=co.realfit.agdkoboe, pid=23675, pid=10339
INFO ActivityManager: Start proc 23675:co.realfit.agdkoboe/u0a339 for activity {co.realfit.agdkoboe/co.realfit.agdkoboe.MainActivity} caller=com.miui.home
INFO ealfit.agdkobo: Late-enabling -Xcheck:jni
ERROR ealfit.agdkobo: Unknown bits set in runtime_flags: 0x8000
INFO ProcessMonitor: onForegroundInfoChanged: ForegroundInfo{mForegroundPackageName='co.realfit.agdkoboe', mForegroundUid=10339, mForegroundPid=23675, mLastForegroundPackageName='com.miui.home', mLastForegroundUid=10086, mLastForegroundPid=2365, mMultiWindowForegroundPackageName='null', mMultiWindowForegroundUid=-1, mFlags=1}
DEBUG PerfEngineController: ForegroundInfo{mForegroundPackageName='co.realfit.agdkoboe', mForegroundUid=10339, mForegroundPid=23675, mLastForegroundPackageName='com.miui.home', mLastForegroundUid=10086, mLastForegroundPid=2365, mMultiWindowForegroundPackageName='null', mMultiWindowForegroundUid=-1, mFlags=1}
DEBUG GameBoosterService: onGameStatusChange foreground:ForegroundInfo{mForegroundPackageName='co.realfit.agdkoboe', mForegroundUid=10339, mForegroundPid=23675, mLastForegroundPackageName='com.miui.home', mLastForegroundUid=10086, mLastForegroundPid=2365, mMultiWindowForegroundPackageName='null', mMultiWindowForegroundUid=-1, mFlags=1}
INFO Perf    : Connecting to perf service.
INFO FeatureParser: can't find olive.xml in assets/device_features/,it may be in /system/etc/device_features
INFO ealfit.agdkobo: Starting a blocking GC Alloc
INFO ealfit.agdkobo: Starting a blocking GC Alloc
INFO ealfit.agdkobo: Waiting for a blocking GC Alloc
INFO ealfit.agdkobo: WaitForGcToComplete blocked Alloc on ClassLinker for 16.015ms
INFO ealfit.agdkobo: Starting a blocking GC Alloc
DEBUG ForceDarkHelper: updateByCheckExcludeList: pkg: co.realfit.agdkoboe activity: co.realfit.agdkoboe.MainActivity@c9169f4
INFO chatty  : uid=10339(co.realfit.agdkoboe) identical 2 lines
DEBUG ForceDarkHelper: updateByCheckExcludeList: pkg: co.realfit.agdkoboe activity: co.realfit.agdkoboe.MainActivity@c9169f4
WARN ealfit.agdkobo: Accessing hidden method Landroid/view/View;->computeFitSystemWindows(Landroid/graphics/Rect;Landroid/graphics/Rect;)Z (greylist, reflection, allowed)
WARN ealfit.agdkobo: Accessing hidden method Landroid/view/ViewGroup;->makeOptionalFitsSystemWindows()V (greylist, reflection, allowed)
DEBUG GameActivity: GameActivity_register
DEBUG GameActivity: Registering com/google/androidgamesdk/GameActivity's 21 native methods...
DEBUG GameActivity: loadNativeCode_native
DEBUG threaded_app: Creating: 0x7a8f02f280
DEBUG threaded_app: Callbacks set: 0x7a8f02f2d0
DEBUG threaded_app: Launching android_app_entry in a thread
DEBUG threaded_app: android_app_entry called
DEBUG threaded_app: android_app = 0x7a8f05fc80
DEBUG threaded_app: config = 0x7ae59127c0
DEBUG threaded_app: activity = 0x7a8f02f280
DEBUG threaded_app: assetmanager = 0x7ae5877b00
DEBUG threaded_app: Config: mcc=405 mnc=855 lang=en cnt=IN orien=1 touch=3 dens=320 keys=1 nav=1 keysHid=3 navHid=0 sdk=29 size=2 long=2 modetype=1 modenight=2
DEBUG GameActivity: onStart_native
DEBUG threaded_app: Start: 0x7a8f02f280
DEBUG threaded_app: activityState=10
INFO main    : Main event: Start
DEBUG GameActivity: onResume_native
DEBUG threaded_app: Resume: 0x7a8f02f280
DEBUG threaded_app: activityState=11
INFO main    : Main event: Resume { loader: StateLoader { app: AndroidAppInner { ptr: 0x7a8f05fc80, config: RwLock { data: Configuration { mcc: 405, mnc: Some(855), lang: Some("en"), country: Some("IN"), orientation: Port, touchscreen: Finger, density: Some(320), keyboard: NoKeys, navigation: NoNav, keys_hidden: Soft, nav_hidden: Any, sdk_version: 29, screen_size: Normal, screen_long: Yes, ui_mode_type: Normal, ui_mode_night: Yes }, poisoned: false, .. }, native_window: RwLock { data: None, poisoned: false, .. } } } }
INFO RustStdoutStderr: thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', C:\Users\hitus\.cargo\registry\src\github.com-1ecc6299db9ec823\ndk-glue-0.6.2\src\lib.rs:60:39
INFO RustStdoutStderr: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
WARN Looper  : Slow Looper main: Activity co.realfit.agdkoboe/.MainActivity is 352ms late (wall=0ms running=0ms ClientTransaction{ callbacks=[android.app.servertransaction.TopResumedActivityChangeItem] }) because of 2 msg, msg 1 took 92ms (seq=2 late=1ms h=android.app.ActivityThread$H w=110), msg 2 took 276ms (seq=3 late=77ms h=android.app.ActivityThread$H w=159)
WARN ealfit.agdkobo: Accessing hidden method Landroid/view/View;->getViewRootImpl()Landroid/view/ViewRootImpl; (greylist, reflection, allowed)
WARN ealfit.agdkobo: Accessing hidden field Landroid/view/View$AttachInfo;->mVisibleInsets:Landroid/graphics/Rect; (greylist, reflection, allowed)
WARN ealfit.agdkobo: Accessing hidden field Landroid/view/ViewRootImpl;->mAttachInfo:Landroid/view/View$AttachInfo; (greylist, reflection, allowed)
DEBUG threaded_app: WindowInsetsChanged: 0x7a8f02f280
INFO main    : Main event: InsetsChanged
INFO AdrenoGLES: QUALCOMM build                   : 9d44e36, Ie3bb699d95
INFO AdrenoGLES: Build Date                       : 09/29/20
INFO AdrenoGLES: OpenGL ES Shader Compiler Version: EV031.27.05.02
INFO AdrenoGLES: Local Branch                     :
INFO AdrenoGLES: Remote Branch                    : refs/tags/AU_LINUX_ANDROID_LA.UM.8.6.2.R1.10.00.00.537.053
INFO AdrenoGLES: Remote Branch                    : NONE
INFO AdrenoGLES: Reconstruct Branch               : NOTHING
INFO AdrenoGLES: Build Config                     : S L 8.0.10 AArch64
DEBUG threaded_app: WindowInsetsChanged: 0x7a8f02f280
INFO main    : Main event: InsetsChanged
INFO AdrenoGLES: PFP: 0x005ff112, ME: 0x005ff066
DEBUG GameActivity: onSurfaceCreated_native
DEBUG GameActivity: onSurfaceCreated_native
DEBUG threaded_app: NativeWindowCreated: 0x7a8f02f280 -- 0x7a8f14e010
DEBUG threaded_app: android_app_set_window called
DEBUG threaded_app: APP_CMD_INIT_WINDOW
DEBUG GameActivity: onSurfaceChanged_native
INFO main    : Main event: InitWindow
INFO main    : Render...
DEBUG threaded_app: NativeWindowResized: 0x7a8f02f280 -- 0x7a8f14e010 ( 720 x 1465 )
INFO main    : Main event: WindowResized
INFO main    : Render...
DEBUG GameActivity: onSurfaceRedrawNeeded_native
DEBUG threaded_app: NativeWindowRedrawNeeded: 0x7a8f02f280 -- 0x7a8f14e010
INFO main    : Main event: RedrawNeeded
INFO main    : Render...
WARN Gralloc3: mapper 3.x is not supported
DEBUG GameActivity: onWindowFocusChanged_native
DEBUG threaded_app: WindowFocusChanged: 0x7a8f02f280 -- 1
INFO main    : Main event: GainedFocus
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO ealfit.agdkobo: ProcessProfilingInfo new_methods=0 is saved saved_to_disk=0 resolve_classes_delay=8000
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
INFO main    : Render...
INFO main    : Timed out
(above two lines repeating ...)

[rib]

I just had a chance to give this a try too and for some reason it worked ok for me: https://github.com/rib/android-activity/pull/11

There was one recent cpal change to bump it up to ndk 0.7 that I saw but maybe there were also some other changes since last trying.

@tshrpl I wonder if you could confirm if the cpal example in the above branch works for you?

[rib]

oh, no, my bad I made a mistake with my testing and I actually get the same error that you got @tshrpl

[rib]

I also only just noticed that the panic is coming from ndk-glue! which is pretty weird here. As far as I could tell ndk-glue is only a dev dependency in cpal, which is just for their android example as far as I can tell.

This almost seems like a cargo bug to me, since dev-dependencies aren't supposed to be inherited by other packages.

If I explicitly remove the ndk-glue dev dependency from cpal then I do actually get some audio output though after about 1 second theres a different crash I haven't looked at yet.

I'm a bit unsure about the right way to deal with the first issue though since that seems to be caused by Cargo. Quoting from the Cargo reference manual it says:

Dev-dependencies are not used when compiling a package for building, but are used for compiling tests, examples, and benchmarks.

These dependencies are not propagated to other packages which depend on this package.

This crash seems to be caused by a violation of that last point because the cdylib we're building in one package is pulling in ndk-glue via a dev-dependency of another package.

[rib]

For the second crash I see - I'm guessing at the moment that there is something unsafe/unsound with the cpal bindings, since I can certainly imagine various ways this crude example code might trigger a use-after-free. I suppose the cpal bindings over oboe might not be that safe.

I'm guessing it'll be much more likely to work by moving the device up into the event loop/main scope instead of trying to synchronously create a temporary device and play audio for one second and hope that it all cleans up neatly when dropped.

I'm guessing there's asynchronous work oboe is still trying to do to play audio samples while all the cpal state is being dropped which leads to the crash.

[rib]

To help workaround this issue, I've created a branch of cpal that just removes the ndk-glue dev dependency: https://github.com/rib/cpal/tree/no-ndk-glue-dev-dependency

[rib]

For reference, I've updated the cpal example in the PR: https://github.com/rib/android-activity/pull/11 so it now uses the above workaround branch of cpal and the code has been re-worked to find a device and create a stream up-front.

I'm finding that it's still crashy when suspending and resuming the app but it's better than it was before :) I think maybe the cpal bindings might just be a bit buggy/unsafe at the moment - maybe since they aren't heavily tested on Android currently.

[rib]

For reference here, I've filed a Cargo issue since it really seems to be going against the documentation for how dev-dependencies are intended to work: rust-lang/cargo#10980

[rib]

Incidentally; if I go back to check the minimal agdk-oboe example, that also crashes when pausing the app, and from seeing this issue too: katyo/oboe-rs#41 I wonder now whether the crash here (not the one related to ndk-glue) is going to turn out to be an oboe issue not a cpal issue.

[rib]

oh, actually this issue is sounding more relevant: katyo/oboe-rs#32

[rib]

okey, so it turned out second crash I was chasing (not the ndk-glue one) was a bit of a red-herring. There was a more general bug causing a crash in all the examples when suspending + resuming, which was caused by an ndk 0.7 change that updated NativeWindow to implement Drop, which now unconditionally calls _release on the underlying ANativeWindow when the wrapper type is dropped. I've just pushed a fix for this to main: rib@93828a1

With that fix then I see that the cpal example seems to work ok now (still with the workaround for the Cargo/ndk-glue issue) so I'll probably merge the PR for the cpal example now.

[rib]

Okey, although we still have this awkward situation where we need a workaround in cpal to avoid the ndk-glue dev dependency we now have cpal and oboe examples that seem to at least be able to play a simple tone which hopefully at least validates that they are usable with android-activity.

ref: https://github.com/rib/android-activity/pull/11

[rib]

As a quick follow up here - it turned out I was jumping the gun with thinking there was an issue with cpal leaking its dev dependency - actually the 0.13.5 version of cpal has ndk-glue as a regular dependency, not just a dev-dependency so when I commented out the dev-dependency and found it worked that was actually redundant and it also works fine to just use cpal master instead of the released 0.13.5 version.

Unfortunately they haven't done a release yet since that change so the cpal example will still have to fetch cpal from git for now.