Evolution of the Android backend

[rib]

Hi,

Sorry in advance this is a fairly long read :) This this aims to open a discussion as much as it does try to highlight some practical issues I found recently while working with the Android backend. The winit Discussions don't seem super active and since I also had some practical issues to bring up I figured I'd post here, but happy to try and split this up if it makes sense...

I've recently been experimenting with an alternative "glue" layer for building native Rust applications on Android that's based on the GameActivity class (in turn based on the widely used AppCompatActivity class) that's part of Google's Android Game Development Kit:

Ref: https://developer.android.com/games/agdk
and https://developer.android.com/games/agdk/integrate-game-activity

Here's the glue layer I've implemented so far: https://github.com/rib/agdk-rust/tree/main/game-activity

The README there also has a bit more context, including a bit about my motivation for looking at this but essentially NativeActivity isn't practically usable in many circumstances (at least not without subclassing in Java to augment with additional functionality, but even then it's impossible to use AppCompatActivity which has become a very standard foundational class for Android development that provides numerous back-ported compatibility APIs on Android).

I originally aimed to base the game-activity glue API on ndk-glue but as I made progress I found I needed to diverge in a number of ways from the ndk-glue API, such as for robustly handling synchronization between the java main thread and native thread, and I also chose to define a standard extern "C" android_main() entry point ABI for applications that doesn't require any special macros.

So although I originally imagined I might be able to enable winit support simply by being a drop in alternative to ndk-glue I ended up working on more substantial winit backend changes to be able to build on this work further.

Just for reference at this point I've published these three minimal Android app examples based on this work, to test creating a winit + wgpu app and a winit-based egui app. I have an experimental Bevy app I've been poking at locally but for now Bevy doesn't really work on Android due to it's lack of handling of lifecycle events.

Please find my example test apps here:
https://github.com/rib/agdk-rust/tree/main/examples

Please find my initial Android backend for Winit here: https://github.com/rib/winit/tree/agdk-game-activity

Now it's a bit tricky to determine the next steps for this work, considering that ideally I'd like to be able to enable this functionality in winit upstream, not just a private branch.

For starters I figured it would be good to share the current status of this work, and then I was thinking I'd list some of the most actionable topics that have come to light in the process of working on this that might help determine some next steps...

Technical backend issues that were noticed:

• The existing Android backend in winit could be made more consistent with e.g. the X11 backend considering being similarly based on polling file descriptors.

Something I found while working on the Android backend was that it was initially unclear (at least to me) if there were potentially multiple points where the loop could block on IO while it didn't follow (what I would consider) a more traditional polling model where there would be one clear point of polling (via a Looper which is Android's wrapper over epoll). For some time I was convinced the existing backend was going to block on its main event reads, in addition to doing a looper poll, but I think to some extent it's just the layout that I found a little misleading to follow. In comparison I found the organisation of the X11 backend quite clear, with a separate function for running a single iteration of the loop and a single point where the loop would block to poll for file descriptor events. When I enabled my own glue layer I ended up following the same structure as the X11 backend.

• It's subjective but the Android event loop feels like it's kind of upside down - where it dispatches events and then polls in different ways according to the control_flow. Again I found the X11 backend structure clearer in this regard - it would poll for events and then run an iteration of event dispatching (which was encapsulated in a function where it was clear to see the logical order of dispatching matched the documented order that's expected).

• A small detail, but the Android backend uses a call_event_handler macro which doesn't need to be a macro. I ended up swapping in a sticky_exit_callback function that was consistent with the X11/Wayland backends.

• Redraw requests are converted into an event in such a way that if they don't get handled by triggering a redraw then the request is lost. I think redraw requests are conceptually expected to be persistent requests that should only be cleared when they are fulfilled. E.g. if a redraw is requested while the loop isn't 'running'  (i.e. suspended) then (currently) the internal event is take()en after waking from the poll but redraws aren't emitted while an app is suspended. I'd intuitively expect that the request should be queued until the app resumes.

• Queuing redraw requests as events introduces some unnecessary latency, and potentially multiple iterations of the event loop before the request might be honoured. At the point of queuing the redraw request there might already be other events pending that will wake up the looper, such as input events and Android's Looper implementation doesn't prioritize delivering a Poll::Wake over any other file descriptor events. Since the redraw request will only be acknowledged once the looper specifically wakes with a Poll::Wake it's possible to handle multiple other pending events before catching up with the redraw request.

Something I did a little differently here, which could potentially be re-used in other backends too was to create a RedrawRequester that encapsulated a shared atomic boolean flag and a 'waker' (i.e. looper.wake()). This gives more ready access to the flag for any subsequent iteration of the event loop (woken for any reason) and doesn't lead to any buffering of data (except for potentially redundant wakes which Android already tries to expunge automatically). The lack of buffering is notable compared to e.g. the X11 backend that uses an mpsc channel for buffering redraw requests which also doesn't seem like an ideal fit for the problem.

Java <-> Native Synchronization in ndk-glue

This is a key area where I was very uncertain about the current ndk-glue design and how robustly is handles synchronization between the java main thread and the rust native thread for operations such as destroying the applications native window and for saving state.

As far as I've seen, ndk-glue employs a purely cooperative synchronization scheme that documents what downstream users of the API must do to ensure synchronization.

There is this comment for the WindowDestroyed event enum:

/// If the window is in use by ie. a graphics API, make sure the lock from
/// [`native_window()`] is held on to until after freeing those resources.
///
/// After receiving this [`Event`] `ndk_glue` will block until that read-lock
/// is released before returning to Android and allowing it to free up the window.
WindowDestroyed,

which essentially documents an implementation detail that says: if you keep the the read lock guard after querying the current native_window() you can effectively block ndk-glue from being able to clear the native window, which will ultimately force synchronization with the Java thread in case it gets notified of a window termination (because it won't be able to write the change).

In practice though winit's Android backend doesn't appear to take a read lock during particularly critical event callbacks, such as for Redraw events so winit doesn't seem to honor the documented synchronization scheme to help block the native window from being torn out in the middle of rendering.

What's also notably different to the original android_native_app_glue provided for use with NativeActivity by Google is that there's no guarantee that the native window remains accessible at least until the WindowDestroyed event has been received and the application has had an opportunity to react. By the time the application sees a WindowDestroyed event the native window could already be long gone.

Something I did differently in the game-activity glue implementation was provide a poll_events() API that takes an FnMut closure that is in some ways comparable to how winit takes a closure that's called for each event. The important thing this enables is that the implementation can place arbitrary pre- and -post logic around the handling of any Android event, which provides a robust place to handle any necessary synchronization with the Java main thread as required for different events, including for window termination and state saving. Since this design also fully encapsulates synchronization, there's no need to downstream users to handle anything and synchronization details can also be changed without affecting applications.

This is something that would also be good to discuss as an ndk-glue issue to see if it makes sense to change its current design but I figure it also makes sense to highlight here too.

High-level Android portability questions

In the process of testing the winit backend and e.g. looking to get egui and Bevy running I realized that if you follow most existing examples for how to use Winit and look at existing integrations you don't tend to end up with an application that is portable to Android (ignoring things like main() function quirks.)

In particular Android is currently unique in requiring applications to be aware that the .native_window() for a winit window will be invalidated between Suspended and Resumed event pairs, and also requiring applications to recreate any render surfaces each time their application is Resumed.

Existing winit integrations tend to assume they can create a window and initialize all graphics state + rendering surfaces up front when they create their event loop, which will ultimately just lead to a panic on Android once the app tries to access a native_window before it has been Resumed.

Here's an example of an upstream PR for egui that attempts to update their winit + wgpu integration abstractions so that it can support Android: emilk/egui#1634

I think their may be some opportunities within Winit to help steer downstream users into building portable integrations and applications. E.g. one idea I've wondered about is whether we could make all platforms consistently deliver a Resumed event (even desktop window systems) and then encourage (by updating examples) that this should be the standard place for all applications to lazily initialize all their graphics state and create their rendering surfaces. There would still be the separate requirement on Android to have to re-create new surfaces for each future Resumed event but it would already be quite a big improvement in terms of consistent application structure that could help encourage portability by default.

Next steps

... okey, I guess I'll stop here for now, since this is already a pretty big dump of information. :)

I'd be interested to gauge interest in any of this, and would be happy to split out separate issues for some of the things mentioned above.

One big question that could be interesting to discuss is whether there might be an interest in updating Winit's Android backend to work with this game-activity glue layer, or something similar considering:

• Some of the concerns around synchronization that the current ndk-glue design appears to have?
• The ability to support AppCompatActivity based Android applications which includes back-ported Android APIs that make it much more practical to develop Android applications that are compatible with a wider range of Android versions.
• Being in a better position to leverage more of the AGDK native libraries, e.g. for improved ime text input handling, game controller support, "swappy" synchronization for rendering etc.

Alternatively maybe it'd be possible to define a standard "glue" API, and maybe have something similar to ndk-context for the glue layer that would make it possible for applications to choose their glue (though I suspect that adding additional abstractions here might also just impede improvements to Android support more than help). I think initially the main challenge with this direction would be with defining a standard input API considering that game-activity is not based on AInputQueue which NativeActivity apps tend to use.

Thanks for your time if you made it this far! :-D

[maroider]

I very much agree that Android is a bit of an ugly duckling compared to our other backends in terms of the handling of the lifetime of the native window, and rectifying that situations is definitely on my radar, although it's not at the top of my list of priorities as of now.

PS: I do appreciate the long issue text, as a fellow poster of such issues/comments :D

[rib]

PS: I do appreciate the long issue text, as a fellow poster of such issues/comments :D

hehe, thanks 👍

I very much agree that Android is a bit of an ugly duckling compared to our other backends in terms of the handling of the lifetime of the native window, and rectifying that situations is definitely on my radar, although it's not at the top of my list of priorities as of now.

I'd be interested to hear if you had any particular ideas in mind already re: handling the way in which Android's native windows come and go over the lifetime of the application?

There are a few discussions open atm relating to the handling of lifecycle events in Winit:
#2185
#2144

where I've noted that the Android backend currently tells a bit of a white lie with regards to the application's lifecycle state. The Android backend currently drives it Suspended / Resumed state based on when it's native surface is destroyed (via an on onSurfaceDestroyed callback from Java. The general assumption / observation is that this usually corresponds to the onPaused lifecycle event, but this isn't something I've ever seen as documented / promised by Android - and I can imagine it's something that could change between versions of Android. A quick search today for a documented connection actually led me to find this page: https://stackoverflow.com/questions/69422482/lifecycle-of-surfaceview which seems to demonstrate an example where that assumption is broken.

I don't think there's anything we can do on Android to simply bypass this protocol of having the surface be destroyed, so I think I currently see two general approaches for better supporting the behaviour in Winit:

1. Embrace the notion that a Winit Window is only a logical window which may conceptually get detached/reattached to some underlying native window/surface if necessary for some backends (like Android). In this case I'd consider adding explicit events for this so we don't have to overload the Suspended/Resumed events and it could be good to change the .native_window() getter to return an Option<> instead of panicking like it does currently on Android.

2. Explicitly start emitting WindowEvent::Destroyed events on Android whenever the native surface is destroyed and then require downstream users to be prepared to create a new window when.... the app next resumes?

When I think about how I've seen downstream integrations build on top of Winit, such as in egui or Bevy my impression is that the more practical option is (1), which is generally similar to the current situation but possibly slightly formalized.

[kchibisov]

I think their may be some opportunities within Winit to help steer downstream users into building portable integrations and applications. E.g. one idea I've wondered about is whether we could make all platforms consistently deliver a Resumed event (even desktop window systems) and then encourage (by updating examples) that this should be the standard place for all applications to lazily initialize all their graphics state and create their rendering surfaces. There would still be the separate requirement on Android to have to re-create new surfaces for each future Resumed event but it would already be quite a big improvement in terms of consistent application structure that could help encourage portability by default.

That could make sense for Wayland. So winit can support layer-shell. It basically gets destroyed from time to time, but the problem is that it differs a lot from normal windows...

Not sure if it'll be a good idea to add special shells in winit.

Embrace the notion that a Winit Window is only a logical window which may conceptually get detached/reattached to some underlying native window/surface if necessary for some backends (like Android). In this case I'd consider adding explicit events for this so we don't have to overload the Suspended/Resumed events and it could be good to change the .native_window() getter to return an Option<> instead of panicking like it does currently on Android

That won't work with applications requiring RawWindowHandle. Since it would mean that it could invalidate and everything should be recreated. Not sure something like that makes sense for other platforms. I mean, that's your intention, but it could be confusing. The clearer aproach is notyfind that the window got destroyed.

The problem is that there should be some notion when system expects you to recreate the window...

[rib]

That won't work with applications requiring RawWindowHandle. Since it would mean that it could invalidate and everything should be recreated. Not sure something like that makes sense for other platforms. I mean, that's your intention, but it could be confusing. The clearer aproach is notyfind that the window got destroyed.

Just to clarify; this is how things currently work on Android - the RawWindowHandle changes over the lifetime of the Winit window, and from an integration pov (e.g. looking at what's convenient in things like Bevy + egui) my impression is that it would complicate things to start mapping this into really destroying the top-level window.

I think terminology matters a lot here and unfortunately I think it's a little misleading on Android; especially the fact that 'surface views' have to be mapped to 'raw window' handles since surface views are a lower level object for rendering - distinct from the the Android 'Window' class (https://developer.android.com/reference/android/view/Window) that doesn't get taken away. (Note: the RawWindowHandle is a native handle that can be used to create a GPU graphics context - where it's platform specific whether this really maps to a logical 'window')

On Android 'raw window handle' can be seen as a bit of a misnomer for that platform. Even the underlying 'surface view' is a bit of a misnomer when it comes to understanding conceptually what it is. A better name could be 'swap chain' or 'buffer queue'. It's more closely related to the kind of surface you create and configure via wgpu for rendering - so more of an implementation detail that impacts rendering. Android has it's own high level "Window" class and it's notable that the application's Android Window isn't getting destroyed / recreated. It probably doesn't help that the original android_native_app_glue from Google refers to surface view destruction/creation as a 'window'.

A Winit Window is something that tends to be most meaningful to applications, and typically they don't care much about the details and just say "I want a window please - and give it this title if possible". From an app pov they'd rather not deal with windows being torn away arbitrarily due to system issues, and it can make sense to associate high-level state with a window that you probably expect to last for the lifetime of the application.

To briefly try and compare it more to Wayland (considering there is overlapping terminology that has different meanings); with Wayland you have wl_surface objects that correspond to 'windows' in winit and then you post wl_buffers to present to those windows/surfaces. On Android when your surface view is destroyed (what a raw handle relates to) that's kind of like an embargo on creating buffers. A surface view conceptually represents a kind of swap chain that's managed by the OS, which you need for rendering, which is notably different to how wayland allocates its buffers). Imagine you had an intermediate 'swap chain' object that you requested wl_buffers from, and that sometimes you'd get notified that the swap chain was destroyed. You would still have your wl_surfaces in tact (your windows for the application are logically still fine and you wouldn't want to tell the application their window has been destroyed) but now you need to wait until you're given a new swap chain before you can allocate any more wl_buffers to post to your surface/window - that's essentially what's happening on Android.

I think the situation with Wayland layer shells is a little different here. In that case an application acting as a shell layer will know what it's designed for. The protocol allows for the shell to notify the application that the layer has effectively become redundant because there's no longer an output for it to be anchored too or the user chose to close the layer widget and at a high level the application would be prepared to respond to that.

I think in the case of layer shells it might be reasonable to map the closed notifications to Destroyed events, but I'd guess it would also be reasonable to expect that this kind of application is Wayland specific (or at least have Wayland specific implementation details that can directly respond to the closed event in its own way.) One of the notable reasons a shell layer would get a closed event is that the user wants decides to close the widget - and in that sense mapping to Destroyed would probably be ok.

So in summary, I'd say I tend to see the transient nature of raw (surface view) handles as a lower-level implementation detail on Android related to rendering, and it doesn't really represent a high level event that applications should be closely concerned with - except as far as knowing they are blocked from rendering. The terminology is what's confusing things here. It's just that other OSs don't have this same architecture where they can take away your swap chain and take away your ability to allocate new back buffers in such an explicit way. Mapping this to window Destroyed events on Android would be a bit like notifying desktop applications that their window was destroyed whenever the user alt tabbed to another app or minimized the app - which would be quite frustrating for applications to deal with.

Hope that all makes sense?

[Hoodad]

Pitching in here. I like to stress that support for getting the Unicode characters from the virtual keyboard is (to my knowledge at least) missing in NativeActivity and resolved in the GameActivity which adds proper support for TextInput.
You can do a dirty conversion (and I have done that) to convert key codes together with meta state and get support for characters A-Z and 0-9 plus some limited number of special characters. But that's where it currently ends. Adding full support does not seem possible.

So if anything this is just more wood on the fire 🔥

[rib]

Cool, thanks, yeah I'm not exactly sure what the ideal next steps here should be atm.

Since I was thinking there might still be an interest in maintaining support for NativeActivity in winit (which might be appealing for anyone that really despises building stuff with Gradle perhaps ;-p) I actually also recently implemented an alternative native-activity glue layer in the same repo: https://github.com/rib/agdk-rust/tree/main/native-activity and with a little iteration I made it API compatible with game-activity to the extent that my Winit branch is now able to conditionally build against either crate.

(If you see the examples here there are now some with an na- prefix that test running with NativeActivity, including with winit, and they can also be built with cargo apk.)

In this case the native-activity crate would essentially be a replacement for ndk-glue but I think some of the design differences (such as with how it handles synchronization with Java) could make it worthwhile to consider. One other feature native-activity currently supports is the ability to save/restore application state which is notably something that depends on correct synchronization between Rust and Java.

I'm half wondering if there would be interest in a PR that would first make some of the smaller tidy up changes I made to the Android backend and switch over to native-activity. Since that's API compatible with game-activity it might then be easier to figure out incremental ways of enabling GameActivity as an optional, alternative Activity.

[msiglreith]

To provide a larger coverage of general features and mobile specific parts, having a custom Activity or moving away from NativeActivity in general seems necessary. SDL has also a non-trivial amount of Java code in their codebase.

Currently I see support for Java/gradle in cargo-apk or xbuild (which seems to be the future) as one main blocking tasks. Porting stuff to Android seems to be a common point of friction, so reducing the overhead for it would be great :D

[MarijnS95]

I'd like to not require additional Java code for pure native applications, or at the very least have that option available with winit. When the user requires Java interop and/or additional features they're probably better off building their app in Java/Kotlin/Flutter and using Mozilla's rust-gradle plugin to take care of the rest.

(Implying lifecycle management and other bits stay in Java-land too - the ndk already has support for rendering to individual surfaces etc).

[rib]

I'd like to not require additional Java code for pure native applications, or at the very least have that option available with winit. When the user requires Java interop and/or additional features they're probably better off building their app in Java/Kotlin/Flutter and using Mozilla's rust-gradle plugin to take care of the rest.

(Implying lifecycle management and other bits stay in Java-land too - the ndk already has support for rendering to individual surfaces etc).

I might be misunderstanding this but I'm not sure I agree with the assertions here...

It's really easy to stray beyond what's possible purely based on NativeActivity because there are a number of important Activity features that require you to subclass and override methods in the Activity and there's no way around that. (I'd probably go so far as to say it's inevitable for any Android application that's more than a bare-bones rendering test)

An example that affected me recently was with using the Bluetooth API on Android and wanting to use the Companion API to select a device. For this you need to launch another activity with an Intent and get a result back (with the user's choice) via the onActivityResult method.

I think there are lots of different use cases for an Android app to launch another Activity via an Intent and want to get a result back (for various system helpers), since that's a fairly primitive Android mechanism.

Another example is the onNewIntent callback which @Hoodad above has recently found they needed: rust-mobile/ndk#275 (comment)

If you look at the docs for the Activity class you can find lots of onXyz methods that aren't all integrated with NativeActivity and there's a good chance for any app to eventually need to integrate with one of them eventually.

I don't think needing these features really implies much about the complexity of the application or whether it should be built via Java/Kotlin/Flutter instead of being a native Rust application.

If your app wants to render with opengl/vulkan/wgpu such as a Bevy game, or using a UI like Egui (or any Rust native UI that renders via wgpu/gl), and especially if you want the app to be portable between mobile and desktop etc then I can see that it'd be desirable to want to structure that as a standard native application (as opposed to writing a Rust plugin for a Java/Kotlin application).

Overall I think that, in practice, any application for Android that isn't a just a quick test/demo/toy will inevitably need to integrate with Java for something - It's just the nature of the beast with Android applications all being JVM based. Like with iOS you wouldn't get far without needing to write some objective C code to access the rest of the system.

When it comes to Winit I think it could be nice to try and preserve support for NativeActivity if possible - due to the convenience of being able to build and run minimal apps without compiling any extra Java code but I also think it'd be good to also help support applications that are based on different Activity subclasses (not necessarily sucblasses of NativeActivity).

[MarijnS95]

When it comes to Winit I think it could be nice to try and preserve support for NativeActivity if possible - due to the convenience of being able to build and run minimal apps without compiling any extra Java code but I also think it'd be good to also help support applications that are based on different Activity subclasses (not necessarily sucblasses of NativeActivity).

That sums it up nicely. Winit doesn't need to lock into "pure" native apps without Java, but it should definitely not be a requirement.

[dvc94ch]

Just came across this issue. I see the native glue stuff is back, which is fine. The first android backend by tomaka made use of it. Not sure why exactly, but it never worked properly for me. While it initially rendered correctly, once the orientation changed of the device the whole thing got extremely confused. I kind of assumed it was the native glue. I didn't have that issue in my own backend. Not that I'm an android expert, getting the orientation changes working was my first android experience.

I agree with the sentiment that java is required for any non trivial android app. The way to do callbacks in java is by creating an abstract class and then subclassing it. Kind of a really weird form of Box<dyn Trait>. I'm a bit disgusted about having to use gradle and don't like that androidx forces a complex java build system on you. I did write a simple maven package manager for xbuild, but I'd rather not have to.

I know it goes completely against the work proposed here (more java and c++), but one approach might be to use /dev/binder directly. While Activity and Service still need to be subclassed to provide an entry point, serializing/deserializing messages in rust could maybe avoid a lot of jni/ndk/java boilerplatte required for writing android apps in rust.

[MarijnS95]

I'm with @dvc94ch on this one. It's okay to have the option to use the C/C++ native_app_glue provided by Google, but not at the detriment of a Rusty glue layer.

You are right in the OP that this implementation brought some syncing issues - the most prevalent being addressed just as you were filing all these reports. OT: Review strongly appreciated, both here in winit and android-nkd-rs as they're blocking the 0.27 release. At the same time the glue layer seems to have been written in parallel with winit, as there's a very loaded API "guideline" between the two (also described by your post).

I think we can address both issues by inventing a common API between the four implementations (Rusty glue, and C/C++ layer, for both NativeActivity and GameActivity), as you proposed. The main questions are:

• Where does that API live?
• Who'll write it? Who'll review and sign off on it?
• Who'll implement/uprev ndk-glue for it?
• Who'll make winit use the new API?

Aside that, extending Java classes seems like an entire new can of worms that I'd like to discuss separately. Perhaps it isn't too hard to extend NativeActivity/GameActivity with commonly requested callbacks that we simply forward to native code for the user, compile that to a .class file and ship it with the crate so that the user doesn't need any Java code? It seems JNI can't insert callbacks into objects but it can load .class files (not that it makes a difference, since Zygote would do that as instructed by AndroidManifest.xml).

[maroider]

Aside that, extending Java classes seems like an entire new can of worms that I'd like to discuss separately.

Perhaps you should open a new issue to this end :D

Perhaps it isn't too hard to extend NativeActivity/GameActivity with commonly requested callbacks that we simply forward to native code for the user, compile that to a .class file and ship it with the crate so that the user doesn't need any Java code?

This sounds intriguing to me. I take it this would let us create as much custom Java glue as we need without making downstream users compile any Java code themselves?

[rib]

I think when it comes to using the C/C++ glue code from Google I tend to see that as an implementation detail.

With the initial game-activity/native-activity crates I wrote then the C/C++ code was fully encapsulated within the implementation and for both the NativeActivity and GameActivity case there's nothing that precludes re-writing that code in Rust, without affecting any public API. The use of existing C/C++ code is purely pragmatic, because that code already exists so it saves time and has been heavily tested (native_app_glue has been around for something like 10 years).

re: handling orientation changes, I think the main gotcha that's not obvious there is that you have to add something like android:configChanges="orientation|screenSize|screenLayout|keyboardHidden" in your AndroidManifest.xml if you want to handle various config changes in a native application (not just NativeActivity and nothing specific to the C/C++ glue) because Android will otherwise re-create the Activity for config changes which breaks the assumptions of how these glue layers are implemented.

Porting the C/C++ code to Rust would be fairly easy for NativeActivity at least but what matters more imho is that the original design of the native app glue had a better encapsulated design for handling synchronization between native and Java code which I think got lost with ndk-glue, and the android_app API also provided a point of reference for application state that's missing with ndk-glue at the moment.

Overall I don't see any issue with building on existing C/C++ glue code initially if it's fully encapsulated. It's not going to be to the detriment of a pure Rust glue layer, and in the short term I tend to think it provides a better foundation (imho) than what we currently have in pure rust - which also means we can look at more interesting problems instead of starting out by porting C code to Rust and potentially introducing bugs.

In terms of creating a pure Rust glue layer, my inclination would be to incrementally re-write the GameActivity glue in Rust while maintaining the same basic design and synchronization model initially. There are already a few C/C++ patches I've had to make to GameActivity and I can start to see a few areas where it could be good to create a RustActivity that can go beyond what GameActivity supports.

[rib]

I think we can address both issues by inventing a common API between the four implementations (Rusty glue, and C/C++ layer, for both NativeActivity and GameActivity), as you proposed. The main questions are:

• Where does that API live?
• Who'll write it? Who'll review and sign off on it?
• Who'll implement/uprev ndk-glue for it?
• Who'll make winit use the new API?

Okey, I'd been meaning to look at this a while ago but managed to get a chance this weekend to take a pass at this and made an android-activity crate that can be configured to run against NativeActivity or GameActivity with a common API:

https://github.com/rib/android-activity
https://crates.io/crates/android-activity

and a corresponding, updated Winit backend based on this: https://github.com/rib/winit/tree/android-activity

The API is essentially the same as the API I had arrived at with the native-activity + game-activity crates (which I'd already been relying on being compatible so I could swap them within the Winit backend). The main difference is that winit doesn't need to juggle multiple alternative crates and handle selection and instead the application crate can make that decision independently.

Technically android-activity merges my previous native-activity and game-activity crates into a common crate and organizes it so they share common code / types and the activity implementations are selected by "native-activity" and "game-activity" features respectively.

Considering other winit discussions we've had recently, there are a few notable changes in the latest incarnation of this common API, compared to my first iterations:

• There's no longer a global native_window() API
• The AndroidApp is also no longer accessible via a global / static and instead it gets passed to the android_main() function.

These changes are essentially to keep open the possibility of supporting multiple activities within a single Android application process - which is something that Android supports but wouldn't have been possible to support with the previous API.

In terms of the bullet-points:

> * Where does that API live?

I guess it could make sense to host the repo under rust-windowing if we think this crate could be usable? I don't think I have any strong opinions here; I'd be happy moving the current repo I have, and adding other contributors/maintainers etc if people think android-activity could be a good basis here.

> * Who'll write it? Who'll review and sign off on it?

Hopefully what I've written can be used and anyone with a vested interest in improving how we support Rust development on Android is welcome to provide feedback / review / contribute

The biggest open issue I currently see with the common API I created relates to a difference with how NativeActivity and GameActivity handle input. In particular, NativeActivity can provide notifications for new input whereas GameActivity expects applications to pull input events as part of their rendering loop. (GameActivity is generally geared towards games that render continuously)

The common API is currently geared towards pulling input events, similar to GameActivity, because that was more practical to make portable but I think it should also support notifying when there's new input for UI applications that don't render continuously. The API itself can support this and I do have a plan for enabling this in the implementations but wanted to highlight this - since it would affect some UI applications that don't necessarily redraw continuously.

> * Who'll implement/uprev ndk-glue for it?

Atm I'm not sure what the idea is here?

The aim as I see it atm is to support multiple Activity subclasses via a common API that e.g. Winit can consume; namely NativeActivity and GameActivity in the short-term but we also want to allow something like a RustActivity in the future where we are fully responsible for the Java side of things too.

I guess this point is about the desire to have a pure rust glue layer, considering that android-activity currently builds on the C/C++ code from Google?

If that's the case; I think that's just a matter of planning to re-implement the C/C++ code in Rust at some point which can be be done whenever someone has the time to do that? If someone were to do that work then I don't think it would be necessary to keep the redundant C/C++ code at that point. I currently see this as an orthogonal piece of work though because the use of C/C++ for the lower-level details doesn't affect the public API.

> * Who'll make winit use the new API?

I think the branch needs to be rebased and re-iterated to tidy a few things up but hopefully this is a useful starting point: https://github.com/rib/winit/tree/android-activity

There are also some minimal examples here: https://github.com/rib/android-activity/tree/main/examples

[rib]

For reference I've just rebased the android-activity branch on winit master

[MarijnS95]

Perhaps you should open a new issue to this end :D

@maroider That's something for android-ndk-rs though, it's mostly users wanting access to various forms of intent handling as far as I've seen.

This sounds intriguing to me. I take it this would let us create as much custom Java glue as we need without making downstream users compile any Java code themselves?

Exactly. It's also how @rib's GameActivity "works", by embedding precompiled class files (in the end, I think that's how they're downloaded from one of these "Java repositories") in an APK for consumption.

Not sure how much boilerplate/tooling we need for that though.

---

re: handling orientation changes, I think the main gotcha that's not obvious there is that you have to add something like android:configChanges="orientation|screenSize|screenLayout|keyboardHidden" in your AndroidManifest.xml if you want to handle various config changes in a native application (not just NativeActivity and nothing specific to the C/C++ glue) because Android will otherwise re-create the Activity for config changes which breaks the assumptions of how these glue layers are implemented.

Looks like we may be missing screenLayout:

https://github.com/rust-windowing/android-ndk-rs/blob/dc38fcbf688a0a0e7a378a18f7b2cf4c095ccf92/ndk-build/src/manifest.rs#L324-L326

(At least on my phone setting the app to be a freefloating window recreates a new Activity instance, and we all know how that works currently... not 😬)

[MarijnS95]

• There's no longer a global native_window() API
• The AndroidApp is also no longer accessible via a global / static and instead it gets passed to the android_main() function.

These changes are essentially to keep open the possibility of supporting multiple activities within a single Android application process - which is something that Android supports but wouldn't have been possible to support with the previous API.

Great, those are exactly the changes I intended to make, for that exact reason and more (prior to riling up this entire discussion around fixing outstanding issues and supporting GameActivity at the same time).

[rib]

For reference I opened an issue to track the need to support input event notifications in android-activity and have made a first pass at adding the support for InputAvailable events if running with NativeActivity - so at least it should be possible to avoid any regression in Winit with respect to what's supported with NativeActivity.

I'm still planning to also support the same event with GameActivity for parity there but will take a look at that later.

I haven't updated the Winit backend to handle this event yet.

[rib]

Okey, following up on this discussion I've just opened a PR for an updated Android backend based on android-activity: #2444

While android-activity has has a number of clean ups recently and considering that it now consistently supports InputAvailable events with both NativeActivity and GameActivity (which makes it possible to support gui applications that don't render continuously) I feel like android-activity 0.2 is in a decent shape at this stage to really consider upstreaming the backend changes I have for using android-activity in Winit.

We also recently made progress with being able to build Egui, EFrame applications based on android-activity, which gives some confidence that it's getting into shape: https://github.com/emilk/egui/pull/1952/commits