SurfaceTexture NDK bindings

[lattice0]

Are there plans to support OpenGL?

[dvc94ch]

there is gl, glutin and probably a dozen other rust crates that provide opengl bindings. I doubt it's in scope for this project, as it's not android specific.

[lattice0]

I've seen one pure rust glutin example that works on Android. However, if I have a Kotlin SurfaceTexture and want to render into it with native code, I need the NDK bindings so I can do stuff like

    egl.eglCreateWindowSurface(eglDisplay, eglConfig, surfaceTexture, null);

that is, pass the surfaceTexture pointer back and forth.

[MarijnS95]

eglCreateWindowSurface is an EGL binding, not an NDK binding. As David said, GL is a generic API that has nothing to do with the specifics of the NDK; would you mind updating this issue and title to be more specific about whatever problem you may be facing?

For starters you could look at how glutin uses eglCreateWindowSurface:

https://github.com/rust-windowing/glutin/blob/279b5a406878ce3638e1b80527ae07958eb6c12b/glutin/src/api/egl/mod.rs#L540-L545

Then, is there any reference code (ie. written in Kotlin, Java or C++) that converts a Kotlin SurfaceTexture to an EGLSurface? It seems that a SurfaceTexture wraps / streams to an OpenGL ES Texture; if you have the texname of this texture (and the appropriate OpenGL context) you might be able to interact with it directly.

[lattice0]

@MarijnS95 in flutter, it creates a SurfaceTexture for me, and it looks like there's no way to get the texName from inside a SurfaceTexture, amd I right?

[lattice0]

Don't I need at least https://developer.android.com/ndk/reference/group/surface-texture? I think I need to at least attach this SurfaceTexture to the opengl context. The problem is, how do I get a SurfaceTexture on the Rust side given only the texId...

ASurfaceTexture_fromSurfaceTexture(JNIEnv *env, jobject surfacetexture) is out of question because I want to call Rust code directly from Flutter, so no JVM, the JVM is only used to create the SurfaceTexture using Flutter's API

[MarijnS95]

I this hinges on ASurfaceTexture_acquireANativeWindow as ANativeWindow is what can be passed into eglCreateWindowSurface.

You'll have to ask @dvc94ch for Flutter support, as I don't have the slightest clue. If Flutter hands you a SurfaceTexture and is implemented in native code, is there a possibility that it simply wraps ASurfaceTexture which you can use to get access to the aforementioned API?

Again, you should rewrite this issue and title to document what you want to achieve, rather than requesting something opaque.

[dvc94ch]

when building a flutter-rs plugin for video playback, flutter had a Texture widget. So you'd create an opengl texture, render the paced frames to the texture and pass the texture id to the Texture widget on the flutter side. When it comes to embedding flutter in a native android ui, don't really know. Don't have any experience with native android ui's.

[lattice0]

@dvc94ch ok, changed the title. I don't want to embed a flutter view on Android, I want to render to a flutter texture in Rust side. It turns out that to create a texture on Flutter you need to call kotlin/java code:

  override fun onMethodCall(@NonNull call: MethodCall, @NonNull result: Result) {
    if (call.method == "getPlatformVersion") {
      result.success("Android ${android.os.Build.VERSION.RELEASE}")
    } else if (call.method=="create_texture") {
      Log.d(LOG_TAG, "createVideoRenderer called")
      val entry: TextureRegistry.SurfaceTextureEntry = texture_registry.createSurfaceTexture();
      val surfaceTexture: surfaceTexture = entry.surfaceTexture();
      //Store the surfaceTexture somewhere for rendering later

It gives me a SurfaceTexture, and to render to it in native side the only way I can see is through https://developer.android.com/ndk/reference/group/surface-texture#group___surface_texture_1ga8928719bfd3fc8a26595c7c8b0040aa2, or at least I though it helped me, if I could access the texture id. How??

Anyways, there is still a major problem: I want to call Rust code through Dart's FFI, which completely ignores the JVM, but ASurfaceTexture_fromSurfaceTexture(JNIEnv *env, jobject surfacetexture) obviously uses JVM. I could call it from the Kotlin side, but will the returned ASurfaceTexture * be valid on the Rust side?

[dvc94ch]

ah I see. looks like what you're missing is some jni bindings to the android flutter embedder. your dart application uses the ffi to call into your rust code, which then uses the jni to create a surface texture. you can then render to the surface texture using opengl.

[lattice0]

@dvc94ch I'm a bit lost. If I load the rust .so through Flutter/Dart, then it wouldn't have access to the JVM, so I cannot call the android embedder, which is Kotlin/Java, right? Or is the embedder C++ and just called through Kotlin/Java?

[dvc94ch]

all android applications are java applications. your activity initializes the flutter android embedder, which initializes the flutter engine which loads your dart code. you need to initialize the ndk-context from your activity and then your dart code can call your rust code, which can use the ndk-context to create the surface texture. if you're asking how cargo-apk helps you with this, the answer is it doesn't. You need to use xbuild or roll your own.

[lattice0]

Yes, the Kotlin code:

  override fun onMethodCall(@NonNull call: MethodCall, @NonNull result: Result) {
    if (call.method == "getPlatformVersion") {
      result.success("Android ${android.os.Build.VERSION.RELEASE}")
    } else if (call.method=="create_texture") {
      Log.d(LOG_TAG, "createVideoRenderer called")
      val entry: TextureRegistry.SurfaceTextureEntry = texture_registry.createSurfaceTexture();
      val surfaceTexture: surfaceTexture = entry.surfaceTexture();
      //Store the surfaceTexture somewhere for rendering later
      //here

which is called from Flutter as a plugin method call, initializes a SurfaceTexture bound to a Flutter texture. Technically I can call C++ on line here on code above and get a pointer to the C++ ASurfaceTexture, then transform this pointer into a java long and pass back through the Flutter method call to Dart code, and then pass it through dart to Rust code, and then Rust code can call

int ASurfaceTexture_attachToGLContext(
ASurfaceTexture *st,
uint32_t texName
)

to attach whatever texture it creates, and also call things like ASurfaceTexture_updateTexImage(ASurfaceTexture *st) and ASurfaceTexture_attachToGLContext(ASurfaceTexture *st, uint32_t texName) to update/render the images.

Is this what you meant? There's still the issue of having to call https://github.com/flutter/engine/blob/8ab4124eeff5b259153e56fadfaabf670655687b/shell/platform/android/io/flutter/embedding/engine/renderer/FlutterRenderer.java#L417 every time a new frame should be rendered, which would be hard to do from Rust as it's Java code. Also, I find it very ugly and unsafe to use Kotlin to call C++ to get the pointer, then pass as a number to Flutter, which passes to Rust, which casts to ASurfaceTexture (by the way, I'd need ASurfaceTexture in Rust which I think should be mapped by this lib, as it's from the NDK).

Is there something I'm missing here?

[dvc94ch]

yep, we can add bindings for ASurfaceTexture. everything else is out of scope. so ndk-sys already contains ASurfaceTexture, you'd just need to add a small high level wrapper to ndk or use ndk-sys directly if you don't want to bother.

[lattice0]

@dvc94ch it'd be nice for me to do a PR to get high level support for ASurfaceTexture. On which file should I add it? https://github.com/rust-windowing/android-ndk-rs/tree/master/ndk/src