crashes when calling c++ in arm android

[yyccR]

System information (version)

• OpenVINO => 2022.3.0
• Operating System / Platform => Android(arm64)
• Compiler => Visual Studio 2017

Detailed description

I was going to use openvino in flutter by calling c++ instead of using the Java api, but it crashed when calling c++ to initialize the model, here is the error msg:
Abort message: 'terminating with uncaught exception of type ov::Exception: Can't get absolute file path for [/data/app/com.test.test-YcefHGmNSq6Rurhx-jV-xQ==/base.apk!/lib/arm64-v8a/libopenvino.so], err = No such file or directory'

here is the c++ code:

#include <iostream>
#include "memory"
#include "openvino/runtime/runtime.hpp"

int initOpenvinoModel(std::string &plugin_xml, std::string &xml_model_path, std::string &bin_model_path) {

    std::shared_ptr<ov::Core> core = std::make_shared<ov::Core>(plugin_xml);
    // std::shared_ptr<ov::Model> model = core.read_model(xml_model_path, bin_model_path);
    // core.compile_model(model, "CPU");
    return 0;
}

extern "C" __attribute__((visibility("default"))) __attribute__((used))
int initOpenvinoModel_c(char* plugin_xml_path, char* xml_model_path, char* bin_model_path) {
    std::string plugin_xml_path_s(plugin_xml_path);
    std::string xml_model_path_s(xml_model_path);
    std::string bin_model_path_s(bin_model_path);
    return initOpenvinoModel(plugin_xml_path_s, xml_model_path_s, bin_model_path_s);
}

and I have build the openvino with arm cpu plugin, below is the shared library and plugins.xml:

libopenvino.so
libopenvino_arm_cpu_plugin.so
libopenvino_auto_batch_plugin.so
libopenvino_auto_plugin.so
libopenvino_hetero_plugin.so

<ie>
    <plugins>
        <plugin name="AUTO" location="libopenvino_auto_plugin.so">
        </plugin>
        <plugin name="BATCH" location="libopenvino_auto_batch_plugin.so">
        </plugin>
        <plugin name="CPU" location="libopenvino_arm_cpu_plugin.so">
        </plugin>
        <plugin name="HETERO" location="libopenvino_hetero_plugin.so">
        </plugin>
        <plugin name="MULTI" location="libopenvino_auto_plugin.so">
        </plugin>
    </plugins>
</ie>

I have linked the libopenvino.so in the project and include all the header files, this is my CMakeLists.txt

cmake_minimum_required(VERSION 3.17.0)
project(test)
set(CMAKE_CXX_STANDARD 17)

include_directories(${CMAKE_CURRENT_LIST_DIR}/libs/openvino-2022.3.0/include)

set(OpenCV_DIR ${CMAKE_SOURCE_DIR}/libs/opencv-mobile-4.5.4-android/sdk/native/jni)
find_package(OpenCV REQUIRED core imgproc highgui)

add_library(openvino SHARED IMPORTED)
set_target_properties(openvino PROPERTIES IMPORTED_LOCATION
        ${CMAKE_CURRENT_LIST_DIR}/libs/openvino-2022.3.0/${ANDROID_ABI}/libopenvino.so)

aux_source_directory(${CMAKE_SOURCE_DIR}/openvino_cpp openvino_src_lists)
add_library(test SHARED ${openvino_src_lists})
target_link_libraries(test ${OpenCV_LIBS})
target_link_libraries(test openvino)

And I've put all the plugin shared libraries in the same directory as libopevino.so。

I also put plugins.xml, openvino_model.xml, openvino_model.bin in the assert directory and copy these three files to the android system when I start the application so that c++ programs can read them.

The error below

Abort message: 'terminating with uncaught exception of type ov::Exception: Can't get absolute file path for [/data/app/com.test.test-YcefHGmNSq6Rurhx-jV-xQ==/base.apk!/lib/arm64-v8a/libopenvino.so], err = No such file or directory'

seems that openvino c++ wants to find the libopenvino library directory and get the plugin shared library, but cannot access it when android is not root

So I've put all the plugin.so in the asset directory, and change the plugins.xml into :

<ie>
    <plugins>
        <plugin name="AUTO" location="/data/user/0/com.test.test/app_flutter/libopenvino_auto_plugin.so">
        </plugin>
        <plugin name="BATCH" location="/data/user/0/com.test.test/app_flutter/libopenvino_auto_batch_plugin.so">
        </plugin>
        <plugin name="CPU" location="/data/user/0/com.test.test/app_flutter/libopenvino_arm_cpu_plugin.so">
        </plugin>
        <plugin name="HETERO" location="/data/user/0/com.test.test/app_flutter/libopenvino_hetero_plugin.so">
        </plugin>
        <plugin name="MULTI" location="/data/user/0/com.test.test/app_flutter/libopenvino_auto_plugin.so">
        </plugin>
    </plugins>
</ie>

The location in plugins.xml is the path to the android system file, which can be read by c++ programs, and is the target copy path to the asset file。

After trying, I still get the same error message.

please let me know if you have a solution.

Thank you so much.

[ilya-lavrenov]

@dkurt do you have suggestions?

[dkurt]

And I've put all the plugin shared libraries in the same directory as libopevino.so

Which directory? /data/app/com.test.test-YcefHGmNSq6Rurhx-jV-xQ==/base.apk! must be a temporal directory which is available only in Java runtime. Also, running C++ requires rooted device, is your device rooted? If so, can you please put the binary and libraries in some regular location.

[yyccR]

@dkurt Thanks，At first my project directory tree:

android
├──app
├────├──src
├────────├──mian

lib
├──my_flutter_calling_cpp_code.dart

assert
├── plugins.xml
├── model.xml
├── mdoel.bin

cpp
├── openvino-2022.3.0
│	├── arm64-v8a
│	│   ├── libopenvino.so
│	│   ├── libopenvino_arm_cpu_plugin.so
│	│   ├── libopenvino_auto_batch_plugin.so
│	│   ├── libopenvino_auto_plugin.so
│	│   ├── libopenvino_hetero_plugin.so
│	├── armeabi-v7a
│	│   ├── libopenvino.so
│	│   ├── libopenvino_arm_cpu_plugin.so
│	│   ├── libopenvino_auto_batch_plugin.so
│	│   ├── libopenvino_auto_plugin.so
│	│   ├── libopenvino_hetero_plugin.so
│	│── include
│	│   ├── ie
│	│   ├── ngraph
│	│   ├── openvino
├── openvino_cpp
│	├── test.cpp
├── CMakeLists.txt

But when got this error msg:

Abort message: 'terminating with uncaught exception of type ov::Exception: Can't get absolute file path for [/data/app/com.test.test-YcefHGmNSq6Rurhx-jV-xQ==/base.apk!/lib/arm64-v8a/libopenvino.so], err = No such file or directory'

I change my project tree into:

android
├──app
├────├──src
├────────├──mian

lib
├──my_flutter_calling_cpp_code.dart

assert
├── plugins.xml
├── model.xml
├── mdoel.bin
├── libopenvino_arm_cpu_plugin.so
├── libopenvino_auto_batch_plugin.so
├── libopenvino_auto_plugin.so
├── libopenvino_hetero_plugin.so

cpp
├── openvino-2022.3.0
│	├── arm64-v8a
│	│   ├── libopenvino.so

│	├── armeabi-v7a
│	│   ├── libopenvino.so
│	│── include
│	│   ├── ie
│	│   ├── ngraph
│	│   ├── openvino
├── openvino_cpp
│	├── test.cpp
├── CMakeLists.txt

When the application starts, I copy all the libraries in the assert directory to the android readable directory，and then change the location of all the plugin.so in the xml file:

<ie>
    <plugins>
        <plugin name="AUTO" location="/data/user/0/com.test.test/app_flutter/libopenvino_auto_plugin.so">
        </plugin>
        <plugin name="BATCH" location="/data/user/0/com.test.test/app_flutter/libopenvino_auto_batch_plugin.so">
        </plugin>
        <plugin name="CPU" location="/data/user/0/com.test.test/app_flutter/libopenvino_arm_cpu_plugin.so">
        </plugin>
        <plugin name="HETERO" location="/data/user/0/com.test.test/app_flutter/libopenvino_hetero_plugin.so">
        </plugin>
        <plugin name="MULTI" location="/data/user/0/com.test.test/app_flutter/libopenvino_auto_plugin.so">
        </plugin>
    </plugins>
</ie>

This directory /data/user/0/com.test.test/app_flutter/ is readable for c++ program. But still got the same error:

Abort message: 'terminating with uncaught exception of type ov::Exception: Can't get absolute file path for [/data/app/com.test.test-YcefHGmNSq6Rurhx-jV-xQ==/base.apk!/lib/arm64-v8a/libopenvino.so], err = No such file or directory'

The error seems that openvino c++ want to access directory /data/app/com.test.test-YcefHGmNSq6Rurhx-jV-xQ==/base.apk! which was denied, because my device not rooted,

I also tried to manually load all the plugin.so myself, here is CMakeLists.txt:

cmake_minimum_required(VERSION 3.17.0)
project(test)
set(CMAKE_CXX_STANDARD 17)

include_directories(${CMAKE_CURRENT_LIST_DIR}/libs/openvino-2022.3.0/include)

set(OpenCV_DIR ${CMAKE_SOURCE_DIR}/libs/opencv-mobile-4.5.4-android/sdk/native/jni)
find_package(OpenCV REQUIRED core imgproc highgui)

add_library(openvino SHARED IMPORTED)
set_target_properties(openvino PROPERTIES IMPORTED_LOCATION
        ${CMAKE_CURRENT_LIST_DIR}/libs/openvino-2022.3.0/${ANDROID_ABI}/libopenvino.so)

add_library(openvino_auto_plugin SHARED IMPORTED)
set_target_properties(openvino_auto_plugin PROPERTIES IMPORTED_LOCATION
        ${CMAKE_CURRENT_LIST_DIR}/libs/openvino-2022.3.0/${ANDROID_ABI}/libopenvino_auto_plugin.so)
add_library(openvino_auto_batch_plugin SHARED IMPORTED)
set_target_properties(openvino_auto_batch_plugin PROPERTIES IMPORTED_LOCATION
        ${CMAKE_CURRENT_LIST_DIR}/libs/openvino-2022.3.0/${ANDROID_ABI}/libopenvino_auto_batch_plugin.so)
add_library(openvino_hetero_plugin SHARED IMPORTED)
set_target_properties(openvino_hetero_plugin PROPERTIES IMPORTED_LOCATION
        ${CMAKE_CURRENT_LIST_DIR}/libs/openvino-2022.3.0/${ANDROID_ABI}/libopenvino_hetero_plugin.so)
add_library(openvino_arm_cpu_plugin SHARED IMPORTED)
set_target_properties(openvino_arm_cpu_plugin PROPERTIES IMPORTED_LOCATION
        ${CMAKE_CURRENT_LIST_DIR}/libs/openvino-2022.3.0/${ANDROID_ABI}/libopenvino_arm_cpu_plugin.so)

aux_source_directory(${CMAKE_SOURCE_DIR}/openvino_cpp openvino_src_lists)
add_library(test SHARED ${openvino_src_lists})
target_link_libraries(test ${OpenCV_LIBS})
target_link_libraries(test openvino)
target_link_libraries(sai openvino_auto_plugin)
target_link_libraries(sai openvino_auto_batch_plugin)
target_link_libraries(sai openvino_hetero_plugin)
target_link_libraries(sai openvino_arm_cpu_plugin)

But another error occur:

Failed to load dynamic library 'libtest.so': dlopen failed: library "/Users/yang/AndroidStudioProjects/test/cpp/openvino-2022.3.0/arm64-v8a/libopenvino_auto_plugin.so" not found

As you mentioned, running C++ requires rooted device, I think this is a little unfriendly when you want to deploy openvino on arm Android.

[dkurt]

There is a similar report dkurt/openvino_java#18, however I cannot reproduce the problem. Can you verify that Java API works?

[yyccR]

Thanks, I think I know what to after looking at the java api source code: https://github.com/openvinotoolkit/openvino_contrib/blob/master/modules/java_api/org/intel/openvino/compatibility/IECore.java#L45

I should load the libopenvino.so library manually in dart code instead of in the camkeLists.txt.

[dkurt]

@yyccR, can you please check your Android API Level? In example, using CPU-Z app (System tab)

If it's below 31, probably you tried the binaries with higher API requirement. For example, in the case with similar error message dkurt/openvino_java#18 binaries from ANDROID_PLATFORM=32 were used on device with API level 29 while device with API level 31 was fine.

Try rebuild OpenVINO with lower ANDROID_PLATFORM

[yyccR]

Thanks, @dkurt
this is my cmake build flags:

cmake -DCMAKE_BUILD_TYPE=Release \
      -DTHREADING=SEQ \
      -DIE_EXTRA_MODULES="/Users/yang/opt/openvino_android/openvino_contrib/modules" \
      -DCMAKE_TOOLCHAIN_FILE="/Users/yang/Library/Android/sdk/ndk/21.4.7075529/build/cmake/android.toolchain.cmake" \
      -DANDROID_ABI=arm64-v8a \
      -DANDROID_PLATFORM=30 \
      -DENABLE_SAMPLES=OFF \
      -DENABLE_OPENCV=OFF \
      -DENABLE_CLDNN=OFF \
      -DENABLE_VPU=OFF \
      -DENABLE_GNA=OFF \
      -DENABLE_MYRIAD=OFF \
      -DENABLE_TESTS=OFF \
      -DENABLE_GAPI_TESTS=OFF \
      -DENABLE_INTEL_MYRIAD=OFF \
      -DENABLE_INTEL_MYRIAD_COMMON=OFF \
      -DBUILD_java_api=ON \
      -DARM_COMPUTE_SCONS_JOBS=$(nproc) \
      -DENABLE_BEH_TESTS=OFF ..

it was built with ANDROID_PLATFORM=30 and my android build.gradle is:

android {
    compileSdkVersion 33
    ndkVersion 	"21.4.7075529"
     defaultConfig {
        minSdkVersion 24
        targetSdkVersion 31
        externalNativeBuild {
            cmake {
               abiFilters 'armeabi-v7a', 'arm64-v8a'
            }
        }
    }
}

This error Can't get absolute file path libopenvino.so is mainly about Android permissions. If we are not root, we cannot directly access files in path /data/app, so if we want to call openvino from c++ inside androind, we need to package all c++ source files together with libopenvino.so, And use the java code System. LoadLibrary ("/readable/path/to/libopenvino. So "); or the dart code DynamicLibrary. Open ("/readable/path/to/libopenvino. So "); to load the dynamic library manually.

This loading method is very different from other inference frameworks such as onnxruntime, tflite, ncnn, paddle-lite.

[ilya-lavrenov]

This error Can't get absolute file path libopenvino.so is mainly about Android permissions

Do you know whether is happens during reading model through the frontend library or compiling for inference?

Will it be convenient for Android scenarios to build OpenVINO with plugins / frontends as a single libopenvino.so library?
Currently, you can build OpenVINO statically and all the code will be built in into a final binary file. In this case, Can't get absolute file path libopenvino.so will not happen, since the code path leading to it - just not called.

Thanks,
Ilya.