onnx_test_runner support for plugin EP by chilo-ms · Pull Request #25754 · microsoft/onnxruntime

chilo-ms · 2025-08-14T22:00:44Z

Description

Make onnx_test_runner be able to link against onnxruntime.dll.
Add support for onnx_test_runner to register plugin EP DLL and run plugin EP.

When running onnx_test_runner to test plugin EP, we highly suggest that the onnx_test_runner is built/linked against onnxruntime.dll, so that both onnx_test_runner and plugin EP are interacting with the same onnxruntime.dll.

[Note]: On Windows, it's possible that incompatible onnxruntime.dll in system folder is used. Make sure correct onnxruntime.dll is either in onnx_test_runner's folder or plugin EP DLL's folder. ORT calls LoadLibraryExW to load plugin EP DLL with the flag LOAD_WITH_ALTERED_SEARCH_PATH ( Windows uses alternative search order so that the directory containing the loaded DLL is searched for that DLL’s dependencies).

New options:

--plugin_ep_libs [registration names and libraries] Specifies a list of plugin execution provider (EP) registration names and their corresponding shared libraries to register.
[Usage]: --plugin_ep_libs "plugin_ep_name_1|plugin_ep_1.dll plugin_ep_name_2|plugin_ep_2.dll ... "
--plugin_eps [Plugin EPs] Specifies a semicolon-separated list of plugin execution providers (EPs) to use.
[Usage]: --plugin_eps "plugin_ep_1;plugin_ep_2;... "
--plugin_ep_options [EP options] Specifies provider options for each EP listed in --plugin_eps. Options (key-value pairs) for each EP are separated by space and EPs are separated by semicolons.
[Usage]:
--plugin_ep_options "ep_1_option_1_key|ep_1_option_1_value ...;ep_2_option_1_key|ep_2_option_1_value ...;..." or
--plugin_ep_options ";ep_2_option_1_key|ep_2_option_1_value ...;..." or
--plugin_ep_options "ep_1_option_1_key|ep_1_option_1_value ...;;ep_3_option_1_key|ep_3_option_1_value ...;..."
--list_ep_devices Prints all available device indices and their properties (including metadata). This option makes the program exit early without performing inference.
--select_ep_devices [list of device indices] A semicolon-separated list of device indices to add to the session and run with.

Motivation and Context

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/test/onnx/utils/utils.cc

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/test/perftest/ort_test_session.cc

…d library

yuslepukhin · 2025-08-18T22:07:05Z

onnxruntime/test/onnx/testcase_driver.cc

+  oss << "Running tests in parallel: at most "
+      << static_cast<unsigned>(parallel_models)
+      << " models at any time";
+  TEST_LOG_VERBOSE(oss.str());


I think this is a code duplicate and data copy, since the macro itself makes use of the std::ostringstream. Would it be possible to output to macro directly?

good catch. Used macro directly.

yuslepukhin · 2025-08-18T22:10:53Z

onnxruntime/test/onnx/utils/utils.cc

      for (auto& pair : ep_names_to_libs) {
        const std::filesystem::path library_path = pair.second;
        const std::string registration_name = pair.first;
        Ort::Status status(Ort::GetApi().RegisterExecutionProviderLibrary(env, registration_name.c_str(), ToPathString(library_path.string()).c_str()));


We should use C++ interfaces.

changed to use C++ API

yuslepukhin · 2025-08-18T22:14:08Z

onnxruntime/test/onnx/utils/utils.cc

-  for (auto& registration_name : test_config.registered_plugin_eps) {
+void UnregisterExecutionProviderLibrary(Ort::Env& env, std::vector<std::string>& registered_plugin_eps) {
+  for (auto& registration_name : registered_plugin_eps) {
    Ort::Status status(Ort::GetApi().UnregisterExecutionProviderLibrary(env, registration_name.c_str()));


changed to use C++ API

yuslepukhin · 2025-08-21T18:00:28Z

cmake/onnxruntime_unittests.cmake


-    target_link_libraries(onnx_test_runner PRIVATE onnx_test_runner_common ${GETOPT_LIB_WIDE} ${onnx_test_libs} nlohmann_json::nlohmann_json)
+    if (onnxruntime_BUILD_SHARED_LIB)
+      # onnx_test_runner calls functions from onnxruntime_test_utils, which depend on ORT internal C++ APIs not exposed through the public C API.


We need clarufy the purpose and the scope of the PR.
This links the test against many internal libraries. I do not think this is needed or intended as it was not present before.
Also, I thought the purpose of the PR was to link against the DLL, meaning talk to ORT only via public interfaces.
The functionality of the utility is such that it only needs to create sessions and run them in essense.
Very much like onnxruntime_shared_lib_test which is specifically written to test and cover public API.

Unlike onnxruntime_shared_lib_test, onnx_test_runner needs to compare the output and that functions eventually call ORT internal library that are not exposed as C API.

As the comment described below, it calls CompareOrtValue(), VerifyValueInfo() ... which rely on onnxruntime::DataTypeImpl::TypeFromProto(), onnxruntime::DataTypeImpl::ToString() ... in the internal library, i.e. onnxruntime_framework, to do the output comparison.

To work with plugin EP, onnx_test_runner needs to be link against onnxruntime.dll.
For onnx_test_runner to work with onnxruntime.dll, we need to link the ORT internal library.

(additional internal dependencies such as onnxruntime_graph and onnxruntime_mlas must also be linked,
as they are transitively required by symbols used in onnxruntime_framework.)

Even though we make onnx_test_runner link many internal libraries, what it really needs is some functions to compare the OrtValues which are all stateless.

I donot think the comparision logic warrants linking the whole world.
It can be re-written by means public interface. It is not that much code that needs to be changed.

okay, i though about making the parts needed by comparison logic be public. Will take a look to see how much change it needs.

There are more changes than i expected.

Use Ort::Value instead of OrtValue.Type(), OrtValue.Get() ....

ONNXType - Using Ort::GetApi().GetValueType() to get the type.

The ONNXType that are supported:

Tensor

Sparse Tensor - including COO and Csr(c) format.

Sequence of Tensors

Sequence of Maps (where map is either 'int64 to float' or 'string to float')

(Need to rewrite the functions to compare two OrtValues with all of the above type)

ORT doesn't have public API to deal with Int4x2 and UInt4x2 tensor type

MLFloat16 - It's only used in training code, maybe we can remove it?

jywu-msft · 2025-08-27T15:08:51Z

there's build errors and merge conflict
onnxruntime_logging_apis_test.vcxproj -> D:\a_work_temp\build\RelWithDebInfo\RelWithDebInfo\onnxruntime_logging_apis_test.exe
Error: D:\a_work\onnxruntime\onnxruntime\onnxruntime\test\onnx\main.cc(482,7): error C2065: 'p_models': undeclared identifier [D:\a_work_temp\build\RelWithDebInfo\onnx_test_runner.vcxproj]
Error: D:\a_work\onnxruntime\onnxruntime\onnxruntime\test\onnx\main.cc(483,7): error C2065: 'concurrent_session_runs': undeclared identifier [D:\a_work_temp\build\RelWithDebInfo\onnx_test_runner.vcxproj]
Error: D:\a_work\onnxruntime\onnxruntime\onnxruntime\test\onnx\main.cc(484,74): error C2065: 'device_id': undeclared identifier [D:\a_work_temp\build\RelWithDebInfo\onnx_test_runner.vcxproj]

chilo-ms · 2025-09-09T20:29:24Z

Have a separate PR that add a new onnxruntime_plugin_ep_onnx_test for plugin EP to run onnx test and uses public api to compare output values.
#25942

chilo-ms added 3 commits August 13, 2025 23:15

make onnx_test_runner be able to link to onnxruntime.dll

c1ff5a8

Update to use Abseil and add options to support plugin EPs

0fa2dfd

update perf test to use common utility functions

6dbc534

chilo-ms added the release:1.23.0 label Aug 14, 2025

chilo-ms added 2 commits August 14, 2025 15:05

fix build error

461637b

fix build issues

ecefeca

github-actions bot reviewed Aug 14, 2025

View reviewed changes

github-advanced-security bot found potential problems Aug 14, 2025

View reviewed changes

onnxruntime/test/onnx/utils/utils.cc Fixed Show fixed Hide fixed

chilo-ms added 6 commits August 14, 2025 15:40

address lintrunner issues and fix build error

f1a34db

fix compile warning

4bcbb59

fix build issue

c6be7b0

update cmake file

b09d702

fix build issue for QNN

8912aa6

fix build issue

9428da6

github-actions bot reviewed Aug 15, 2025

View reviewed changes

onnxruntime/test/perftest/ort_test_session.cc Outdated Show resolved Hide resolved

chilo-ms added 2 commits August 15, 2025 23:46

Not using default logger in onnx_test_runner when building as a share…

cb2d5cc

…d library

small fix for onnxruntime_perf_test

8f50442

chilo-ms marked this pull request as ready for review August 16, 2025 06:47

chilo-ms added 5 commits August 17, 2025 20:44

add missing header

9bed92b

replace improper character in the file

78d5bab

revert file

72b7052

forgot to update ep name in test config

c9417a5

only apply tolerance value when user explicitly specifies it

7298cd8

yuslepukhin requested changes Aug 18, 2025

View reviewed changes

chilo-ms added 5 commits August 20, 2025 13:56

merge main

80cb063

Merge branch 'main' into chi/onnx_test_runner_for_plugin_ep

b029fd5

address reviwer's comments

a44a93d

fix duplication from mergeing

aec51ad

address reviewer's comment

64601ed

chilo-ms requested a review from yuslepukhin August 20, 2025 22:07

yuslepukhin reviewed Aug 21, 2025

View reviewed changes

jywu-msft removed the release:1.23.0 label Aug 28, 2025

chilo-ms mentioned this pull request Sep 9, 2025

Add onnxruntime_plugin_ep_onnx_test for plugin EPs to run onnx test #25942

Open

Merge branch 'main' into chi/onnx_test_runner_for_plugin_ep

76fabf5

Conversation

chilo-ms commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chilo-ms Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jywu-msft commented Aug 27, 2025

Uh oh!

chilo-ms commented Sep 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chilo-ms commented Aug 14, 2025 •

edited

Loading

yuslepukhin Aug 21, 2025 •

edited

Loading

chilo-ms Aug 28, 2025 •

edited

Loading