[WIP] Out-Tree EP feature by jslhcl · Pull Request #21450 · microsoft/onnxruntime

jslhcl · 2024-07-23T00:36:08Z

Description

Out-Tree EP feature.

Motivation and Context

Decouple ExecutionProvider from ONNXRuntime and make it a standalone class which will benefit 3rd party EP authors to write their own EP.
Binary compatibility is required for this feature.

samples/outTreeEp/out_tree_ep.cc

samples/c_test/test.cpp

samples/outTreeEp/out_tree_ep.h

@@ -0,0 +1,35 @@
+#pragma once


onnxruntime/core/framework/provider_adapter.h

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/c_test/test.cpp

onnxruntime/core/session/inference_session.cc

jslhcl · 2024-07-23T22:11:31Z

samples/outTreeEp/out_tree_ep.cc

+#ifdef __cplusplus
+extern "C" {
+#endif
+OrtExecutionProviderFactory* RegisterCustomEp() {


return Status instead #Resolved

Do we have to do this? This function will new a factory object by invoking its constructor which has no return type

… EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure

samples/outTreeEp/out_tree_ep.cc

samples/c_test/test.cpp

…roviderAdapter::Compile()

adrianlizarraga · 2024-07-30T21:24:49Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+} OrtMetaDef;
+
+typedef struct OrtIndexedSubGraph {
+  OrtMetaDef* meta_def; // TODO(leca): how to define a nested structure pointer?


Does this have to be a pointer to an OrtMetaDef? It may be simpler if this meta_def is contained by value instead. #Resolved

It looks we will check the pointer is null or not to distinguish between single node mode and fused node mode (See base class IExecutionProvider::GetCapability() which does not set this pointer and TryAssignSingleNode() which will check this pointer)

adrianlizarraga · 2024-07-30T21:34:37Z

samples/outTreeEp/out_tree_ep.cc

+
+OutTreeEp::OutTreeEp(const char* ep_type, const OutTreeEpInfo& ep_info) : info(ep_info) {
+    type = ep_type;
+    OrtExecutionProvider::GetCapability = [](const OrtExecutionProvider* this_, const OrtGraphViewer* graph, size_t* cnt, OrtIndexedSubGraph*** indexed_sub_graph) {


If I'm understanding correctly, the type of the OrtIndexedSubGraph*** indexed_sub_graph parameter is essentially asking the EP to fill out an array of pointers to OrtIndexedSubGraph objects.

Would it be simpler to change this to OrtIndexedSubgraph** indexed_sub_graph so that the EP fills out an array of OrtIndexedSubGraph objects directly? Each OrtIndexedSubgraph struct is a simple POD that can be created on the stack and copied around. It seems like it would result in less pointer tracking. #Resolved

Also, who is responsible for freeing this memory and when? If the EP allocates an array, then the EP should free it. The currently example leaks the allocations.

Edit: one possibility is to have onnxruntime call a new EP function (e.g., ReleaseOrtIndexedSubGraph()) so the the EP can free the memory. onnxruntime would call this once it is done using the indexed_sub_graph.

The problem is that we don't know how many OrtIndexedSubGraph would be before we call GetCapability() function. I will fix the leak issue in the coming commits

samples/outTreeEp/out_tree_ep.cc

onnxruntime/core/framework/provider_factory_adapter.h

@@ -0,0 +1,21 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/outTreeEp_kernel/kernel_ep.cc

@@ -0,0 +1,89 @@
+#include "kernel_ep.h"


samples/outTreeEp_kernel/kernel_ep.h

@@ -0,0 +1,36 @@
+#pragma once


include/onnxruntime/core/framework/ort_type_constraints.h

@@ -0,0 +1,15 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


onnxruntime/core/framework/ort_type_constraints.cc

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_model_wrapper.cc

@@ -0,0 +1,627 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_model_wrapper.h

@@ -0,0 +1,285 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_quant_params_wrapper.cc

@@ -0,0 +1,266 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_quant_params_wrapper.h

@@ -0,0 +1,147 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_utils.cc

@@ -0,0 +1,557 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_utils.h

@@ -0,0 +1,110 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/qnn_execution_provider.cc

@@ -0,0 +1,54 @@
+#include "qnn_execution_provider.h"


samples/qnnEp/qnn_execution_provider.h

@@ -0,0 +1,33 @@
+#pragma once


adrianlizarraga · 2024-12-04T01:36:05Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+   *
+   * \since Version 1.xx.
+   */
+  ORT_API2_STATUS(SessionOptionsAppendPluginExecutionProvider, _In_ OrtSessionOptions* options, _In_ const char* ep_name, _In_ OrtEnv* env,


Can this can be removed? We may be able to use the existing C API function called SessionOptionsAppendExecutionProvider. The existing C API does not take an OrtEnv parameter, but we can just get the default OrtEnv since there is only one per process.

yuslepukhin · 2024-12-04T18:15:44Z

General comment, please, re-format so the lines do not exceed 120 chars limit.

yuslepukhin · 2024-12-04T18:16:57Z

include/onnxruntime/core/framework/execution_provider.h

    return InlinedVector<const Node*>();
  }

+  bool IsBuiltInEp() const { return builtin_ep_; }


const

noexcept

yuslepukhin · 2024-12-04T18:18:15Z

include/onnxruntime/core/framework/ort_type_constraints.h

+#include <string>
+#include <set>
+
+struct OrtTypeConstraints {


truct OrtTypeConstraints {

Please, add documentation to all of the classes

yuslepukhin · 2024-12-04T18:19:40Z

include/onnxruntime/core/session/environment.h

 #include "core/common/logging/logging.h"
 #include "core/framework/allocator.h"
+#include "core/session/onnxruntime_c_api_ep.h"



Can we forward declare the types and avoid public header inclusion in the header?

yuslepukhin · 2024-12-04T18:20:09Z

include/onnxruntime/core/session/environment.h


 #include <atomic>
 #include <memory>
+#include <unordered_set>


<unordered_set>

Please, use inlined containers

yuslepukhin · 2024-12-04T18:26:12Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+   *
+   * \param[in] kernel_registry Opaque pointer of KernelRegistry object
+   * \param[in] custom_op Custom Op where the kernel compute function is defined
+   * \param[in] type_constraints


param[in] type_constraints

Given the fact that type_constraing pointer is not-const, does it mean it gets modified or the ownership is taken by the registry? Please, add this to the documentation.

yuslepukhin · 2024-12-04T18:27:31Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+  OrtExecutionProvider*(ORT_API_CALL* CreateExecutionProvider)(OrtExecutionProviderFactory* this_, const char* const* ep_option_keys, const char* const* ep_option_values, size_t option_size);
+} OrtExecutionProviderFactory;
+
+struct OrtGraphApi {


This is really a read-only graph viewer.
Suggest to name it appropriatly
Perhaps, the name should reflect the fact that this API is specifically for EP interaction.

yuslepukhin · 2024-12-04T18:29:50Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+  ONNXTensorElementDataType data_type;
+  const char* data;
+  size_t data_len;
+} OrtTensorRef;


This seems to be a repeat of the existing API such as TensorTypeAndShape. Can we re-use that part?

yuslepukhin · 2024-12-04T18:33:57Z

include/onnxruntime/core/session/onnxruntime_c_api.h

                  size_t num_external_initializer_files);
-};
+
+  /** \brief Create OrtDevice object.


/** \brief Create OrtDevice object.

Suggestion is to create a table that is separate from
the main API. Reasons:
Most of the clients do not need that code.
But it does affect language bindings.
For example, we now need to pad C# imported API structure, although it is unlikely we would ever need that in the C#, but if we do we can add that separate.

yuslepukhin · 2024-12-04T18:39:29Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+ORT_API2_STATUS(OrtGraph_IsConstantInitializer, const OrtGraphViewer* graph, const char* name, bool check_outer_scope, _Out_ bool* out);
+
+/** \brief Get the NodeIndex values of the graph nodes sorted in topological order
+ *


It looks like the output ptr data is const, please, specify that it is not to be freed by the client.

yuslepukhin · 2024-12-04T18:45:52Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+  size_t node_index_len;
+} OrtIndexedSubGraph;
+
+typedef struct OrtComputeContext {


Suggest adding a constructor for _cplusplus

yuslepukhin · 2024-12-04T18:46:48Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+/** \brief Gets the path of the owning model if any
+ *
+ * \param[in] graph The graph to query
+ * \param[out] model_path The path of the owning model if any


model_path The path of the owning model if any

Should this be an ORTTCHAR?

yuslepukhin · 2024-12-04T18:48:08Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+ * \param[out] out True if the graph is a subgraph
+ *
+ */
+ORT_API2_STATUS(OrtGraph_IsSubgraph, const OrtGraphViewer* graph, _Out_ bool* out);


ORT_API2_STATUS

Suggest this returns a bool

yuslepukhin · 2024-12-04T19:02:10Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+ * \remarks The caller is responsible for freeing the byte array using OrtFreeMem.
+ *
+ */
+ORT_API2_STATUS(OrtGraph_SerializeToArray, const OrtGraphViewer* graph, _Out_ void** data, _Out_ size_t* data_size);  // TODO(leca): review and discuss


oid** data

suggest this to be an array of bytes such as unsigned char

yuslepukhin · 2024-12-04T19:18:39Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+ * \param[in] onnx_model_path The file path to save to
+ *
+ */
+ORT_API2_STATUS(OrtGraph_DumpOnnxModel, const OrtGraphViewer* graph, const char* onnx_model_path);


const char* onnx_model_path)

ORTTCHAR

yuslepukhin · 2024-12-04T19:25:00Z

onnxruntime/core/framework/session_state.cc

-  auto& eps = GetExecutionProviders();
-  for (auto& ep : eps) {
-    ep->RegisterStreamHandlers(GetStreamHandleRegistryInstance(), *allocators_);
+  std::string register_resource_after = "";


= "";

redundant

yuslepukhin · 2024-12-04T19:25:11Z

onnxruntime/core/framework/session_state.cc

+  std::string register_resource_after = "";
+  IExecutionProvider* plugin_ep = nullptr;
+  for (auto& ep : execution_providers_) {
+    if (register_resource_after == "") {


== ""

.empty()

yuslepukhin · 2024-12-04T19:26:58Z

onnxruntime/core/framework/ort_type_constraints.cc

+    std::unordered_map<std::string, std::set<ONNXTensorElementDataType>>::iterator iter = type_constraints_.find(type_symbol);
+    if (iter == type_constraints_.end()) {
+        std::set<ONNXTensorElementDataType> types{type};
+        type_constraints_[type_symbol] = types;


types;

std::move()

…pCtxGraph

- Modify CMakeLists.txt for TRT EP plugin - Add "-l" for specifying EP plugin lib path for onnxruntime_perf_test

adrianlizarraga

Some memory/lifetime comments for OrtIndexedSubGraph in GetCapability()

adrianlizarraga · 2024-12-18T23:37:15Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+} OrtMetaDef;
+
+typedef struct OrtIndexedSubGraph {
+  OrtMetaDef* meta_def; // TODO(leca): how to define a nested structure pointer?


nit: I don't think it's necessary for this to be a separate memory allocation. I see that it was based on our internal IndexedSubGraph implementation, but in my view reducing the number of memory allocation makes things simpler. Perhaps meta_def can be stored inline and can add a boolean to indicate if the meta_def is valid.

typedef struct OrtMetaDef { bool is_valid; // ... } OrtMetaDef; typedef struct OrtIndexedSubGraph { OrtMetaDef meta_def; // ... } OrtIndexedSubGraph;

adrianlizarraga · 2024-12-19T00:00:42Z

onnxruntime/core/framework/provider_adapter.h

+  virtual std::vector<std::unique_ptr<ComputeCapability>> GetCapability(const GraphViewer& graph_viewer, const IKernelLookup& kernel_lookup) const override {
+    size_t cnt = 0;
+    OrtIndexedSubGraph** indexed_subgraph = nullptr;
+    if (ep_impl_->GetCapability) ep_impl_->GetCapability(ep_impl_, reinterpret_cast<const OrtGraphViewer*>(&graph_viewer), &cnt, &indexed_subgraph);


This is currently passing an OrtIndexedSubGraph*** to the EP's GetCapability() function and asking the EP to allocate an array of pointers to OrtIndexSubGraph objects. Because the EP is allocating the memory, we currently have a separate API to allow the EP to delete this memory.

I wonder if it would be simpler to allow ORT to pass in an OrtAllocator to the EP. The EP would use this ORT-owned allocator to allocate memory for the array. This woud remove the need for a separate C API to clean up the memory. Also, it would allow the parameter to be a OrtIndexedSubGraph** instead of OrtIndexedSubGraph***.

### Description This PRs sets the foundation for the EP ABI, which allows plugin-EPs to interface with ORT using a binary stable interface. A plugin-EP can be built separately from ORT and is not tied to a specific commit of ORT. Currently, this PR adds basic APIs necessary to allow an example plugin-EP to compile and run a simple model with a single `Mul` node. - Example plugin-EP implementation: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/onnxruntime/test/autoep/library/example_plugin_ep.cc - APIs: - Graph IR: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/include/onnxruntime/core/session/onnxruntime_c_api.h#L5290-L5439 - Plugin EP: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/include/onnxruntime/core/session/onnxruntime_c_api.h#L6177-L6481 - Example app code (from unit tests): https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/onnxruntime/test/autoep/test_autoep_selection.cc#L614 ### Motivation and Context Based on #21450 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>

snnn · 2025-07-03T18:00:52Z

This pull request has been automatically closed because it has merge conflicts and has been inactive for more than 30 days. Please rebase on the target branch and open a new PR.

### Description This PRs sets the foundation for the EP ABI, which allows plugin-EPs to interface with ORT using a binary stable interface. A plugin-EP can be built separately from ORT and is not tied to a specific commit of ORT. Currently, this PR adds basic APIs necessary to allow an example plugin-EP to compile and run a simple model with a single `Mul` node. - Example plugin-EP implementation: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/onnxruntime/test/autoep/library/example_plugin_ep.cc - APIs: - Graph IR: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/include/onnxruntime/core/session/onnxruntime_c_api.h#L5290-L5439 - Plugin EP: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/include/onnxruntime/core/session/onnxruntime_c_api.h#L6177-L6481 - Example app code (from unit tests): https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/onnxruntime/test/autoep/test_autoep_selection.cc#L614 ### Motivation and Context Based on microsoft#21450 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>

jslhcl added 2 commits July 17, 2024 20:39

opaque pointer for graph

0e6a80c

ORT C API RegisterOrtExecutionProviderLibrary work

c30a639

jslhcl requested review from adrianlizarraga, chilo-ms, jywu-msft and souptc July 23, 2024 00:36

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

ORT C-API SessionOptionsAppendOrtExecutionProvider work

7bfe57e

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

jslhcl commented Jul 23, 2024

View reviewed changes

onnxruntime/core/session/inference_session.cc Outdated Show resolved Hide resolved

jslhcl commented Jul 23, 2024

View reviewed changes

Test Relu with compile based EP, build work, runtime error of loading…

8e7d28d

… EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure

github-advanced-security bot found potential problems Jul 26, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc Fixed Show fixed Hide fixed

samples/c_test/test.cpp Fixed Show fixed Hide fixed

jslhcl added 2 commits July 29, 2024 17:49

prototype works with hardcode node_compute_info's index in ExecutionP…

808bfc3

…roviderAdapter::Compile()

prototype works without hardcode

49e396c

adrianlizarraga reviewed Jul 30, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc Outdated Show resolved Hide resolved

jslhcl added 2 commits July 31, 2024 20:38

fix comments for Compile function

e790105

add provider_factory_adapter.h

92f529d

github-advanced-security bot found potential problems Aug 1, 2024

View reviewed changes

onnxruntime/core/framework/provider_factory_adapter.h

@@ -0,0 +1,21 @@

// Copyright (c) Microsoft Corporation. All rights reserved.

Check warning

Code scanning / lintrunner

CLANGFORMAT/format

See https://clang.llvm.org/docs/ClangFormat.html. Run `lintrunner -a` to apply this patch.

jslhcl added 2 commits August 5, 2024 19:27

fix crash after introducing kernel based EP

3d83ed1

kernel based EP work with type constraint check commented out

e29499a

github-advanced-security bot found potential problems Aug 6, 2024

View reviewed changes

add kernel type constraints from out tree EP

f3678c4

github-advanced-security bot found potential problems Aug 7, 2024

View reviewed changes

jslhcl added 2 commits August 7, 2024 16:27

add API ReleaseOrtTypeConstraints

ac5ae0a

introduce qnn ep

0cc78e8

github-advanced-security bot found potential problems Aug 12, 2024

View reviewed changes

more graph/node C API

740a687

adrianlizarraga reviewed Dec 4, 2024

View reviewed changes

yuslepukhin reviewed Dec 4, 2024

View reviewed changes

jslhcl and others added 5 commits December 4, 2024 19:42

Fix Chi's comment and rollback the change on OrtGraph_CreateOrUpdateE…

e6be85e

…pCtxGraph

Add c++ wrapper for plugin ep api (#23045)

ce76175

refine ep plugin c++ wrapper (#23050)

fefbe27

[TRT EP Plugin] Fix issues of building on Windows (#23099)

ce6630c

- Modify CMakeLists.txt for TRT EP plugin - Add "-l" for specifying EP plugin lib path for onnxruntime_perf_test

refine ep plugin c++ wrapper (#23131)

dc6674b

adrianlizarraga reviewed Dec 19, 2024

View reviewed changes

adrianlizarraga mentioned this pull request May 31, 2025

Add GetCapability/Compile infrastructure for EP ABI #24887

Merged

snnn closed this Jul 3, 2025

		@@ -0,0 +1,14 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,21 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,15 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,627 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,285 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,266 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,147 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,557 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,110 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

Conversation

jslhcl commented Jul 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

Uh oh!

Uh oh!

Check warning

Check warning

Uh oh!

Uh oh!

jslhcl Jul 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adrianlizarraga Jul 30, 2024 • edited by jslhcl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adrianlizarraga Jul 30, 2024 • edited by jslhcl Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adrianlizarraga Jul 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jslhcl Jul 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Check warning

Choose a reason for hiding this comment

Uh oh!

yuslepukhin commented Dec 4, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jslhcl commented Jul 23, 2024 •

edited

Loading

jslhcl Jul 23, 2024 •

edited

Loading

adrianlizarraga Jul 30, 2024 •

edited by jslhcl

Loading

adrianlizarraga Jul 30, 2024 •

edited by jslhcl

Loading

adrianlizarraga Jul 30, 2024 •

edited

Loading

jslhcl Jul 31, 2024 •

edited

Loading

yuslepukhin Dec 4, 2024 •

edited

Loading

yuslepukhin Dec 4, 2024 •

edited

Loading

yuslepukhin Dec 4, 2024 •

edited

Loading

yuslepukhin Dec 4, 2024 •

edited

Loading