Add GetCapability/Compile infrastructure for EP ABI by adrianlizarraga · Pull Request #24887 · microsoft/onnxruntime

adrianlizarraga · 2025-05-28T18:40:03Z

Description

This PRs sets the foundation for the EP ABI, which allows plugin-EPs to interface with ORT using a binary stable interface. A plugin-EP can be built separately from ORT and is not tied to a specific commit of ORT.

Currently, this PR adds basic APIs necessary to allow an example plugin-EP to compile and run a simple model with a single Mul node.

Example plugin-EP implementation: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/onnxruntime/test/autoep/library/example_plugin_ep.cc
APIs:
- Graph IR: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/include/onnxruntime/core/session/onnxruntime_c_api.h#L5290-L5439
- Plugin EP: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/include/onnxruntime/core/session/onnxruntime_c_api.h#L6177-L6481
Example app code (from unit tests): https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/onnxruntime/test/autoep/test_autoep_selection.cc#L614

Motivation and Context

Based on #21450

…ible with OrtEpApi.

…uct sizes for ep api)

…tGraph to the model editor api

…ph/capability name

Copilot

Pull Request Overview

This PR lays the groundwork for a binary-stable plugin EP ABI in ORT, adds a sample plugin EP that runs a single Mul operator, and extends the core/session stack and C APIs to load and invoke plugin EPs.

Introduce example_plugin_ep.cc showing how a plugin-EP implements OrtEp and node compute callbacks.
Extend the C API (onnxruntime_c_api.h, ep_api.h/cc) and internal types (abi_ep_types.h, ep_plugin_provider_interfaces.*) to support graph/node enumeration and OrtNodeComputeInfo.
Update session creation and partitioner code (utils.cc, provider_policy_context.cc, graph_partitioner.cc, etc.) to recognize and wrap plugin EPs.

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/autoep/library/example_plugin_ep.cc	Sample EP implementation with `MulKernel` and EP API usage.
include/onnxruntime/core/session/onnxruntime_c_api.h	Add `OrtNodeComputeInfo` and `NodeComputeContext_NodeName`.
core/session/ep_api.h / ep_api.cc	Implement `Graph_Get`, `Node_Get`, and `EpGraphSupportInfo_AddSupportedNodes`.
core/session/ep_plugin_provider_interfaces.*	Define `PluginExecutionProviderFactory` and `PluginExecutionProvider`.
core/session/utils.cc	Construct plugin vs. internal EP factory based on devices.
core/session/provider_policy_context.cc	Invoke `CreateEp` instead of `ORT_NOT_IMPLEMENTED`.
core/session/ep_factory_internal.*	Change constructor signatures to use `gsl::span`.
core/graph/model_editor_api_types.* / model_editor_c_api.cc	Add internal/external conversion and null checks.
core/graph/graph.cc	Adapt `LoadFromModelEditorApiModel` to new `ModelEditorGraph`.
core/framework/graph_partitioner.cc	Update `PlaceNode` to accept `ComputeCapability` and handle subgraphs.
core/framework/compute_capability.h	Add `use_subgraph_name_as_fused_node_name` flag.

Comments suppressed due to low confidence (2)

onnxruntime/test/autoep/library/example_plugin_ep.cc:115

[nitpick] The variable name type_shape0 is ambiguous. Consider renaming to type_shape_info_input0 (and similarly for type_shape1) for clarity.

OrtTensorTypeAndShapeInfo* type_shape0 = nullptr;

onnxruntime/core/framework/graph_partitioner.cc:366

The code now takes a ComputeCapability but still refers to capability.sub_graph, which no longer exists on that type. You need to either restore the IndexedSubGraph parameter or update the logic to use the correct subgraph member from ComputeCapability.

if (nullptr == capability.sub_graph->GetMetaDef()) {

onnxruntime/test/autoep/library/example_plugin_ep.cc

include/onnxruntime/core/session/onnxruntime_c_api.h

onnxruntime/core/session/ep_plugin_provider_interfaces.cc

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

include/onnxruntime/core/session/onnxruntime_c_api.h

### Description Missed a few documentation errors in the [previous PR](#24887). This PR fixes the C/C++ API documentation generation action: https://github.com/microsoft/onnxruntime/actions/runs/15749541590 ### Motivation and Context Fix the C/C++ API documentation generation GitHub action.

This PRs adds additional Node_GetAttributes C API for EP ABI use. It's based on #24887

Runtime C API. PR microsoft#24887 allows plugin-EPs to interface with ORT using a binary stable interface. It is an important feature for a plugin EP to generate an EP context model as specified by https://onnxruntime.ai/docs/execution-providers/EP-Context-Design.html The EP needs to read and write the `ep_cache_context` attribute of type `ORT_OP_ATTR_STRING` as specified by https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/graph/contrib_ops/contrib_defs.cc#L3301-L3305 The current implemention of `ReadOpAttr` regards an attribute of type `ORT_OP_ATTR_STRING` as a null terminated string. https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/session/custom_ops.cc#L437 It is very common that `ep_cache_context` is a sequence of bytes, as specificed by ONNX https://github.com/ankane/onnxruntime-1/blob/95843a5dbc3100062be88bcb0d06fd36877f3f77/onnxruntime/core/protobuf/onnx-ml.proto#L148 This commit adds a new operator attribute type `ORT_OP_ATTR_BYTES` to the ONNX Runtime C API, which allows plugin EPs to read and write `ep_cache_context` as a sequence of bytes.

### Description Add a new ORT API `GetSessionOptionConfigEntries`. ### Motivation and Context #24887 allows plugin-EPs to interface with ORT using a binary stable interface. #24445 allows an EP to handle the extraction of EP options from the session option configurations. For an EP like VitisAI EP to comply with the requirements, it is necessary for a plugin-EPs to access all config entries in a session option. ```c++ OrtKeyValuePairs * kvps = nullptr; auto status = GetSessionOptionConfigEntries(session_option, &kvps); if(status) { throw status; } std::unique_ptr<OrtKeyValuePairs, void (*)(OrtKeyValuePairs*)> config_entries(kvps, ort_api.ReleaseKeyValuePairs); const char* const* keys = nullptr; const char* const* values = nullptr; size_t num_keys = 0; // Get keys and values from the config entries Ort::GetApi().GetKeyValuePairs(config_entries.get(), &keys, &values, &num_keys); for (size_t i = 0; i < num_keys; ++i) { // process keys[i] and values[i] } ```

Runtime C API. PR microsoft#24887 allows plugin-EPs to interface with ORT using a binary stable interface. It is an important feature for a plugin EP to generate an EP context model as specified by https://onnxruntime.ai/docs/execution-providers/EP-Context-Design.html The EP needs to read and write the `ep_cache_context` attribute of type `ORT_OP_ATTR_STRING` as specified by https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/graph/contrib_ops/contrib_defs.cc#L3301-L3305 The current implemention of `ReadOpAttr` regards an attribute of type `ORT_OP_ATTR_STRING` as a null terminated string. https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/session/custom_ops.cc#L437 It is very common that `ep_cache_context` is a sequence of bytes, as specificed by ONNX https://github.com/ankane/onnxruntime-1/blob/95843a5dbc3100062be88bcb0d06fd36877f3f77/onnxruntime/core/protobuf/onnx-ml.proto#L148 This commit adds a new operator attribute type `ORT_OP_ATTR_BYTES` to the ONNX Runtime C API, which allows plugin EPs to read and write `ep_cache_context` as a sequence of bytes.

### Description Add a new ORT API `GetSessionOptionConfigEntries`. ### Motivation and Context microsoft#24887 allows plugin-EPs to interface with ORT using a binary stable interface. microsoft#24445 allows an EP to handle the extraction of EP options from the session option configurations. For an EP like VitisAI EP to comply with the requirements, it is necessary for a plugin-EPs to access all config entries in a session option. ```c++ OrtKeyValuePairs * kvps = nullptr; auto status = GetSessionOptionConfigEntries(session_option, &kvps); if(status) { throw status; } std::unique_ptr<OrtKeyValuePairs, void (*)(OrtKeyValuePairs*)> config_entries(kvps, ort_api.ReleaseKeyValuePairs); const char* const* keys = nullptr; const char* const* values = nullptr; size_t num_keys = 0; // Get keys and values from the config entries Ort::GetApi().GetKeyValuePairs(config_entries.get(), &keys, &values, &num_keys); for (size_t i = 0; i < num_keys; ++i) { // process keys[i] and values[i] } ```

### Description To support writing an Execution Provider (EP) using the new EP ABI introduced in #24887, this PR adds value info for EP Context nodes to prevent shape inference errors during `Graph::Resolve`. ### Motivation and Context When creating a new EP Context node whose input is the output of another EP Context node, Graph::Resolve fails to set the type for the new node's arguments. This is because EP Context nodes do not have a TypeAndShapeInferenceFunction defined, as shown here: https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/graph/contrib_ops/contrib_defs.cc#L3289-L3337 As a result, an exception is thrown during shape inference: https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/graph/graph.cc#L2964 Specifically: EP Context nodes lack TypeAndShapeInferenceFunction, so onnx_inferred_type is unavailable. existing_type is nullptr due to the logic in: https://github.com/microsoft/onnxruntime/blob/9de58ac7a3d18d6ae7f7ae502b3f91361067f1b5/onnxruntime/core/session/ep_plugin_provider_interfaces.cc#L279 https://github.com/microsoft/onnxruntime/blob/9de58ac7a3d18d6ae7f7ae502b3f91361067f1b5/onnxruntime/core/session/ep_plugin_provider_interfaces.cc#L285 ### Implementation This PR attempts to add type information to EP Context nodes with best effort, ensuring that Graph::Resolve can proceed without errors even when type inference is not explicitly defined.

### Description Add a new ORT API `GetSessionOptionConfigEntries`. ### Motivation and Context microsoft#24887 allows plugin-EPs to interface with ORT using a binary stable interface. microsoft#24445 allows an EP to handle the extraction of EP options from the session option configurations. For an EP like VitisAI EP to comply with the requirements, it is necessary for a plugin-EPs to access all config entries in a session option. ```c++ OrtKeyValuePairs * kvps = nullptr; auto status = GetSessionOptionConfigEntries(session_option, &kvps); if(status) { throw status; } std::unique_ptr<OrtKeyValuePairs, void (*)(OrtKeyValuePairs*)> config_entries(kvps, ort_api.ReleaseKeyValuePairs); const char* const* keys = nullptr; const char* const* values = nullptr; size_t num_keys = 0; // Get keys and values from the config entries Ort::GetApi().GetKeyValuePairs(config_entries.get(), &keys, &values, &num_keys); for (size_t i = 0; i < num_keys; ++i) { // process keys[i] and values[i] } ```

### Description To support writing an Execution Provider (EP) using the new EP ABI introduced in microsoft#24887, this PR adds value info for EP Context nodes to prevent shape inference errors during `Graph::Resolve`. ### Motivation and Context When creating a new EP Context node whose input is the output of another EP Context node, Graph::Resolve fails to set the type for the new node's arguments. This is because EP Context nodes do not have a TypeAndShapeInferenceFunction defined, as shown here: https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/graph/contrib_ops/contrib_defs.cc#L3289-L3337 As a result, an exception is thrown during shape inference: https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/graph/graph.cc#L2964 Specifically: EP Context nodes lack TypeAndShapeInferenceFunction, so onnx_inferred_type is unavailable. existing_type is nullptr due to the logic in: https://github.com/microsoft/onnxruntime/blob/9de58ac7a3d18d6ae7f7ae502b3f91361067f1b5/onnxruntime/core/session/ep_plugin_provider_interfaces.cc#L279 https://github.com/microsoft/onnxruntime/blob/9de58ac7a3d18d6ae7f7ae502b3f91361067f1b5/onnxruntime/core/session/ep_plugin_provider_interfaces.cc#L285 ### Implementation This PR attempts to add type information to EP Context nodes with best effort, ensuring that Graph::Resolve can proceed without errors even when type inference is not explicitly defined.

### Description Add a new ORT API `GetSessionOptionConfigEntries`. ### Motivation and Context microsoft#24887 allows plugin-EPs to interface with ORT using a binary stable interface. microsoft#24445 allows an EP to handle the extraction of EP options from the session option configurations. For an EP like VitisAI EP to comply with the requirements, it is necessary for a plugin-EPs to access all config entries in a session option. ```c++ OrtKeyValuePairs * kvps = nullptr; auto status = GetSessionOptionConfigEntries(session_option, &kvps); if(status) { throw status; } std::unique_ptr<OrtKeyValuePairs, void (*)(OrtKeyValuePairs*)> config_entries(kvps, ort_api.ReleaseKeyValuePairs); const char* const* keys = nullptr; const char* const* values = nullptr; size_t num_keys = 0; // Get keys and values from the config entries Ort::GetApi().GetKeyValuePairs(config_entries.get(), &keys, &values, &num_keys); for (size_t i = 0; i < num_keys; ++i) { // process keys[i] and values[i] } ```

### Description To support writing an Execution Provider (EP) using the new EP ABI introduced in microsoft#24887, this PR adds value info for EP Context nodes to prevent shape inference errors during `Graph::Resolve`. ### Motivation and Context When creating a new EP Context node whose input is the output of another EP Context node, Graph::Resolve fails to set the type for the new node's arguments. This is because EP Context nodes do not have a TypeAndShapeInferenceFunction defined, as shown here: https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/graph/contrib_ops/contrib_defs.cc#L3289-L3337 As a result, an exception is thrown during shape inference: https://github.com/microsoft/onnxruntime/blob/5fdd4e4f2a2b6705a9a49a378a3b3496805067ee/onnxruntime/core/graph/graph.cc#L2964 Specifically: EP Context nodes lack TypeAndShapeInferenceFunction, so onnx_inferred_type is unavailable. existing_type is nullptr due to the logic in: https://github.com/microsoft/onnxruntime/blob/9de58ac7a3d18d6ae7f7ae502b3f91361067f1b5/onnxruntime/core/session/ep_plugin_provider_interfaces.cc#L279 https://github.com/microsoft/onnxruntime/blob/9de58ac7a3d18d6ae7f7ae502b3f91361067f1b5/onnxruntime/core/session/ep_plugin_provider_interfaces.cc#L285 ### Implementation This PR attempts to add type information to EP Context nodes with best effort, ensuring that Graph::Resolve can proceed without errors even when type inference is not explicitly defined.

### Description This PRs sets the foundation for the EP ABI, which allows plugin-EPs to interface with ORT using a binary stable interface. A plugin-EP can be built separately from ORT and is not tied to a specific commit of ORT. Currently, this PR adds basic APIs necessary to allow an example plugin-EP to compile and run a simple model with a single `Mul` node. - Example plugin-EP implementation: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/onnxruntime/test/autoep/library/example_plugin_ep.cc - APIs: - Graph IR: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/include/onnxruntime/core/session/onnxruntime_c_api.h#L5290-L5439 - Plugin EP: https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/include/onnxruntime/core/session/onnxruntime_c_api.h#L6177-L6481 - Example app code (from unit tests): https://github.com/microsoft/onnxruntime/blob/adrianl/ep-abi/onnxruntime/test/autoep/test_autoep_selection.cc#L614 ### Motivation and Context Based on microsoft#21450 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>

### Description Missed a few documentation errors in the [previous PR](microsoft#24887). This PR fixes the C/C++ API documentation generation action: https://github.com/microsoft/onnxruntime/actions/runs/15749541590 ### Motivation and Context Fix the C/C++ API documentation generation GitHub action.

This PRs adds additional Node_GetAttributes C API for EP ABI use. It's based on microsoft#24887

### Description Missed a few documentation errors in the [previous PR](microsoft/onnxruntime#24887). This PR fixes the C/C++ API documentation generation action: https://github.com/microsoft/onnxruntime/actions/runs/15749541590 ### Motivation and Context Fix the C/C++ API documentation generation GitHub action.

This PRs adds additional Node_GetAttributes C API for EP ABI use. It's based on microsoft/onnxruntime#24887

adrianlizarraga added 22 commits May 28, 2025 11:38

Create IExecutionProvider wrapper for OrtEp

732f29a

Add OrtEp::GetCapability and stub implementation in dummy ep

9c39966

Simplify GetCapability for OrtEp (no allocator). Make OrtGraph compat…

db2cda3

…ible with OrtEpApi.

Add API OrtGraph_GetNodes()

93900ba

Derive from OrtNode and OrtGraph instead of using variant (reduce str…

5676355

…uct sizes for ep api)

Separate model-editor and ep-api graph types into different files

353fdb8

Revert variable name change to reduce diff

62eb132

Add some comments

38b5398

Add unit test that adds the example plugin ep to a session.

09cea7a

Merge branch 'main' into adrianl/ep-abi

a659a80

Add C APIs to get a node's name and op_type

9f45c18

Update error message when user passes in the wrong kind of OrtNode/Or…

7cac7d1

…tGraph to the model editor api

Add ToInternal() methods to convert from OrtNode to its derived classes

5b38f4b

clean up

8abe39e

Rename AddSupportedNodes api func. Dont force eps to provide a subgra…

9abc3ff

…ph/capability name

Add OrtEp::Compile()

df0636d

Fill out Compile() wrapper function more

177828a

Fix annotation

0206de9

Update NodeComputeInfo handling

a3ab415

Merge branch 'main' into adrianl/ep-abi

59b5367

Finish making Mul work in example plugin EP

77b3037

rename file

1b616d8

jywu-msft requested review from chilo-ms, Copilot and skottmckay May 31, 2025 20:16

Copilot AI reviewed May 31, 2025

View reviewed changes

onnxruntime/test/autoep/library/example_plugin_ep.cc Outdated Show resolved Hide resolved

jywu-msft requested a review from HectorSVC May 31, 2025 20:20

adrianlizarraga commented May 31, 2025

View reviewed changes

Apply suggestions from code review

217aca1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

adrianlizarraga requested a review from Copilot May 31, 2025 20:43

edgchen1 reviewed Jun 18, 2025

View reviewed changes

include/onnxruntime/core/session/onnxruntime_c_api.h Outdated Show resolved Hide resolved

edgchen1 reviewed Jun 18, 2025

View reviewed changes

include/onnxruntime/core/session/onnxruntime_c_api.h Show resolved Hide resolved

edgchen1 previously approved these changes Jun 18, 2025

View reviewed changes

adrianlizarraga added 2 commits June 18, 2025 17:46

Renamed GetData -> GetMutableData, GetConstData -> GetData

5be3304

Merge branch 'main' into adrianl/ep-abi

e1fe569

adrianlizarraga dismissed edgchen1’s stale review via e1fe569 June 19, 2025 00:47

edgchen1 approved these changes Jun 19, 2025

View reviewed changes

adrianlizarraga merged commit 0ef8213 into main Jun 19, 2025
89 of 90 checks passed

adrianlizarraga deleted the adrianl/ep-abi branch June 19, 2025 04:19

adrianlizarraga mentioned this pull request Jun 19, 2025

Fix C/C++ documentation generation #25112

Merged

chilo-ms mentioned this pull request Jun 23, 2025

Add Node_GetAttributes C API for EP ABI #25143

Merged

chilo-ms added a commit that referenced this pull request Jun 28, 2025

Add Node_GetAttributes C API for EP ABI (#25143)

5ddd34e

This PRs adds additional Node_GetAttributes C API for EP ABI use. It's based on #24887

wcy123 mentioned this pull request Jul 3, 2025

Add a new ORT API GetSessionOptionConfigEntries #25277

Merged

wcy123 mentioned this pull request Jul 7, 2025

Add a new operator attribute type ORT_OP_ATTR_BYTES to the ORT C API #25300

Closed

wcy123 mentioned this pull request Jul 15, 2025

fix shape inference error for ep context nodes #25398

Merged

quic-ankus pushed a commit to CodeLinaro/onnxruntime that referenced this pull request Nov 25, 2025

Add Node_GetAttributes C API for EP ABI (microsoft#25143)

d860108

This PRs adds additional Node_GetAttributes C API for EP ABI use. It's based on microsoft#24887

qti-jkilpatrick pushed a commit to onnxruntime/onnxruntime-qnn that referenced this pull request Jan 26, 2026

Add Node_GetAttributes C API for EP ABI (#25143)

605d362

This PRs adds additional Node_GetAttributes C API for EP ABI use. It's based on microsoft/onnxruntime#24887

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GetCapability/Compile infrastructure for EP ABI#24887

Add GetCapability/Compile infrastructure for EP ABI#24887
adrianlizarraga merged 91 commits intomainfrom
adrianl/ep-abi

adrianlizarraga commented May 28, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

adrianlizarraga commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

adrianlizarraga commented May 28, 2025 •

edited

Loading