Skip to content

Refactor for stronger use of autocxx#28

Draft
theHamsta wants to merge 91 commits intorustnn:mainfrom
theHamsta:interfaces
Draft

Refactor for stronger use of autocxx#28
theHamsta wants to merge 91 commits intorustnn:mainfrom
theHamsta:interfaces

Conversation

@theHamsta
Copy link
Contributor

@theHamsta theHamsta commented Mar 9, 2026

The autocxx docs recommend using UniquePtr for C++ owned data as the easiest option. This allows to use TRT methods directly on the objects

Pin<&mut nvinfer1::ITensor>/Pin<&mut nvinfer1::ILayer> can't use a UniquePtr because their destructor is protected. We don't need to run the destructor for those objects since the objects since they are owned by INetworkDefintion.

I've added lifetime annotations for weights and the objects created by IBuilder->INetworkDefintion->IXXX . This allows to detect some segfault like this one we currently have in the code base

image(15)

Regarding API safety, we have two choices regarding the network definition API:

  • we could leave the API as-is: if NetworkDefintion, Tensor, Layers are !Send, "!Sync"`
    - they can not be transferred to other threads, even with a Mutex
    - API is safe since single-threaded we are only in one of the TRT functions, so mutable refs are exclusive
  • we could add &mut network/&network to all layer and tensor APIs:
    - we could add !Send safely, since all APIs always require either &mut network/&network. The borrow checker would then validate that network either are have multiple &network or one exclusive &mut network
    - the objects can be used from multiple threads if network is protected by a mutex or transferred to another thread.

TODO:

The mock API will also use the same autocxx API. The idea is that real objects would just use non-null UniquePtr while mock API uses nullptr (uniqueptr would then just panic if an API is not yet mocked)

Fixes #26
Fixes #19

@theHamsta
Copy link
Contributor Author

This is one big PR to test whether the whole concept would work. It can be split up into multiple PRs if necessary.

@theHamsta
Copy link
Contributor Author

theHamsta commented Mar 9, 2026

When checking with RustNN, I saw that requiring &mut Tensor in API is quite annoying. Since we're already requiring a borrow of network. We could probably use a UnsafeCell for Tensor for interior mutablity (before I had a mutex to allow this, but both mutex and refcell have a runtime cost).

&mut tensor make it difficult to do something like let square = network.add_element_wise(&mut a, &mut a). Also, it makes it impossible to take more than one mutable borrow into the hashmap that RustNN used (you need to remove and readd the entries.

Apart from that the API seems to work for RustNN (although quite some lifetime annotations were needed.

@theHamsta theHamsta force-pushed the interfaces branch 4 times, most recently from 56e2e4b to 80a91fa Compare March 11, 2026 14:31
@theHamsta theHamsta force-pushed the interfaces branch 4 times, most recently from c7f2dff to 2c1b9c1 Compare March 13, 2026 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lifetimes for weights Use all generated autocxx methods for all the types

1 participant