SaT ONNX constructor crashes silently when run inside NVIDIA Triton Python backend

[samyak112]

Description

I’m trying to deploy wtpsplit (SaT models) using ONNX Runtime inside the NVIDIA Triton Python backend.
The same code works in:

1. Google Colab
2. A normal Python process inside a Docker container
3. Even in triton when i ran without ONNX runtime

However, when the same SaT constructor is executed inside Triton’s Python backend, the process exits silently during model initialization, causing Triton to mark the model as UNHEALTHY.

There is no Python exception, no traceback, and no stderr output — the Python backend process simply terminates.

I checked online and tried these things (Not sure i should have tried them but I did)

1. TOKENIZERS_PARALLELISM=false
2. OMP_NUM_THREADS=1, MKL_NUM_THREADS=1
3. Explicit HF cache paths (HF_HOME, TRANSFORMERS_CACHE)
4. Disabling TensorRTExecutionProvider
5. Restricting ORT providers to CUDA + CPU
6. Running Triton Python backend in spawn mode instead of fork
7. Verifying ORT providers include CUDA (['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'])

I checked the other issues but didnt found any similar issue, would be great if someone can help in debugging this

[markus583]

Hi @samyak112, thanks for the detailed error report!

It doesn’t look like a wtpsplit/SaT bug to me. If Triton’s Python backend goes unhealthy with no Python traceback, it’s usually a native crash in a dependency. In your setup the most likely culprit is onnxruntime + CUDAExecutionProvider inside the Python-backend (version / CUDA / driver mismatch etc.).

Two questions to further pinpoint the issue:

1. Does it work if you force CPU only (providers=["CPUExecutionProvider"])?
2. Can you try a minimal ORT init in initialize() (just import onnxruntime + create a tiny InferenceSession) and see if that alone kills it?

A potential practical workaround would be to run the ONNX model with Triton’s ONNXRuntime backend, and keep Python backend only for pre/post-processing (call the ONNX model via ensemble/BLS). That avoids loading ORT(CUDA) inside the Python session, which may be problematic.

[samyak112]

Thanks a lot, I will try this and let you know