Themis.Inference.TensorRT 4.0.0-preview

Themis.Inference.TensorRT

Themis.Inference.TensorRT is an experimental ONNX Runtime integration for classic TensorrtExecutionProvider on Linux x64. Register it alongside AddThemis:

services.AddThemis(options => options.Model = "laya-production");
services.AddThemisTensorRT();

The default configuration requires a TensorRT GPU and fails when it is not available. TensorRT is appended before CUDAExecutionProvider, so nodes TensorRT cannot claim can use CUDA; ONNX Runtime's CPU provider remains available under its normal partitioning behavior. The CUDA fallback must identify exactly one GPU with the same vendor and device IDs as the selected TensorRT device. Multi-GPU hardware with ambiguous matching devices is rejected instead of routing fallback work to a different GPU.

The prototype targets net10.0 and linux-x64 with Microsoft.ML.OnnxRuntime.Gpu.Linux 1.31.0. Its packaged TensorRT provider library links libcudart.so.13, libcublas.so.13, libcudnn.so.9, libnvinfer.so.10, and libnvonnxparser.so.10. The NuGet package includes ONNX Runtime's TensorRT and CUDA provider libraries, but does not include these system CUDA, cuDNN, TensorRT, or NVIDIA driver dependencies. Install compatible host runtimes and expose their libraries to the Linux loader. This baseline identifies ABI majors; it does not establish an exact TensorRT patch release or a supported deployment combination.

The initial deployment assumption is Ubuntu 22.04 x64, net10.0, and linux-x64, with a single NVIDIA GPU. The integration test uses the pinned default RLCD bundle and typed choice, score, and noul request; model-derived input dimensions are not generalized into a broader coverage claim. The performance hypothesis is that TensorRT-compatible graph partitions may outperform CUDA on the same workload. No benefit or operator coverage has been measured; the integration profile must establish TensorRT placement before any comparison is meaningful.

Engine and timing caches are explicitly disabled. The prototype therefore does not share cache files across models, runtime versions, GPUs, containers, or concurrent session creation, and has no stale-cache or cache-permission behavior to rely on. Enabling persistent caches requires a separately validated per-model/runtime/GPU identity and writable, concurrency-safe cache policy.

The package is published with a -preview NuGet version as an experimental integration. No target GPU, TensorRT model coverage, native deployment, cancellation/concurrency behavior, or performance benefit has been validated. Verify TensorRT-assigned nodes in ONNX Runtime profiles before relying on execution or making a support claim. Windows ML's NvTensorRtRtxExecutionProvider is a separate integration and is not covered by this package.

References:

• ONNX Runtime TensorRT requirements and provider options
• ONNX Runtime CUDA requirements
• ONNX Runtime Linux installation requirements