Themis.Inference 4.0.0

Themis.Inference

Themis.Inference provides the default ONNX Runtime path, prepared-model validation, typed inference, and ONNX packers. GGUF decision inference and packing live in the separate Themis.Inference.Gguf package. Themis.Inference.Ollama provides optional System One decision inference without depending on ONNX Runtime or tokenizers.

The package targets net10.0 and provides the default ONNX Runtime path. Windows applications can add the optional Themis.Inference.WinML package and call AddThemisWinML() from Themis.Inference.Extensions to enable Windows ML execution-provider discovery. For Qualcomm NPU inference, add the optional Themis.Inference.QNN package and call AddThemisQNN(); it selects the QNN NPU rather than the QNN GPU device.

Themis.Inference does not depend on Themis.Inference.Gguf. Applications using GGUF decision inference or packing should add that optional package directly and use Themis.Inference.Gguf.Runtime. The GGUF package does not depend on ONNX Runtime.

Hugging Face downloads

All Hugging Face artifact downloads are disabled by default. Runtime acquisition requires both ThemisOptions.AllowModelDownloads = true and an explicit IModelDownloader registration through Themis.Inference.Artifacts.AddThemisHuggingFaceModelDownloader(). A cache hit remains usable without network access. If a remote cache miss is not permitted or the downloader is not registered, inference fails with an actionable message explaining the required opt-in or local model setup.

Model pack requests also default to downloads disabled and require ModelPackRequest.AllowDownload = true for remote sources. Local model files can be packed without registering a downloader when the selected packer supports local inputs; the built-in GGUF packer accepts a local artifact directory.

RLCD GGUF decisions

Use Themis.Inference.Gguf and Themis.Inference.Gguf.Runtime for new GGUF code. AddThemisRlcdGgufDecisionInference() registers the typed decision provider for the model-specific GGUF backbone and RLCD safetensors head. It sends token IDs to an externally managed llama.cpp /embedding endpoint and runs the two-layer decision head in the .NET process. The application manages the external server lifecycle.

For an Ollama-served System One model, register AddThemisOllamaSystemOneDecisionInference(). The THEMIS_RLCD_GGUF_MODEL_PATH environment variable identifies the packed bundle directory. When unset, the default is %LOCALAPPDATA%\Rldc\models\laya-gguf.

Ollama System One decisions

AddThemisOllamaSystemOneDecisionInference() adapts Ollama's /v1/systemone endpoint to IDecisionInferenceProvider. It is distinct from the llama.cpp-backed RLCD GGUF provider. Ollama must be version 0.35.0 or later, and the selected local model must be trained for System One. Ollama-compatible GGUF support does not imply that an arbitrary GGUF model implements System One.

The provider maps choice, score, and noul results to Themis' typed answers. It uses the shared decision request's state, questions, and namespaced extras. Themis-owned model context is placed under extras.rldc, including the central question and any attribute-driven currentResults; application extras remain in their own namespaces. The Ollama adapter sends this context in a versioned System One state envelope. Task and Language are routing hints rather than model inputs. System One does not support tools, streaming, or generation controls.

[IncludePreviousResults] enriches DecisionRequest.Extras while keeping the original State unchanged. ONNX and RLCD GGUF providers project those extras into their existing model-input shape; requests without extras retain their existing input.

Decision inference and host-owned chat

The built-in ONNX contracts produce typed decision logits, and the RLCD GGUF provider requests embeddings before applying its typed decision head. Neither path performs autoregressive text generation or produces assistant tool calls. Themis is focused on typed RLCD decisions. Applications that need chat or tool generation should use a dedicated provider package and can connect that host client to typed decisions through Themis.Maf's decision proxy or function registration.

Inference contracts

OnnxDecisionInferenceProvider dispatches by the contract declared in the bundle manifest. Transformers sequence classification is an ONNX contract, not a separate runtime: it shares the provider's session, BPE input preparation, and answer mapping, while using the contract's single logits output. Keeping that route in a private method separates contract-specific work without duplicating service registration or native-session ownership.

Set THEMIS_MODEL_PATH to an absolute model-data directory when resolving a bundle by local name or caching downloads. Direct local ModelBundleDirectory configuration does not need that variable. Hugging Face credentials are read only from HF_TOKEN; downloaded bundles are cached beneath THEMIS_MODEL_PATH.

Runtime memory

OnnxDecisionInferenceProvider owns its ONNX session and artifact bundle and loads only the tokenizer required by the bundle contract. Dispose the provider when it is no longer needed. ONNX Runtime allocates native memory, so the managed heap alone does not describe inference memory use; compare process private bytes and working set with managed heap measurements.

Implementing custom model support

Themis' built-in ONNX provider is not a universal ONNX graph runner. ONNX manifest schema version 1 accepts these contracts:

• typed-heads: BPE tokenizer, token IDs and attention mask (optional token type IDs), fixed sequence length, and separate choice, score, and noul logits.
• transformers-sequence-classification-v1: BPE tokenizer, token IDs and attention mask (optional token type IDs), fixed sequence length, label count, and one logits output. Themis maps the declared labels to typed answers.
• rlcd-v1: token IDs, attention mask, marker positions/mask, question type, decision/action logits, sequence length, maximum option count, and an optional opt-in capability for multi-question batching.

RLCD bundles declare rlcdContractType: "rlcd-v1" and rlcdModelContract. The loader rejects the removed laya-v1 contract and layaModelContract manifest property; repack or migrate those manifests before upgrading.

All ONNX manifests identify the model and architecture, artifact paths, input and output shapes, tokenizer format, and files to validate. Each declared file may include a byte length and SHA-256 checksum. The loader currently requires the Hugging Face tokenizer JSON format with a BPE model and rejects unknown schema versions, runtimes, or contract types.

The built-in answer mapping is part of these contracts: choice logits become softmax probabilities; score logits become a probability distribution and an expected ordinal score; scalar noul logits use a sigmoid, while RLCD's two-class noul logits use softmax. RLCD bundles may provide temperature calibration by question type or question type and option-count bucket. Themis does not infer these semantics from arbitrary graph outputs. When an RLCD ONNX graph declares a separate action-logits output, Themis also exposes the first class's softmax probability under each answer's optional extras.laya.actionProbability field. The answer-level extras value can hold arbitrary JSON for provider-specific metadata without adding fields to every typed answer. The built-in packer reads rl_agent_config.json as well as the legacy laya_config.json for calibration. Multi-question batching is declared by rlcdModelContract.supportsQuestionBatching in the bundle manifest, defaults to false, and must be enabled only after validating the graph with a batch larger than one. ONNX dynamic dimensions alone are not evidence of safe batching. The DirectML batch-size-one compatibility override applies only when this capability is disabled. The built-in packers package/copy model artifacts and write manifests; they do not train or convert arbitrary models.

The separate RLCD GGUF bundle is also model-specific: it combines a GGUF backbone, a Hugging Face tokenizer/config, and a float32 safetensors head. The manifest declares backbone/head/tokenizer/config paths, encoder and head sequence lengths, hidden width, head layer count, and optional calibration temperatures. The .NET head implementation expects its defined tensor names and shapes, three question-type embeddings, attention heads of width 64, and a four-times-expanded feed-forward layer. These values and tensors are not a generic GGUF extension contract.

For a model that fits one of the supported contracts, an author can implement an IModelPacker and register it by format key with the Core packer registry. For a different tokenizer, graph contract, preprocessing rule, or answer mathematics, implement an IDecisionInferenceProvider and its own bundle reader/validator as needed. Register custom services through the AddThemis configureServices callback; set ThemisOptions.DefaultModel to the model ID your provider expects. Existing abstractions are extension points, not a promise that the built-in manifest loader will accept arbitrary fields.

For an ONNX model using an additional ONNX Runtime execution provider, register an IInferenceRuntimeInitializer that makes that provider available and an IOnnxExecutionProviderPolicy for its selection priority, provider options, session settings, and any worker-thread or serialized-run requirements. Add the policy through AddThemisOnnxExecutionProviderPolicy<TPolicy>(); the composition root passes registered policies to local and remote ONNX providers. Use PreferAccelerator, RequireAccelerator, or RequireGpu when the policy must explicitly select a discovered device; RuntimeDefault deliberately leaves selection and session creation to ONNX Runtime. When Windows ML or another runtime registers several execution providers, set RequiredExecutionProviderName to the exact ONNX Runtime provider name to select one deterministically. The named provider is mandatory: if it is not registered, inference fails rather than falling back to CPU. The execution preference still enforces the selected device type.

services.AddThemisWinML(new InferenceRuntimeConfiguration
{
    ExecutionPreference = InferenceExecutionPreference.RequireGpu,
    RequiredExecutionProviderName = "DmlExecutionProvider"
});

The built-in DirectML policy preserves DirectML's current session and serialized-run requirements. Unrecognized execution providers use the generic ONNX session defaults; they are not automatically validated or supported by Themis.

For Intel GPU/NPU execution, add the optional Themis.Inference.OpenVINO package and call AddThemisOpenVINO(). It registers the Intel OpenVINO plugin and applies its session policy while reusing the existing ONNX bundle and answer contracts. The default configuration requires an accelerator; use PreferAccelerator only when CPU fallback is acceptable. The upstream plugin package provides Windows x64 and Linux x64 native assets. Validate model compatibility and actual device execution on the target hardware; package assets or provider registration alone are not evidence of accelerator support.

For AMD GPU execution on Linux, add the optional Themis.Inference.MIGraphX package and pass the absolute path to an ONNX Runtime-compatible MIGraphX plugin library to AddThemisMIGraphX(pluginPath). The plugin must match the managed ONNX Runtime version used by Themis; ROCm and MIGraphX native dependencies are supplied by the deployment, not this package. The integration requires a GPU by default and does not fall back to CPU unless configured with PreferAccelerator. The AMD hardware path has not been validated in this repository.

For the native ONNX Runtime WebGPU plugin prototype on Linux x64, add Themis.Inference.WebGpu and call AddThemisWebGpu(). It is available through the NuGet preview lane, uses the upstream WebGPU plugin package under a -preview version, and requires a system Vulkan loader. The integration has not been validated on target hardware and is not a device/model support claim.

For Apple Silicon, add the optional Themis.Inference.CoreML prototype and call AddThemisCoreML(). The package is published with a -preview NuGet version. The host must load a matching macOS arm64 ONNX Runtime native library built with Core ML support (--use_coreml); the prototype does not bundle or replace that library. It prefers an exposed Neural Engine over GPU and matches MLComputeUnits to the selected device, but does not guarantee accelerator placement for any model node. Apple-hosted package, signing, and hardware-profile validation remain outstanding, and preview publication is not a platform or model support claim.

This boundary allows optional provider packages for technologies such as NVIDIA CUDA/TensorRT or Qualcomm QNN without putting their native dependencies in the base package. A vendor package should own its runtime registration, provider policy, platform/RID constraints, and hardware-specific tests. For a non-ONNX runtime, implement IDecisionInferenceProvider instead of forcing it through the ONNX policy interface.

using Themis.Core;
using Themis.Core.Extensions;
using Themis.Core.Interfaces;

services.AddThemisDecisionRouter(new DecisionRouterOptions("vendor-model-v1")
{
    RouteUnspecifiedRequestsToDefaultModel = true
});
services.AddSingleton<IDecisionInferenceProvider, VendorDecisionProvider>();

services.AddThemisModelPackers();
services.AddKeyedSingleton<IModelPacker, VendorModelPacker>("vendor-format-v1");

ModelPackRequest provides common source/output, model/tokenizer/config file, external-data, sequence-length, option-count, revision, and download controls. For additional author-specific settings, give the custom packer its own options/configuration and use a model-owned builder to create that request and the associated bundle.

An author may use a Builder pattern in their own package or tool to compose source discovery, model-specific packing, calibration, manifest writing, and provider registration. Themis does not require a generic Builder: its current extension seams are IModelPacker, IDecisionInferenceProvider, and IServiceCollection. A custom packer can receive additional author-specific options from DI rather than expanding the shared ModelPackRequest for unrelated model families.

License and model notices

The Themis inference code is licensed under MIT, and the NuGet package includes the Themis license text. Model weights and tokenizers are separate assets with their own terms; they are not relicensed by Themis or included in this NuGet package. The built-in packers copy root-level LICENSE, LICENCE, NOTICE, COPYING, and COPYRIGHT files from a source repository when present (including common extensions and suffixes). Check the upstream terms yourself and supply any required notices that are not available as root-level files before redistributing a bundle.