ReactantServer.jl

Production inference for Reactant-compiled models

KServe V2 over gRPC from one GPU to many, with compiled XLA models, Julia-first pre and postprocessing, and the most models per card.

Tutorials API Reference View on GitHub

⚡

KServe V2, natively

Speaks the KServe V2 inference API over gRPC, so standard Triton and KServe clients connect unchanged.

🚀

XLA under the hood

Models compile ahead of time through Reactant and XLA into device executables; the runtime is device agnostic, CUDA today with CPU for development.

🧩

Julia-first

A bundle's model.jl registers pre and postprocessing in plain Julia, and every convention follows Julia's, column-major with the batch axis last.

💾

On-demand weights

Weights stay in host RAM and stream to the GPU under a byte budget with cost-aware eviction, so a card serves more models than fit in VRAM.

🔀

A coalescing scheduler

A deficit-weighted, cost-aware scheduler merges same-model requests into one execution at a compiled batch size.

🔁

Hot reload

In dynamic mode the server watches the model repository and reloads bundles online, with no restart.

The server is built for workloads where many models share a GPU, such as computer vision and scientific computing. Every model is compiled ahead of time through Reactant and XLA into an optimized executable for the target device, and compiled models are fast. The runtime is device-agnostic: CUDA today, with CPU for development.

The design balances two resources that pull against each other. Compute is managed by a cost-aware scheduler that coalesces concurrent requests for the same model into one execution. Memory is managed by the on-demand weights cache, which keeps weights in host RAM and moves them onto the GPU within a byte budget. Scheduling policy, pinned models, and multi-GPU placement can all be changed at runtime, without a restart.

The server is a set of packages, split so that only the worker and the export tooling load Reactant and XLA; the client, the gateway, and the supervisor install without them. Architecture describes each package.

Get started

The Tutorials follow a model from export to inference, with two ways to serve it in between:

From Julia. Every package is in the General registry, so a Julia environment is all you need, on any machine, with or without a GPU. Start with Serve a Model.
With the container image. The supported deployment: a host needs only the NVIDIA driver and a container runtime. Start with Serve on One GPU.

To see real models, Serve an Object Detector and Serve Transformer Text Models take PyTorch models through export, serving, and inference.

The server is designed for a trusted network, so read Security before exposing an endpoint.

Reference

Bundles: the bundle format, the manifest's named-axis notation, and the export frontends.
Node Configuration: the one typed YAML file that describes a machine.
Scheduling: the cost-aware worker scheduler and batch coalescing.
On-demand Weights: host-RAM weights, the byte budget, and fragmentation-aware admission.
Multi-GPU Gateway: the reverse proxy and its scheduling modes.
Client Usage: the Reactant-free client and its shared-memory transport.
Meta Models (experimental): chaining models with data-dependent Julia between stages.
Deployment: how a node behaves, the container image and its settings, and the multi-node contract.
The API: every documented name, collected automatically.