API reference

register_heartbeat!(f) -> f

Register f(event::Symbol, detail) to receive this process's serving heartbeat:

• :up, detail::ServerConfig: a server finished bring-up and is accepting traffic
• :request, detail::String: a ModelInfer request for that model completed or failed
• :down, detail::ServerConfig: stop! tore the server down (a blocking serve fires it when run returns)
• :load / :unload, detail::String: the directory watcher loaded (or reloaded) or unloaded that bundle

:request fires on failure too: a failing client is still a client using this server. The callback is fired synchronously on the request task and must not block.

register_meta_model(name; run)

Called from a meta bundle's model.jl to register the orchestration function. run has the form run(inputs::Vector{NamedTensor}, call) -> Vector{NamedTensor}, where call(model_name, inputs) invokes another model. The meta runs as a scheduled unit holding the GPU exclusively, and call invokes the sub-model's compiled executable directly in-process (no queue re-entry, no gateway hop).

register_model(name; preprocess=identity, postprocess=identity)

Called from a bundle's model.jl to register custom pre/post-processing. Both hooks receive and return a Vector{NamedTensor}. Omitted hooks default to identity.

register_serve_guard!(f) -> f

Register f(cfg::ServerConfig) to be called before every serve brings a server up. A guard refuses by throwing; the error propagates to the caller of serve unchanged and nothing has been allocated yet. Guards run in registration order.

serve(node_path; worker=nothing, backend=ReactantBackend(), blocking=true) -> nothing | RunningServer

Load the node config at node_path, resolve this process's worker, bring up the runtime and its assigned models, and start the gRPC control plane. worker selects which worker entry to serve; it may be omitted when the node has exactly one worker. When blocking is false the server runs in the background and a RunningServer is returned (stop it with stop!).

serve_worker(node_path, worker; backend=ReactantBackend(), blocking=true)

Convenience alias for serve that names the worker positionally.

stop!(s::RunningServer)

Shut down a server started with serve(...; blocking=false). Stops the model-directory watcher (if running), closes the metrics endpoint (if running) and the gRPC server, halts the scheduler's dispatch loop, and tears down any registered shared-memory regions. Returns nothing.

AbstractInferenceIO

Interface for streaming a dataset through batched inference with infer_async / infer_sync. A concrete subtype implements length(io), item_input_bytes(io), infer_encode_chunk!(io, range, slot) (stage a chunk's inputs into the pool slot and return the input descriptors), and infer_decode_chunk!(io, range, response) (consume the response). The pool owns the staging bytes; the IO must not retain slot references past infer_encode_chunk!.

A subtype may optionally implement output_specs to read its outputs back through shared memory. The default is empty, which keeps every output inline (the response carries raw_output_contents as before). Declaring outputs is transparent to infer_decode_chunk!: the driver reads the shared-memory results back into the response before handing it over, so InferOutput works the same on either transport.

KServeModel(host, port, model_name; secure=false, max_batch_size=1, deadline=10.0, ...)
KServeModel(url, model_name; max_batch_size=1, deadline=10.0, ...)

A handle to one model served by a KServe V2 gRPC endpoint (a ReactantServer worker or the gateway). The second form parses a url of the form grpc://host:port (or host:port); grpcs/https selects a secure channel. max_batch_size caps how many items the batched infer_async / infer_sync drivers coalesce per request; deadline is the per-request timeout in seconds. max_send_message_length / max_receive_message_length bound a single gRPC message; each defaults to 512 MiB (matching the worker and gateway), or the REACTANT_CLIENT_GRPC_MAX_SEND_MSG_BYTES / REACTANT_CLIENT_GRPC_MAX_RECV_MSG_BYTES environment variable when set and the kwarg is omitted.

shared_memory controls system shared-memory transport for staged inputs/outputs:

• :auto (default): probe the server with IsSameIPCNamespace; use shared memory only if the server confirms it shares this client's IPC namespace. If the server returns false, or does not implement the RPC, fall back to inline transport. There is no silent runtime fallback.
• :on: force shared memory. The server confirming a different namespace is a hard error (fail loudly, no fallback). If the server does not implement IsSameIPCNamespace (e.g. stock Triton), shared memory is still attempted via SystemSharedMemoryRegister; making that work is then the caller's responsibility.
• :off: never use shared memory; always send inline. The probe is not sent.

OutputSpec(name, dtype, per_item_dims)

Declares an expected network output so the driver can read it back through shared memory. name is the output tensor name, dtype its element type, and per_item_dims its Julia column-major shape for a single item, excluding the leading batch dimension. For each chunk the driver reserves sizeof(dtype) * prod(per_item_dims) * items_in_chunk bytes in the staging slot and asks the server to write that output there. See output_specs.

InferInput(name, array) -> ModelInferRequest.InferInputTensor

Build a wire input tensor named name from a Julia array, shipping the bytes inline. You pass the array in its natural Julia column-major shape (W, H, …, N); the client reverses it to the network's row-major (N, …, H, W) internally (the bytes are unchanged). Pass a vector of these to the one-shot infer_sync(model, inputs). Variants taking an explicit Julia column-major shape and a typed contents vector are also provided.

InferOutput(name, response, dtype) -> Array
InferOutput(name, response) -> Array

Extract the output tensor named name from a ModelInferResponse as a Julia array. The wire row-major shape (N, …, H, W) is reshaped (no copy) to Julia column-major (W, …, N). Pass the element type as dtype for a type-stable result; the two-argument form reads the dtype from the response metadata.

infer_async(model, io::AbstractInferenceIO)

Run inference over every item in io, staging inputs through the shared-memory [BufferPool] and dispatching chunks concurrently (bounded by the pool's slot count). Each chunk acquires a disjoint slot, so this is safe to call from multiple threads against one model. Results are delivered through io's infer_decode_chunk!. Use infer_sync for serial dispatch.

infer_sync(model, io::AbstractInferenceIO)
infer_sync(model, network_inputs) -> ModelInferResponse

Synchronous inference. The AbstractInferenceIO form drives io one chunk at a time (no concurrency). The second form is a one-shot call: network_inputs is a vector of wire tensors built with InferInput, sent inline in a single ModelInferRequest, and the decoded ModelInferResponse is returned for reading with InferOutput.

kserve_init(; pool_bytes=DEFAULT_POOL_BYTES, n_slots=DEFAULT_POOL_SLOTS, shm_reprobe_interval=60.0)

Initialize the gRPC subsystem and (re)set the staging-pool parameters. n_slots is the number of fixed-size slots each pool is divided into, which bounds how many chunks can be in flight concurrently against a single pool. shm_reprobe_interval (seconds) is how often the background task re-probes endpoints that fell back to inline transport and restores shared memory if the server has recovered; set it to 0 (or negative) to disable that background recovery.

kserve_shutdown()

Tear down the client: unregister and unlink the shared-memory pool from every server it was registered with, drop the cached pools and per-URL routes, and shut the gRPC subsystem down. Pair with kserve_init.

manifest_io_spec(path) -> ModelIOSpec

Load a model's input/output spec from a manifest YAML at path, with no running server. Reuses the server's own wire encoding (encode_model_metadata), so the result matches model_io_spec for the same model. Suitable for an offline precompile or build-time check. The reported shapes are Julia column-major (batch axis last), so they read in the same axis order as the manifest's einsum letters.

model_io_spec(model::KServeModel) -> ModelIOSpec

Fetch a model's input/output spec from a running server over the ModelMetadata RPC. Throws if the server is unreachable or does not implement ModelMetadata; the call is explicit, so failing loudly is intended. Pair with validate_io or use it to introspect a model's I/O directly. The reported shapes are Julia column-major (batch axis last), matching the order you build arrays in; the row-major KServe wire shape is reversed away by the client.

output_specs(io) -> Vector{OutputSpec}

Outputs an AbstractInferenceIO declares for shared-memory read-back. Defaults to empty, which keeps every output inline (raw_output_contents) exactly as before: no output declaration means no shared memory, the safe fallback. Returning a non-empty vector opts the IO into shared-memory outputs and explicit-output mode: the request asks the server for exactly these outputs in this order, so every output the IO consumes must be declared. Outputs with data-dependent (dynamic) shapes cannot be sized ahead of time and must stay inline (leave them out, which means returning empty here).

validate_io(spec::ModelIOSpec, io::AbstractInferenceIO; items=1)
validate_io(model::KServeModel, io::AbstractInferenceIO; items=1)

Dry-run io against a model's true I/O spec without sending an inference request. Runs the user's infer_encode_chunk! and infer_decode_chunk! (see AbstractInferenceIO) once for the first items items against a synthetic, spec-shaped request and response, checking input/output names, dtypes, and shapes and surfacing indexing or shape errors in the user's own code as exceptions.

spec comes from manifest_io_spec (offline) or model_io_spec (online); the KServeModel form fetches it online. The harness runs the user's real methods, so it has side effects (it may write into the io's buffers at positions 1:items); call it on a representative or dummy io. Zeroed synthetic data does not exercise data-dependent branches.

The dry run runs the user methods inside with_bounds_checks, so indexing written with @infer_inbounds is bounds-checked here even though it elides in normal use. Bare @inbounds is not affected by that context; to catch out-of-bounds in bare-@inbounds code, start Julia with --check-bounds=yes (which Pkg.test does).

with_bounds_checks(f)

Run f with forced bounds checking enabled for any @infer_inbounds blocks reached during the call (task-local, restored on exit). validate_io uses this so the dry run honors bounds checks even where the IO would normally elide them.

scratch(slot, name, dims, T) -> PoolInferInput
scratch(slot, ["name1" => (dims1, T1), "name2" => (dims2, T2), ...]) -> Vector{PoolInferInput}

Carve one input buffer per named spec from the chunk's slot, advancing its cursor so the buffers occupy disjoint, contiguous byte ranges, and return the wire descriptors ready to hand back from infer_encode_chunk!. dims is the Julia column-major shape (per-item dims then the batch axis), or a bare integer for a vector. Get the writable views with pool_view: one descriptor returns one view, and splatting the vector returns all of them to destructure at once.

The scalar form returns one PoolInferInput; the vector form returns a Vector{PoolInferInput}. infer_encode_chunk! accepts either as its return value, so a single-input IO can return the scalar form directly without wrapping it in a vector.

function infer_encode_chunk!(io, r, slot)    n = length(r)    inputs = scratch(slot, ["INPUT__0" => ((4, n), Float32), "MASK" => ((n,), Int32)])    feats, mask = pool_view(inputs...)    for (j, i) in enumerate(r); feats[:, j] .= io.feats[i]; mask[j] = io.labels[i]; end    return inputsend

The returned vector is homogeneous (Vector{PoolInferInput}), so it never triggers element promotion the way a literal of differently-typed arrays would. item_input_bytes(io) must equal the per-item sum over these buffers, since the driver sizes the slot from it before infer_encode_chunk! runs. The lower-level path (carve a subslot, pool_view it, build InferInput by hand) is still supported and may be mixed into the returned vector.

@infer_inbounds expr

Like @inbounds, but the elision is conditional: outside a with_bounds_checks context the wrapped expr runs with bounds checks elided (as @inbounds); inside one it runs with bounds checks. Use it instead of @inbounds in infer_encode_chunk! / infer_decode_chunk! so a validate_io dry run still catches out-of-bounds indexing. Wrap a whole loop or block so the runtime branch is taken once, not per element.

function ReactantServerClient.infer_decode_chunk!(io::MyIO, r, response)    out = InferOutput("OUTPUT__0", response, Float32)    @infer_inbounds for (j, i) in enumerate(r)        io.results[i] = collect(out[:, j])    end    return nothingend

ReactantServerExport

Offline tooling that assembles ReactantServer model bundles (manifest.yaml, model[.b{N}].mlir, weights.safetensors) from StableHLO modules and weights, and the Reactant tracing frontend that turns a model + parameters into a bundle. This package is the bundle-format authority for producers; it is not depended on by the server runtime. The format is kept in sync with the server by the round-trip tests.

write_bundle/IOSpec are the low-level writer. export_bundle traces a Reactant model (the former LuxExport; it needs only Reactant, not Lux). PyTorch support (export_bundle(:pytorch, ...) and export_bundle(:torchscript, ...)) lives in a package extension that loads when PythonCall is present — using ReactantServerExport, PythonCall enables it.

IOSpec(name, dtype, shape; batch_axis=nothing, letters=nothing)

A tensor's client-facing spec. shape is the network (row-major) shape with a concrete value at the batch axis; batch_axis is the 0-based network axis that carries the batch dimension, or nothing. The manifest serialization encodes the shape as an einsum-style letter string ("chwn") plus a dims map of letter→size; the batch axis is emitted as n. Non-batch letters are auto-allocated from _AXIS_LETTERS and carry no semantic meaning across tensors.

letters optionally overrides that auto-allocation with explicit non-batch axis letters, one per non-batch axis in shape order (the batch axis is still emitted as n). Pass it to give axes meaningful names, e.g. ['w','h'] for an image input so its manifest reads whn. The reserved markers n/b are rejected.

assert_bundle_arity(dir) -> NamedTuple

bundle_arity_report as a refusal: raise unless every compiled module in dir takes exactly declared inputs + serialized weights arguments. Returns the report when it passes.

bundle_arity_report(dir) -> NamedTuple

Read a written bundle's three numbers and say whether they agree: the inputs its manifest.yaml declares, the tensors its weights.safetensors holds, and the arity of each compiled module.

Returns (; name, n_inputs, n_weights, expected, modules, servable), where modules is one (; module_file, entry_args, servable) per *.mlir. Nothing is thrown for a mismatch, so this can be run across a directory of bundles to triage them; assert_bundle_arity is the same check as a refusal.

bundle_entry_arity(mlir_path) -> Int

The number of arguments the compiled program in mlir_path takes, read out of the StableHLO portable artifact.

model*.mlir is a serialized vhlo artifact, so parse(MLIR.IR.Module, ...) fails with "dialect 'vhlo' does not implement the bytecode interface": it has to be deserialized against a Reactant context first. That is the whole reason this is a function rather than a regex at a call site.

collect_provenance(repo_dir=pwd(); extra=Dict()) -> Dict{String,Any}

Best-effort reproducibility provenance for a bundle. Captures the state of the model repo at repo_dir (repo_remote, git_commit, git_tree_sha1, git_branch, git_dirty) plus the export environment (julia_version, reactantserverexport_version) and an exported_at UTC timestamp. Merge the result into export_bundle's provenance; the frontends separately stamp their framework versions (reactant_version, or torch_version/torchax_version).

When the work tree is dirty, the uncommitted changes are captured as git_diff (git diff --binary HEAD), so git_commit plus the patch reconstructs the exact exported code (untracked files are not captured). write_bundle extracts git_diff into a working_tree.patch file in the bundle dir rather than embedding it in the manifest.

Every git field is best-effort: when git is missing, repo_dir is not inside a work tree, or a repo has no origin remote, the affected fields are omitted (with a warning for the non-repo case) and the function never throws. extra entries are merged last and override collected fields.

export_bundle(frontend::Symbol, args...; kwargs...) -> dir

Export a model to a bundle, naming the frontend explicitly so the call site is unambiguous:

• export_bundle(:lux, model, ps, st, example_input; ...) — a Lux-style model with separate parameters/state (any Reactant-traceable model(x, ps, st); Lux itself is not required).
• export_bundle(:reactant, f, inputs::Tuple, weights; ...) — any Reactant-traceable function f(inputs..., weights...) with explicit name => array weight pairs.
• export_bundle(:pytorch, model, example_inputs::Tuple; ...) — a torch.nn.Module; provided by the package extension, so it requires using PythonCall.

export_bundle(:lux, model, ps, st, example_input; dir, name, input_name="input",
              output_name="output", batch_sizes=[1], provenance=Dict()) -> dir

Trace model(x, ps, st) (taking the first return as the output) at each batch size and write a bundle. The batch dimension is the last Julia axis (Lux convention) and the leading network axis. Works for any Reactant-traceable model; Lux itself is not required.

export_bundle(:lux, model, ps, st, example_inputs::Tuple; dir, name,
              input_names=nothing, output_names=nothing,
              output_select = y -> (y isa Tuple ? y : (y,)),
              input_batch_axes=nothing, output_batch_axes=nothing,
              client_inputs=nothing, client_outputs=nothing,
              batch_sizes=[1], provenance=Dict()) -> dir

Multi-input / multi-output Lux export. client_inputs/client_outputs (each nothing or a Vector{IOSpec}) declare the wire-facing spec when a shipped model.jl postprocess transforms the executable tensors into different client tensors; they are passed through to write_bundle. Traces model(x_tuple, ps, st) where x_tuple is the tuple of array inputs the model's forward expects as its single positional argument. output_select(first(model(...))) maps the raw model output to the ordered tuple of arrays to export; non-array returns (e.g. an Int step count) are dropped by the selector and must not appear in its result. Weights are extracted from ps automatically (same as the single-array method). Per-tensor batch axes default to each array's last Julia axis (natural Julia batch-last, the Lux convention) so every exported input and output is batch-last without the caller hand-specifying it. Override with input_batch_axes/output_batch_axes (1-based Julia axes) when the batch axis is not last (e.g. a (D, N, K+1) output whose batch axis N is the middle one). An entry of nothing in either vector opts that one tensor out of batching (a genuinely unbatched tensor), so single-dispatch or partially-unbatched models can still be exported; note the server requires all executable inputs to be batched or none, so a per-tensor opt-out that leaves a mix of batched and unbatched inputs is rejected at load time.

export_bundle(:reactant, f, inputs::Tuple, weights; dir, name, input_names=nothing,
              output_name="output", provenance=Dict()) -> dir

Generic single-size export for any Reactant-traceable f(inputs..., weights...). weights is an ordered collection of name => array pairs. Produces one unbatched model.mlir.

write_bundle(dir; name, executable_inputs, executable_outputs, modules, weights,
             client_inputs=nothing, client_outputs=nothing, input_shapes=nothing,
             provenance=Dict()) -> dir

Write a bundle. With input_shapes === nothing, modules is a Dict keyed by batch size of StableHLO modules (or text or bytes); a single entry under key 0 writes model.mlir, otherwise each writes model.b{N}.mlir. weights is an ordered collection of name => array pairs whose order becomes the safetensors argument_order. The per-input batch axis is recorded in each input's IOSpec.batch_axis; the manifest derives batching.batch_dim from there.

input_shapes (a Vector{Vector{Int}} of compiled input-shape variants) turns on the multi-shape layout: the variable executable-input axes are marked -1 in executable_inputs, and each variant gives the concrete sizes of those axes in (input, axis) order. modules is then keyed by variant (each key a Vector{Int} equal to one input_shapes entry) and maps to that variant's batch-size module dict; the files are written as model.v{i}.*.mlir (i indexing input_shapes), all sharing the single weights.safetensors. The variants must share one set of batch sizes.

client_inputs/client_outputs (each nothing or a Vector{IOSpec}) declare the wire-facing spec when it differs from the executable spec, for bundles that ship a model.jl whose preprocess/postprocess transform between the two. They are emitted only when given. A variable (non-batch) axis is encoded by passing -1 for that axis size (e.g. the variable detection count of a postprocessed detector). The server requires these only when a model.jl is present, so the caller is responsible for also shipping model.jl into the bundle dir (see the converter handlers).

probe_worker_ready(node_path, worker=nothing) -> Bool

Resolve a worker's port from the node file and call its KServe ServerReady on localhost, returning whether it reported ready. Used as the worker container's healthcheck (a Julia replacement for the former Go reactant-healthprobe). worker may be omitted when the node has a single worker.

serve_gateway(gateway_path=nothing; blocking=true) -> nothing | RunningGateway

Load gateway.yml (listen addresses and the worker endpoint list), build the worker client pool, start the admin HTTP server and the readiness/discovery prober (which probes each endpoint's ServerReady and RepositoryIndex and swaps in the discovered routing table), and serve the KServe gRPC proxy. When blocking is false the server runs in the background and a RunningGateway is returned (stop it with stop!).

gateway_path may be omitted (or nothing) to configure the gateway from defaults and REACTANT_GATEWAY_* environment variables alone; the endpoint list then comes from REACTANT_GATEWAY_WORKERS. The node supervisor uses this to run an embedded gateway without a gateway.yml.

supervise(node_path; role=nothing, gateway_path=nothing, sink=stdout, env=ENV,
          install_signal_handlers=true, kwargs...) -> Int

Run the node: spawn one worker subprocess per visible GPU (and the embedded gateway in the default all-in-one role), multiplex their output onto sink with [name] line prefixes, restart children that die, and block until SIGTERM/SIGINT (or a crash-loop budget breach). Returns the process exit code.

TIMEOUT_NS_PARAM

Key of the request-level KV parameter carrying the caller's REMAINING budget in nanoseconds (relative, not an absolute timestamp). Like the shared-memory region parameters, this is an extension to KServe V2 passed through ModelInferRequest.parameters. It is relative so each hop converts it to its own local absolute deadline (time_ns() + budget), which makes it robust to cross-process monotonic-clock differences and lets it ride unchanged through the gateway's raw-byte request forwarding. See deadline_params.

BatchingSpec

The set of batch sizes a bundle was compiled for (compiled_batch_sizes). At inference the request's size along the batch axis must equal one of these; the scheduler coalesces requests up to a compiled size and selects the matching executable.

BufferPool(n_bytes; n_slots=8, use_shm=true, name="reactant_server_pool")

Allocate a staging pool of n_bytes divided into n_slots equal slots. slot_bytes is fixed at construction (n_bytes ÷ n_slots), not recomputed per request, so the allocator can hand disjoint slots to concurrent callers. A SHM-backed pool can be registered with a server; an inline pool (use_shm=false) is the fallback transport.

DType

Canonical element-type enumeration shared across the server, the single source of truth for dtype translation. The companion maps convert between three representations: the manifest token form (e.g. "f32", "bf16"), the Julia element type (e.g. Float32, BFloat16), and the KServe V2 wire datatype string (e.g. "FP32", "BF16").

The DType to XLA primitive-type mapping deliberately lives in the Reactant backend, not here, so this layer carries no Reactant dependency.

FP8 (F8E5M2, F8E4M3) has no standard KServe wire datatype, so those two variants are intentionally absent from the wire mapping and may appear only on executable-internal tensors, never on client-facing inputs or outputs. Conversions are performed by dtype_from_token, dtype_token, julia_type, dtype_of, dtype_size, kserve_string, and dtype_from_kserve.

DeadlineExceeded(model_name)

Raised when a request's deadline has already passed at dispatch admission: the scheduler refuses to begin GPU work that is already expired, and a meta orchestration refuses to issue a further sub-call once its budget is gone. It never interrupts a running PJRT/GPU call; it only declines to start new work. The gRPC layer maps it to DEADLINE_EXCEEDED.

Dim

A single axis of a tensor shape. kind is one of FIXED, BATCH, or VARIABLE; size is meaningful only when kind == FIXED (it is 0 otherwise). A FIXED dim has a concrete size, a BATCH dim is the batch axis (from the reserved n/b shape letters), and a VARIABLE dim (a -1 in the manifest dims map) is a dynamic non-batch axis.

EndpointsConfig

The listen addresses (the endpoints: config block): host, the gRPC port, the optional metrics_port for the Prometheus exposition endpoint (0 = disabled), and max_concurrent_requests, the cap on simultaneously-handled RPCs (0 = uncapped). For a worker fronted by the gateway, bind host to all interfaces (0.0.0.0) so the gateway and Prometheus can reach it; the gRPC port is usually derived from the node file's base_port and the metrics port from metrics_base_port.

max_concurrent_requests is a worker-level overload backstop: past the cap, new requests are shed immediately with RESOURCE_EXHAUSTED rather than queued. Keep it strictly above the gateway's per-worker outbound stream limit so it never sheds traffic the gateway has already admitted (and so it never rejects a meta-model's loopback sub-call); in single-worker mode (no gateway, clients hit the worker directly) it is the only inbound admission control.

GrpcConfig

gRPC transport limits for a worker's server. max_recv_msg_bytes / max_send_msg_bytes bound a single gRPC message in each direction (decode/encode caps, not allocations). Configured under the grpc: block of a worker (or node global:) config, with INFERENCE_SERVER_GRPC_MAX_RECV_MSG_BYTES / INFERENCE_SERVER_GRPC_MAX_SEND_MSG_BYTES environment overrides. Default DEFAULT_GRPC_MSG_BYTES (512 MiB).

InferRequest

A decoded inference request, the scheduler's unit of work. It names the target model (model_name), the requested_outputs the caller wants returned, and the input tensors (inputs, a Vector{NamedTensor}). deadline_ns is an absolute local time_ns() deadline (0 means none): a remaining-budget timeout carried over the wire is converted to this local absolute form at decode, so cross-process monotonic-clock differences never matter. The codec produces it from a wire ModelInferRequest; the scheduler and runtime consume only this transport-agnostic form.

Manifest

The parsed and validated manifest.yaml of a model bundle. It records the format_version, the bundle name and description, the executable input/output specs (executable_inputs/executable_outputs), the optional client-facing specs (client_inputs/client_outputs, present only when a model.jl transforms the I/O), the BatchingSpec, provenance metadata, and the derived 0-based input_batch_dim. Tensor specs are TensorSpec values; see TensorSpec and Dim for the einsum-style shape encoding.

ModelControlMode

How a worker manages the set of loaded models over its lifetime (mirrors NVIDIA Triton's model-control-mode). STATIC loads and compiles every bundle once at startup and never changes the set. DYNAMIC (the default) additionally runs a filesystem watcher that polls the model repository every model_poll_seconds and hot-swaps bundles as they are added, changed, or removed on disk. EXPLICIT cedes authority to an upstream control plane (the externally-managed residency behavior): no autonomous watcher, and non-resident models are not served until the control plane pins them.

ModelSchedConfig

Per-model scheduler overrides (an entry under scheduler.models). weight is the model's relative compute share (default 1.0, so all-default weights yield uniform shares; consulted only by the fair discipline). residency is the model's initial residency floor (see ResidencyState); nothing means unspecified, which the server resolves at startup to PINNED_SYSTEM when the on-demand weight cache is enabled (so every model's weights are materialized into host RAM and an on-demand GPU load is a pure host-to-device transfer) and UNPINNED otherwise. max_batch_size caps how many rows the scheduler coalesces into one dispatch of this model; nothing means uncapped. The cap does not change compiled shapes: a partial fill still pads up to the smallest compiled batch size, and a single request larger than the cap is still served (requests are never split).

NamedTensor(name, dtype, shape, data)
NamedTensor(name, data)

A named host tensor carried across the transport boundary as both an input and an output. It pairs a tensor name with its DType, its shape (Julia column-major dimensions), and the backing data array. The two-argument form derives dtype and shape from a typed host Array.

NumericsMode

How f32 matmul/convolution precision is resolved at model compile time (the runtime.numerics knob). NUMERICS_AUTO (the default) is hardware-adaptive: TF32 is used where the GPU supports it (compute capability >= 8.0) and explicit TF32 algorithms are stripped where it does not, so the same bundle compiles everywhere but its numerics follow the hardware. NUMERICS_F32 pins full f32 everywhere: TF32 DotAlgorithms are rewritten to f32 and every algorithm-free f32 dot_general/convolution gets precision_config = HIGHEST, so numerics are identical across GPU generations (the mode for validated deployments; costs tensor-core throughput on TF32-capable GPUs). NUMERICS_TF32 compiles exactly like auto (TF32 permitted; kernel choice stays with XLA/cuBLAS, and StableHLO cannot force TF32 for convolutions at all) but makes the hardware requirement a guarantee: the worker fails startup on hardware that cannot run TF32, rather than silently degrading per worker in a mixed fleet.

PoolAcquireTimeout(span, waited_ns)

Raised by acquire_slot! when a deadline_ns was supplied and passed before span contiguous slots became free. The waiter is dequeued before this is thrown, so it never stalls the line. Callers that carry a request deadline (e.g. a meta model's fan-out) translate this into their own deadline-exceeded error.

PrivateWeightStore()

The default store: each worker materializes its own host copy of a model's weights.

ResidencyMode

Who owns device residency on a worker, fixed at startup. SELF_MANAGED lets the worker autonomously transfer and evict weights above each model's floor within the device budget; EXTERNALLY_MANAGED makes a control plane authoritative (no autonomous eviction, non-resident models are not served until pinned). This is no longer configured directly: it is derived from ModelControlMode (explicit ⇒ EXTERNALLY_MANAGED, otherwise SELF_MANAGED).

ResidencyState

The residency floor an operator (self-managed) or control plane (externally-managed) sets for a model's weights. UNPINNED keeps no guaranteed residency (loaded from the mmap on demand); PINNED_SYSTEM guarantees the weights stay resident in host RAM (and must be transferred to the device before execution); PINNED_DEVICE guarantees them resident on the GPU for the server's lifetime (exempt from eviction).

RuntimeConfig

Runtime and device settings (the runtime: config block). backend selects CPU or CUDA execution; device_ordinal picks the GPU among several visible ones; mem_fraction is the fraction of device memory claimed for the pool; preallocate claims that pool up front; allow_cpu_fallback permits falling back to CPU when the device is unavailable; weight_cache_fraction is the single knob sizing on-demand (unpinned) weight residency, the fraction of the BFC arena (mem_fraction * device memory) devoted to all weights, pinned plus on-demand (1.0, the default, uses the whole arena minus measured scratch and wiggle; 0 disables the cache so every model's weights stay resident; resolved to a byte budget at startup, GPU only); weight_cache_wiggle_fraction is the fraction of the arena kept free as anti-fragmentation slack and feeds the startup auto-sizing (the worker probes each model's peak device usage and sizes the cache so pinned weights + on-demand + scratch + this slack fit); residency_mode selects self-managed or externally-managed residency; shared_host_weights opts the node into the shared-memory host-weight store so same-node workers share one host copy of each system-pinned model; and shared_host_weights_mode sets the permission bits (an octal string, default "666") for those shared regions and their lock files. The "666" default keeps cross-UID container setups working but is world-writable; "660" is recommended for production and multi-user systems. autotune (default true) enables XLA's GPU compile autotuner; set it false to compile with xla_gpu_autotune_level=0 (default gemm/conv algorithm selection, no timing trials), which removes autotuning's run-to-run non-determinism and its device scratch that otherwise inflates the startup memory probe (_probe_max_scratch!) on the first, un-cached start. autotune_cache (default nothing, meaning inherit Reactant's LocalPreferences.toml) toggles the persistent per-fusion autotune cache, and autotune_cache_dir (default "", inherit) sets its directory; both are applied to Reactant's compile cache at worker startup, so a container can drive them by env. numerics (default auto) sets the f32 matmul/convolution precision policy; see NumericsMode.

SchedulerConfig

Global scheduler settings (the scheduler: config block). ema_halflife_seconds is the half-life of the recent-compute moving average that drives fairness; recency_penalty_cap bounds the recency adjustment; coalescing_discount is the cost discount applied to coalesced batches; cost_ema_alpha is the smoothing factor for the learned per-batch-size cost; max_queue_depth caps each model's queue independently (a full model rejects new requests without affecting admission for the others); dispatch_timeout_seconds is the per-request execution timeout; discipline selects the inter-model ordering (see SchedulingDiscipline); compaction_interval runs worker-local memory compaction every N on-demand weight-cache loads (0 disables), the standalone (no-gateway) trigger, off by default so a gateway-fronted worker never self-compacts; and models holds the per-model ModelSchedConfig overrides.

SchedulingDiscipline

The inter-model dispatch ordering. FAIR is the deficit-weighted, cost-aware policy with per-model weights and the coalescing discount; FIFO serves in global arrival order; EDF (earliest-deadline-first) serves the model whose most-urgent queued request has the soonest deadline. Coalescing runs underneath all three.

Guidance: FAIR is for deployments where models share this worker with no upstream placement authority, a single-GPU worker or a multi-GPU fleet behind the round-robin gateway, where the worker itself must stop one model from crowding out the rest. Under the gateway's lpt_packing scheduling the gateway is the fairness authority (placement concentration plus the per-worker share cap), and workers must run FIFO or EDF so the two do not fight; lpt_packing supersedes the role FAIR played on manually-assigned multi-GPU fleets.

EDF is for deadline-sensitive deployments where requests carry a remaining-budget timeout (see the request-level timeout parameter). It degrades to FIFO whenever queued requests share the same deadline, so its only divergence from FIFO is to promote requests with less budget left, in practice the in-flight meta-model sub-calls that have already spent part of their budget on an earlier stage. It also sheds work that cannot finish within its learned compute cost (laxity), so it trades some throughput (batch fragmentation, and no per-model fairness) for hitting more deadlines under load. NOTE: EDF derives urgency purely from the deadline, so issuing different per-client deadlines for the same model reorders that model's service and therefore affects fairness across clients; uniform deadlines keep it behaving like FIFO.

ServerConfig

The fully resolved configuration for a single worker process, frozen for the process lifetime. It is produced from a node file (see node.jl) with environment-variable overrides applied, then checked by validate_config. Fields: model_dirs (bundle search paths), cache_dir, the RuntimeConfig, SchedulerConfig, and EndpointsConfig sub-configs, models_include (an allowlist of model names to load; empty means load all), model_poll_seconds (the dynamic-mode interval at which the worker re-scans its model_dirs for added, changed, or removed bundles and hot-swaps them), model_control_mode (see ModelControlMode: static, dynamic, or explicit), and the GrpcConfig grpc sub-config (gRPC message-size limits). ReactantServer.serve also accepts a ServerConfig directly.

SharedWeightStore(; mode=0o666)

Opt-in store backing each model's host weights with a node-level POSIX shared-memory region so same-node workers share one copy. See the file header for the flock-coordinated protocol. mode sets the permission bits for the regions and their lock files. The default 0o666 lets workers running as unrelated UIDs (for example different containers) share the regions, but is world-writable; 0o660 is recommended for production and multi-user systems.

TensorSpec

One tensor in a Manifest: its name, DType, shape (a Vector{Dim}), and batch_axis (the 1-based index of the batch dim, or nothing when the tensor is not batched).

WeightStore

Host-RAM residency backend for a model's materialized weights. PrivateWeightStore (the default) gives each worker its own copy; SharedWeightStore shares one copy per model across same-node workers through a POSIX shared-memory region. See the file header for the coordination protocol.

acquire_slot!(pool, span=1; deadline_ns=0) -> PoolSlot

Block until span physically contiguous slots are free, then return them as one slot whose [offset, offset + span*slot_bytes) range no other in-flight slot overlaps. Throws an ArgumentError immediately if span exceeds the pool's total slot count, since such a request could never be satisfied and would otherwise deadlock. Waiters are served in FIFO order. Pair with release_slot!, which frees the whole run.

deadline_ns is an absolute time_ns() deadline (0 = wait indefinitely, the default and prior behavior). When set, a waiter that has not acquired by the deadline throws [PoolAcquireTimeout] instead of parking past it. A one-shot timer wakes the waiter at the deadline even if no slot is released in the meantime, so a starved waiter fails fast rather than burning a request's whole budget in the park.

deadline_params(budget_ns) -> Dict{String,InferParameter}
deadline_params(PB::Module, budget_ns) -> Dict{String,InferParameter}

Build the request-level parameters map carrying a remaining-budget timeout of budget_ns nanoseconds (see TIMEOUT_NS_PARAM). A non-positive budget_ns yields an empty map (no deadline). Merge the result into a ModelInferRequest's parameters. The PB-first form builds the map from pb module PB (consumer packages pass their own generated module; the single-argument form uses this package's inference module).

decode_infer_request(msg, registry=nothing) -> DecodedRequest
decode_infer_request(PB::Module, msg, registry=nothing) -> DecodedRequest

Translate a decoded ModelInferRequest message into the boundary InferRequest. The transport (gRPC) hands us the already-decoded protobuf message, so the codec never touches wire bytes. Input tensor data is read from a registered shared-memory region (preferred when the tensor declares one), otherwise from rawinputcontents, otherwise from the typed contents field. The PB-first form accepts a message from pb module PB (consumer packages pass their own generated module; the shorter form uses this package's inference module).

decode_infer_response(msg) -> Vector{NamedTensor}
decode_infer_response(PB::Module, msg) -> Vector{NamedTensor}

Translate a ModelInferResponse into boundary NamedTensor outputs. Data is read from rawoutputcontents when present, otherwise from the typed contents field. Shared-memory-backed outputs are not supported on this path (the caller never requests them). The PB-first form accepts a message from pb module PB (consumer packages pass their own generated module).

encode_infer_request(model_name, inputs; requested_outputs=String[], id="") -> ModelInferRequest
encode_infer_request(PB::Module, model_name, inputs; requested_outputs=String[], id="") -> ModelInferRequest

Build a ModelInferRequest from boundary NamedTensor inputs, with tensor data inline in rawinputcontents. requested_outputs, when non-empty, names the outputs to return. The PB-first form builds the message from pb module PB (consumer packages pass their own generated module).

encode_infer_request_shm(model_name, inputs, region, offsets; requested_outputs, id)
encode_infer_request_shm(PB::Module, model_name, inputs, region, offsets; requested_outputs, id)

Encode a request whose inputs are ALL staged in shared-memory region at the given byte offsets (parallel to inputs); no raw_input_contents (the receiver reads each tensor via shm_read). The receiver must have region registered. This is all-or-nothing per request: the decode path treats raw_input_contents as parallel-to-inputs, so a request never mixes raw and SHM inputs. The PB-first form builds the message from pb module PB.

encode_infer_response(model_name, id, outputs) -> ModelInferResponse
encode_infer_response(PB::Module, model_name, id, outputs) -> ModelInferResponse

Build the response message with outputs entirely inline (rawoutputcontents). The transport serializes the returned message. The PB-first form builds the message from pb module PB (consumer packages pass their own generated module; the shorter form uses this package's inference module).

encode_infer_response(model_name, decoded, outputs, registry) -> ModelInferResponse
encode_infer_response(PB::Module, model_name, decoded, outputs, registry) -> ModelInferResponse

Build the response message, writing any output whose requested entry named a shared-memory region into that region (and referencing it in the response) instead of inline. rawoutputcontents holds the inline outputs in order. The PB-first form builds the message from pb module PB (consumer packages pass their own generated module).

encode_model_metadata(name, manifest, platform) -> ModelMetadataResponse
encode_model_metadata(PB::Module, name, manifest, platform) -> ModelMetadataResponse

Build a ModelMetadataResponse message from the manifest's client-facing I/O spec. The PB-first form builds the message from pb module PB (consumer packages pass their own generated module).

encode_repository_index(names) -> RepositoryIndexResponse
encode_repository_index(entries::AbstractVector{<:Pair}) -> RepositoryIndexResponse
encode_repository_index(PB::Module, names) -> RepositoryIndexResponse
encode_repository_index(PB::Module, entries::AbstractVector{<:Pair}) -> RepositoryIndexResponse

Build a RepositoryIndexResponse. The first form lists every model as READY (direct-client introspection). The second takes name => ready::Bool pairs and reports READY or UNAVAILABLE per model, so the gateway can discover which replicas actually serve a model (readiness reflects residency on the worker). The PB-first forms build the message from pb module PB.

is_meta(m::Manifest) -> Bool

True when the bundle is a meta model (a Julia orchestration over other models) rather than a regular StableHLO executable.

load_manifest(path) -> Manifest

Parse a manifest YAML file at path into a Manifest. This runs the structural parsing and validation of parse_manifest but not the bundle-directory checks in validate_manifest, so it is usable wherever only the manifest's contents are needed, for example a client deriving a model's I/O spec offline.

load_node(path) -> Dict{String,Any}

Read and structurally validate a node config file.

load_node_raw(path) -> Dict{String,Any}

Read a node config file without structural validation. Used by the supervisor, which may need to synthesize the workers: list (see materialize_node!) before validating.

materialize_host_weights!(store, key, digest, specs, fill!) -> Vector{Any}

Return a model's host weight Arrays (in weight order), populating them via fill!(arrays) when this worker is the one that must materialize them. specs is a vector of (eltype, dims) per tensor. For PrivateWeightStore the arrays are freshly allocated; for SharedWeightStore they alias a node-shared region.

materialize_node!(node, devices; cpu_workers=1) -> Vector{Union{String,Nothing}}

Prepare a raw node config for supervised single-container deployment: assign one visible device per worker and return the per-worker device selectors (in worker declaration order; nothing means the worker gets no CUDA_VISIBLE_DEVICES of its own, the CPU case).

With no workers: list, one is synthesized: worker0..workerN-1, one per device (or cpu_workers workers when devices is empty). An explicit workers: list wins: each worker is assigned devices[i] positionally, or devices[gpu+1] when the entry carries a gpu: key. The gpu: key is consumed here (deleted after assignment) so each child process, seeing a single device through CUDA_VISIBLE_DEVICES, resolves device ordinal 0. Assigning one device to two workers, or having more workers than devices, is a ConfigError. Call validate_node on the result before use.

node_gpus(node) -> :auto | Int | Vector{String}

Parse the optional top-level gpus: key: auto (the default when absent) asks the supervisor to enumerate visible devices; an integer is a device count (expanded to ordinals 0..n-1); a list gives explicit device selectors (ordinals or GPU UUIDs, passed to CUDA_VISIBLE_DEVICES verbatim).

node_server_config(node, worker) -> (ServerConfig, applied_overrides, worker_name)

Resolve the ServerConfig for one worker of a node. worker may be nothing when the node has exactly one worker (it defaults to that sole entry); otherwise it must name a defined worker. Environment overrides (INFERENCE_SERVER_*) are applied on top, as for any server config. Does not validate; call validate_config on the result.

release_host_weights!(store, key) -> nothing

Release a model's host weights. For SharedWeightStore this detaches the region and, if this was the last holder on the node (a non-blocking upgrade to an exclusive flock succeeds), unlinks the region and its lock file. A no-op for the private store. The caller must drop all references to the arrays first.

release_slot!(slot)

Return a slot acquired with acquire_slot! to the pool's free set, freeing every physical slot in its span. Releasing a derived subslot (index == 0) is a no-op. Throws if any slot in the span is already free (double release).

rename_host_weights!(store, old, new) -> nothing

Rekey a model's attached host-weight region from old to new (a model rename; the weights are unchanged). The region itself keeps its original content-addressed SHM name; only this worker's bookkeeping key moves, so a later release_host_weights!(store, new) detaches the same region. A no-op for the private store and when nothing is attached under old.

same_ipc_namespace(name) -> Bool

Return whether the POSIX shared-memory object name is visible in this process's IPC namespace. The client passes the name of an object it created; we answer true only if we can open that same object ourselves, which is what determines whether system shared-memory transport can work between the two processes. The open is read-only and detached immediately; nothing is registered or kept mapped. Any failure (object absent because we are in a different namespace, permission error, malformed name) is reported as false.

scratch(slot, dims, T) -> Array{T}
scratch(slot, [dims1 => T1, dims2 => T2, ...]) -> Vector{Array}

Carve one typed buffer per dims => T spec from slot in a single call, advancing the slot's cursor so the buffers occupy disjoint, contiguous byte ranges. Each buffer is an Array{T} aliasing the pool's backing (via pool_view; zero-copy, and uniform across SHM- and Memory-backed pools since it is just pool_base + offset); write into the returned arrays directly. dims is a shape tuple (or a bare integer for a vector).

This is the buffer-request interface shared by the meta-model call.scratch and the client driver: ask for ALL buffers up front in one call. The carved buffers' lifetime is bounded by slot (and by the pool, which the caller keeps alive); they become invalid once it is released.

shm_read(registry, name, offset, byte_size) -> Vector{UInt8}

Copy [offset, offset+byte_size) of the named region into a fresh byte vector. The copy is done while the region's lock is held, so a concurrent shm_unregister! of the same region cannot detach the mapping underneath the read.

shm_register!(registry, name, key, offset, byte_size)

Attach the existing POSIX shared-memory object key read-write and register it under name. Re-registering a name replaces (and detaches) the previous mapping.

shm_unregister!(registry, name)

Unregister and detach a region, or all regions when name is empty. Idempotent: unregistering a name that is not registered is a successful no-op (it matches KServe semantics and lets the gateway fan-out and the client's pre-emptive cleanup unregister succeed without a spurious error). Registration, by contrast, fails loudly (see shm_register!) so a bad region surfaces at register time rather than during inference.

shm_write!(registry, name, offset, bytes)

Copy bytes into [offset, offset+length(bytes)) of the named region. The copy is done while the region's lock is held, so a concurrent shm_unregister! of the same region cannot detach the mapping underneath the write.

validate_node(node)

Structural validation of a parsed node config. Raises ConfigError on a malformed file: missing model_repo, duplicate worker names, colliding ports, or a models: entry that targets an undefined worker. The models: map is optional for any node: when omitted, every worker loads every bundle in the repo; when present, it is a per-model override that pins the named models to device memory on the listed workers (see worker_raw_config).

weights_digest(key, specs; content=0) -> UInt64

Digest over a model's identity, weight layout (name, per-tensor dtype/size/shape, a format version), and a content token identifying the weight file's on-disk version (see weights_file_token). Two workers computing this for the same model and the same file agree on the same region key. Without the content token a weights-only update (same name, same tensor layout) would collide with the previous version's region and silently serve stale weights, including across server restarts (regions in /dev/shm outlive the process by design).

weights_file_token(path) -> UInt64

Identity token for a weights file's current on-disk version, folded into weights_digest via its content keyword. Derived from the file's size and mtime: cheap (no content read, so the shared store's zero-copy attach path stays zero-read) and identical across same-node workers stat'ing the same file. Returns 0 for an empty or missing path (hand-built test entries), which reproduces the layout-only digest.

wire_batch_spec(m::Manifest) -> Union{Nothing,Tuple{String,Int}}

The name and 1-based batch axis of the first WIRE-FACING input that declares one, or nothing when the model declares no batch axis at all (every request is then one item).

"Wire-facing" is client_inputs when the bundle has them and executable_inputs otherwise: client_inputs is present exactly when a model.jl intercepts the request (see validate_manifest), and without one the client sends the executable inputs directly.

This exists so a gateway can count the items in a request it never fully decodes. It cannot infer the axis: the position is per tensor and genuinely varies across bundles (whcn puts it last, nc first), the first input need not carry one at all (a cross-encoder's query is unbatched while its keys are batched), and it has moved between exports of the same model. Picking the first input that declares an axis mirrors _derive_input_batch_dim, and the name lets the reader match it against the request's own name-addressed tensors rather than trusting their order.

worker_names(node) -> Vector{String}

The worker names in declaration order.

worker_raw_config(node, name) -> Dict{String,Any}

Resolve a single worker's raw config dict from a (validated) node config: deep-merge global with the named worker's override blocks, then set model_dirs to the shared repo, the endpoint port from base_port, the runtime device ordinal from the worker's GPU, and the node-level shared_host_weights flag. Every worker loads every bundle in the repo; the top-level models: map is an optional per-model override that pins the named models to device memory on the listed workers (translated here into scheduler.models.<name>.residency: device). The result has the shape build_config consumes.

Scheduler

The deficit-weighted, cost-aware, coalescing dispatch engine. Concurrent requests land on per-model queues; a single dispatch loop runs one GPU execution at a time, coalescing same-model requests into one batched execution at a compiled size and sharing GPU time by relative model weight and a learned per-batch-size cost estimate. It holds the model registry, the backend, the device memory pool, the SchedulerConfig, per-model dispatch state, and an optional on-demand WeightCache. Submit work with infer and read observability counters with scheduler_metrics.

infer(scheduler, request) -> Vector{NamedTensor}

Submit a request and block until the scheduler returns the result. Re-raises any error captured during dispatch.

Runs the model's preprocess/postprocess hooks here, on the caller's task, rather than on the dispatch loop: preprocess before the request is queued, postprocess on the raw device outputs the loop hands back. Because each gRPC request runs on its own task, many requests' hook work proceeds in parallel and overlaps the single, serialized GPU execution. The dispatch loop coalesces and runs qr.prepared and never executes a request whose preprocess has not finished, since a request is only enqueued (made visible to the loop) after preprocess returns.

scheduler_metrics(scheduler) -> Dict{String,NamedTuple}

Snapshot per-model observability: dispatch count, total compute consumed, current recent-compute EMA, queue-wait P50/P99, the histogram of dispatch batch sizes, and residency.

set_policy!(scheduler, name; weight=nothing) -> nothing

Update a model's live scheduler policy. weight is consulted only by the fair discipline. Available in both residency modes. Raises on an unknown model.

set_residency!(scheduler, name, target::ResidencyState) -> ResidencyState

Move a model to the target residency floor. Only meaningful in externally-managed mode; the worker rejects it otherwise. Runs on the dispatch thread (the sole residency mutator) and blocks until applied. Raises on an unknown model or in self-managed mode.

control_status(scheduler) -> NamedTuple

A control-plane snapshot of the worker: its residency mode and scheduling discipline, plus a per-model view. A meta is reported as a single model whose footprint is the sum of its sub-models' weights; the internal sub-models are folded into it and not reported on their own, so the gateway packs and routes the group as one unit and never sees the stages.

compact!(cache, registry; reload) -> Int

Defragment the device arena. Frees every resident non-pinned device weight buffer at once so the BFC allocator coalesces its free list, then reloads each non-pinned model named in reload. Models not in reload are left freed; in self-managed mode they reload lazily through acquire! on their next dispatch. Host floors (host_weights) are never touched, so the reload is a pure host->device copy where a floor exists.

Device-pinned models are deliberately left in place: they are loaded once at startup, before any on-demand traffic, so they sit at the base of the arena, and compaction neither frees them nor pays to re-read them from disk. Only the on-demand churn region above them is freed and coalesced.

Runs on the dispatch-loop thread (the sole residency mutator), so no execution reads a buffer while it is being freed. In externally-managed mode acquire! will not autonomously load, so the reload list is ignored there (the control plane re-pins what it needs). Returns the number of models reloaded.

compact!(scheduler; reload_models=String[]) -> Int

Free the resident non-pinned device weights so the allocator coalesces its free list, then reload each model named in reload_models. An empty list frees only; non-pinned models not listed reload lazily on their next dispatch (self-managed mode). Device-pinned models are left in place (loaded once at startup, never reloaded from disk). A no-op when the on-demand weight cache is disabled. Runs on the dispatch thread (the sole residency mutator) and blocks until applied. Returns the number of models reloaded.

acquire!(cache, entry) -> nothing

Guarantee entry.executable.weights is resident before the model runs. Device-pinned and already-resident models return immediately (the latter is bumped to most-recently-used). In self-managed mode an evicted model is loaded autonomously, evicting LRU victims until it fits the budget (a model larger than the whole budget is loaded anyway after evicting everything, with a warning, since it cannot run otherwise). In externally-managed mode the worker does not autonomously load: an evicted model raises NotResidentError.

NotResidentError

Raised by acquire! in externally-managed mode when a request targets a model whose weights are not currently resident. A control plane is authoritative for residency in that mode, so the worker does not autonomously load; the model must be pinned first.

RunningServer

Handle to a server started with serve(...; blocking=false). It holds the resolved ServerConfig, the model registry, the running Scheduler, the device memory pool, the shared-memory registry, the underlying gRPC server, and the listen port. Pass it to stop! to shut the server down.

weight_budget(; arena, fraction, wiggle, max_scratch, pinned_bytes) -> (; on_demand_budget, weight_pool, scratch_ceiling, pinned_over_commit)

Solve for the on-demand weight-cache byte budget from the unified memory model. The arena holds, at the worst instant, pinned weights + on-demand weights + one model's execution max_scratch + a wiggle fraction of free slack:

scratch_ceiling  = (1 - wiggle) * arenaweight_pool      = min(fraction * arena, scratch_ceiling - max_scratch)   # for pinned + on-demandon_demand_budget = clamp(weight_pool - pinned_bytes, 0, weight_pool)

fraction caps the share of the arena devoted to weights (1.0 = all of it, minus scratch + wiggle). Pinned models reserve their footprint off the top, so the operator never subtracts it by hand. pinned_over_commit (pinned exceed the pool) flags a genuine over-commit no sizing can fix. Pure arithmetic, no device access, so it is unit testable.

weight_cache_stats(cache) -> NamedTuple

Snapshot the cache counters under its lock for observability.

free_weights!(backend, bufs) -> nothing

Release every device buffer in bufs. Used to evict an unpinned model's resident weights.

transfer_to_device(backend, pool, hosts) -> Vector{Any}

Transfer already-materialized host weight Arrays to device buffers, in order. This is the only cost paid on an on-demand load when weights are pinned in host RAM.

RetryPolicy(; enabled=true, initial_backoff=1.0, factor=2.0, max_backoff=Inf,
              min_budget=0.05, jitter=true, retry_unavailable=true)

Client-side retry policy for requests the server sheds under overload. When a ModelInfer call is rejected with RESOURCE_EXHAUSTED (a worker or gateway at its concurrency cap, see endpoints.max_concurrent_requests), the client waits initial_backoff seconds and retries, growing the interval by factor each attempt (1s, 2s, 4s, ...) up to max_backoff, until the model's deadline budget is spent. The per-attempt deadline sent to the server shrinks to the remaining budget, so the retries never push total wall time past the original deadline.

jitter picks each wait uniformly in [0, backoff] (full jitter) so that concurrent chunk requests do not synchronize into a retry storm against the same worker. min_budget (seconds) is the smallest remaining budget worth another attempt; below it the client stops and surfaces the shed error. retry_unavailable also retries UNAVAILABLE (no replica reachable). Deadline, NOTFOUND, and INVALIDARGUMENT errors always fail fast. Set enabled=false to restore the old fail-fast behavior.

RunningGateway

Handle to a gateway started with serve_gateway(...; blocking=false). Pass it to stop! to shut the gRPC server, the readiness prober, and the admin HTTP server down.