Client API

Every exported name in ReactantServerClient that carries a docstring: the Julia client: models, the batched inference driver and the shared-memory transport. See the API overview for the other packages and the unexported internals the guides refer to.

ReactantServerClient.AbstractInferenceIO — Type
AbstractInferenceIO

Interface for streaming a dataset through batched inference with infer_async / infer_sync. A concrete subtype implements length(io), item_input_bytes(io), infer_encode_chunk!(io, range, slot) (stage a chunk's inputs into the pool slot and return the input descriptors), and infer_decode_chunk!(io, range, response) (consume the response). The pool owns the staging bytes; the IO must not retain slot references past infer_encode_chunk!.

A subtype may optionally implement output_specs to read its outputs back through shared memory. The default is empty, which keeps every output inline (the response carries raw_output_contents as before). Declaring outputs is transparent to infer_decode_chunk!: the driver reads the shared-memory results back into the response before handing it over, so InferOutput works the same on either transport.

source
ReactantServerClient.KServeModel — Type
KServeModel(host, port, model_name; secure=false, max_batch_size=1, deadline=10.0, ...)
KServeModel(url, model_name; max_batch_size=1, deadline=10.0, ...)

A handle to one model served by a KServe V2 gRPC endpoint (a ReactantServer worker or the gateway). The second form parses a url of the form grpc://host:port (or host:port); grpcs/https selects a secure channel. max_batch_size caps how many items the batched infer_async / infer_sync drivers coalesce per request, and must be at least 1 (an ArgumentError otherwise; describe_model reports 0 for a meta model and when the limit is unknown, so do not pass its value through unchanged); deadline is the per-request timeout in seconds. max_send_message_length / max_receive_message_length bound a single gRPC message; each defaults to 512 MiB (matching the worker and gateway), or the REACTANT_CLIENT_GRPC_MAX_SEND_MSG_BYTES / REACTANT_CLIENT_GRPC_MAX_RECV_MSG_BYTES environment variable when set and the kwarg is omitted.

shared_memory controls system shared-memory transport for staged inputs/outputs:

  • :auto (default): probe the server with IsSameIPCNamespace; use shared memory only if the server confirms it shares this client's IPC namespace. If the server returns false, or does not implement the RPC, fall back to inline transport. There is no silent runtime fallback. When /dev/shm has no room for the client's staging pool (its pages are allocated when it is created), the endpoint falls back to inline transport with a warning, and the background re-probe (see kserve_init) restores shared memory once the pool can be created.
  • :on: force shared memory. The server confirming a different namespace is a hard error (fail loudly, no fallback), and so is a staging pool /dev/shm has no room for. If the server does not implement IsSameIPCNamespace (e.g. stock Triton), shared memory is still attempted via SystemSharedMemoryRegister; making that work is then the caller's responsibility.
  • :off: never use shared memory; always send inline. The probe is not sent.

On macOS system shared memory is not supported yet: :auto sends inline without probing (and warns once per process), and :on throws an ArgumentError when the model is constructed.

source
ReactantServerClient.ModelDescription — Type
ModelDescription

What describe_models reports for one model:

  • name, kind (:model, or :meta for a Julia pipeline over other models), description
  • inputs, outputs: TensorDescriptions, in declaration order
  • batch_sizes: the batch sizes the bundle was compiled for, ascending (empty when it declares none)
  • max_batch_size: the most rows the worker coalesces into one execution, after any configured cap (0 for a meta model, or when the server does not know it). KServeModel refuses a max_batch_size below 1, so in that case pass the largest of batch_sizes, or 1 when it is empty
  • calls: a meta model's sub-models, which are internal: request the meta model, not them
  • variants: the compiled input shapes of a model whose inputs have variable axes, one Dict(input name => Julia column-major shape) per variant with the batch axis left at -1. Empty when no input has a variable axis, and when the model's preprocessing transforms its inputs (the compiled shapes then do not constrain what a client sends).
source
ReactantServerClient.OutputSpec — Type
OutputSpec(name, dtype, per_item_dims)

Declares an expected network output so the driver can read it back through shared memory. name is the output tensor name, dtype its element type, and per_item_dims its Julia column-major shape for a single item, excluding the leading batch dimension. For each chunk the driver reserves sizeof(dtype) * prod(per_item_dims) * items_in_chunk bytes in the staging slot and asks the server to write that output there. See output_specs.

A negative dimension is refused with an ArgumentError: an output with a variable axis (which describe_model reports as -1) cannot be sized ahead of time, so leave it out of output_specs and it comes back inline.

source
ReactantServerClient.TensorDescription — Type
TensorDescription

One input or output of a described model: its name, KServe datatype string (e.g. "FP32"), and shape in Julia column-major order, the order you build arrays in, with -1 on every batch or variable axis. axes gives the kind of each axis of shape: :fixed, :batch, or :variable.

source
ReactantServerClient.InferInput — Method
InferInput(name, array) -> ModelInferRequest.InferInputTensor

Build a wire input tensor named name from a Julia array, shipping the bytes inline as typed contents, encoded element by element (the one-shot path warns once about this cost; see infer_sync). You pass the array in its natural Julia column-major shape (W, H, …, N); the client reverses it to the network's row-major (N, …, H, W) internally (the bytes are unchanged). Pass a vector of these to the one-shot infer_sync(model, inputs). Variants taking an explicit Julia column-major shape and a typed contents vector are also provided.

source
ReactantServerClient.InferOutput — Method
InferOutput(name, response, dtype) -> Array
InferOutput(name, response) -> Array

Extract the output tensor named name from a ModelInferResponse as a Julia array. The wire row-major shape (N, …, H, W) is reshaped (no copy) to Julia column-major (W, …, N). Pass the element type as dtype for a type-stable result; the two-argument form reads the dtype from the response metadata.

source
ReactantServerClient.describe_models — Method
describe_models(url; names = String[], deadline = 10.0, grpc = gRPCClient.grpc_global_handle())
    -> Vector{ModelDescription}

Ask the server at url (the gateway or a worker, grpc://host:port) to describe the models in names, or every model it routes requests to when names is empty, sorted by name. The unfiltered list leaves out the internal sub-models of meta models; name one to describe it. Throws the server's NOT_FOUND for a name it does not serve. A server that predates the RPC answers UNIMPLEMENTED. Like every call on the global gRPC handle, it needs kserve_init first.

source
ReactantServerClient.infer_async — Method
infer_async(model, io::AbstractInferenceIO)

Run inference over every item in io, staging inputs through the shared-memory [BufferPool] and dispatching chunks concurrently (bounded by the pool's slot count). Each chunk acquires a disjoint slot, so this is safe to call from multiple threads against one model. Results are delivered through io's infer_decode_chunk!. Use infer_sync for serial dispatch.

When a chunk fails, no further chunk starts, and a chunk already in flight finishes the attempt it has sent but makes no further retry (a result that still arrives is delivered). The call waits for those chunks to finish, so once it returns or throws, nothing writes into io or holds a pool slot for it. The error is the first failing chunk's exception as the chunk raised it, not wrapped in a TaskFailedException, so it is caught by type just as from infer_sync, and an unrecoverable shared-memory failure falls back to inline transport the same way. When several chunks fail, an unrecoverable shared-memory failure takes precedence over the others, so that fallback happens even if another chunk failed first. The chunk's backtrace stays on the exception stack (current_exceptions()).

source
ReactantServerClient.infer_sync — Method
infer_sync(model, io::AbstractInferenceIO)
infer_sync(model, network_inputs) -> ModelInferResponse

Synchronous inference. The AbstractInferenceIO form drives io one chunk at a time (no concurrency) and stops at the first chunk that fails, throwing its exception. The second form is a one-shot call: network_inputs is a vector of wire tensors built with InferInput, sent inline in a single ModelInferRequest, and the decoded ModelInferResponse is returned for reading with InferOutput. The one-shot form encodes tensor data element by element as typed contents, so it emits a one-time performance warning (once per session) steering large or repeated workloads to the AbstractInferenceIO drivers, which stage through the buffer pool and ship raw bytes on every transport.

source
ReactantServerClient.kserve_init — Method
kserve_init(; pool_bytes=DEFAULT_POOL_BYTES, n_slots=DEFAULT_POOL_SLOTS, shm_reprobe_interval=60.0)

Initialize the gRPC subsystem and (re)set the staging-pool parameters. n_slots is the number of fixed-size slots each pool is divided into, which bounds how many chunks can be in flight concurrently against a single pool. shm_reprobe_interval (seconds) is how often the background task re-probes endpoints that fell back to inline transport and restores shared memory if the server has recovered; set it to 0 (or negative) to disable that background recovery.

source
ReactantServerClient.kserve_shutdown — Method
kserve_shutdown()

Tear down the client: unregister and unlink the shared-memory pool from every server it was registered with, drop the cached pools and per-URL routes, and shut the gRPC subsystem down. Pair with kserve_init.

source
ReactantServerClient.manifest_io_spec — Method
manifest_io_spec(path) -> ModelIOSpec

Load a model's input/output spec from a manifest YAML at path, with no running server. Reuses the server's own wire encoding (encode_model_metadata), so the result matches model_io_spec for the same model. Suitable for an offline precompile or build-time check. The reported shapes are Julia column-major (batch axis last), so they read in the same axis order as the manifest's einsum letters.

source
ReactantServerClient.model_io_spec — Method
model_io_spec(model::KServeModel) -> ModelIOSpec

Fetch a model's input/output spec from a running server (a worker or the gateway) over the KServe ModelMetadata RPC. Throws if the server is unreachable or does not implement ModelMetadata; the call is explicit, so failing loudly is intended. describe_model reports more: which axes are batch or variable, and the compiled batch sizes and input shapes. Pair with validate_io or use it to introspect a model's I/O directly. The reported shapes are Julia column-major (batch axis last), matching the order you build arrays in; the row-major KServe wire shape is reversed away by the client.

source
ReactantServerClient.output_specs — Method
output_specs(io) -> Vector{OutputSpec}

Outputs an AbstractInferenceIO declares for shared-memory read-back. Defaults to empty, which keeps every output inline (raw_output_contents) exactly as before: no output declaration means no shared memory, the safe fallback. Returning a non-empty vector opts the IO into shared-memory outputs and explicit-output mode: the request asks the server for exactly these outputs in this order, so every output the IO consumes must be declared. Outputs with data-dependent (dynamic) shapes cannot be sized ahead of time and must stay inline (leave them out, which means returning empty here).

source
ReactantServerClient.validate_io — Method
validate_io(spec::ModelIOSpec, io::AbstractInferenceIO; items=1)
validate_io(model::KServeModel, io::AbstractInferenceIO; items=1)

Dry-run io against a model's true I/O spec without sending an inference request. Runs the user's infer_encode_chunk! and infer_decode_chunk! (see AbstractInferenceIO) once for the first items items against a synthetic, spec-shaped request and response, checking input/output names, dtypes, and shapes and surfacing indexing or shape errors in the user's own code as exceptions.

spec comes from manifest_io_spec (offline) or model_io_spec (online); the KServeModel form fetches it online. The harness runs the user's real methods, so it has side effects (it may write into the io's buffers at positions 1:items); call it on a representative or dummy io. Zeroed synthetic data does not exercise data-dependent branches.

The dry run runs the user methods inside with_bounds_checks, so indexing written with @infer_inbounds is bounds-checked here even though it elides in normal use. Bare @inbounds is not affected by that context; to catch out-of-bounds in bare-@inbounds code, start Julia with --check-bounds=yes (which Pkg.test does).

source
ReactantServerCore.scratch — Method
scratch(slot, name, dims, T) -> PoolInferInput
scratch(slot, ["name1" => (dims1, T1), "name2" => (dims2, T2), ...]) -> Vector{PoolInferInput}

Carve one input buffer per named spec from the chunk's slot, advancing its cursor so the buffers occupy disjoint, contiguous byte ranges, and return the wire descriptors ready to hand back from infer_encode_chunk!. dims is the Julia column-major shape (per-item dims then the batch axis), or a bare integer for a vector. Get the writable views with pool_view: one descriptor returns one view, and splatting the vector returns all of them to destructure at once.

The scalar form returns one PoolInferInput; the vector form returns a Vector{PoolInferInput}. infer_encode_chunk! accepts either as its return value, so a single-input IO can return the scalar form directly without wrapping it in a vector.

function infer_encode_chunk!(io, r, slot)    n = length(r)    inputs = scratch(slot, ["INPUT__0" => ((4, n), Float32), "MASK" => ((n,), Int32)])    feats, mask = pool_view(inputs...)    for (j, i) in enumerate(r); feats[:, j] .= io.feats[i]; mask[j] = io.labels[i]; end    return inputsend

The returned vector is homogeneous (Vector{PoolInferInput}), so it never triggers element promotion the way a literal of differently-typed arrays would. item_input_bytes(io) must equal the per-item sum over these buffers, since the driver sizes the slot from it before infer_encode_chunk! runs. The lower-level path (carve a subslot, pool_view it, build InferInput by hand) is still supported and may be mixed into the returned vector.

source
ReactantServerClient.@infer_inbounds — Macro
@infer_inbounds expr

Like @inbounds, but the elision is conditional: outside a with_bounds_checks context the wrapped expr runs with bounds checks elided (as @inbounds); inside one it runs with bounds checks. Use it instead of @inbounds in infer_encode_chunk! / infer_decode_chunk! so a validate_io dry run still catches out-of-bounds indexing. Wrap a whole loop or block so the runtime branch is taken once, not per element.

function ReactantServerClient.infer_decode_chunk!(io::MyIO, r, response)    out = InferOutput("OUTPUT__0", response, Float32)    @infer_inbounds for (j, i) in enumerate(r)        io.results[i] = collect(out[:, j])    end    return nothingend
source