Worker API

Every exported name in ReactantServer that carries a docstring: the inference worker: serving, the scheduler, on-demand weights and the gRPC services. See the API overview for the other packages and the unexported internals the guides refer to.

ReactantServer.InvalidInputError — Type
InvalidInputError(msg)

Throw this from a bundle's preprocess hook (see register_model) to reject a request whose data the hook cannot accept, such as text that is not valid UTF-8 or a length tensor that does not match its batch. The gRPC layer maps it to INVALID_ARGUMENT, so the client is told the request itself is wrong and the gateway does not retry it on another replica. Any other exception a hook throws is a server-side failure and maps to INTERNAL. set_policy! raises it too, for a fair weight that is not positive and finite.

source
ReactantServer.register_heartbeat! — Method
register_heartbeat!(f) -> f

Register f(event::Symbol, detail) to receive this process's serving heartbeat:

  • :up, detail::ServerConfig: a server finished bring-up and is accepting traffic
  • :request, detail::String: a ModelInfer request for that model completed or failed
  • :down, detail::ServerConfig: stop! tore the server down (a blocking serve fires it when run returns)
  • :load / :unload, detail::String: the directory watcher loaded (or reloaded) or unloaded that bundle

:request fires on failure too: a failing client is still a client using this server. The callback is fired synchronously on the request task and must not block.

source
ReactantServer.register_meta_model — Method
register_meta_model(name; run)

Called from a meta bundle's model.jl to register the orchestration function. run has the form run(inputs::Vector{NamedTensor}, call) -> Vector{NamedTensor}, where call(model_name, inputs) invokes another model in-process (no gateway hop). The orchestration runs on the request task under the meta gate (a compute-only meta, with no sub-calls, skips it), and each call re-enters the scheduler as a committed request.

Meta models are experimental and their design is subject to change; do not build production pipelines on them. See the Meta Models guide.

source
ReactantServer.register_model — Method
register_model(name; preprocess=identity, postprocess=identity)

Called from a bundle's model.jl to register custom pre/post-processing. Both hooks receive and return a Vector{NamedTensor}. Omitted hooks default to identity.

source
ReactantServer.register_serve_guard! — Method
register_serve_guard!(f) -> f

Register f(cfg::ServerConfig) to be called before every serve brings a server up. A guard refuses by throwing; the error propagates to the caller of serve unchanged and nothing has been allocated yet. Guards run in registration order.

source
ReactantServer.serve — Method
serve(node_path; worker=nothing, backend=ReactantBackend(), blocking=true) -> nothing | RunningServer

Run one worker from the node file at node_path, in this process: load the file, resolve the worker entry, bring up the runtime and its models, and start the gRPC server (the KServe V2 inference API and the worker control service). worker selects which worker entry to serve; it may be omitted when the node file has exactly one worker. When blocking is false the server runs in the background and a RunningServer is returned (stop it with stop!). A blocking server serves until a drain is requested (request_drain!, or under the node supervisor end of file on stdin), then drains, shuts everything down and returns. Should its dispatch loop die, a blocking server instead exits the process with status 70, whether it is serving or draining, so the node supervisor sees the failure (and restarts the worker unless it is shutting down).

To run every worker in a node file, one process per device, use ReactantServerNode.supervise.

source
ReactantServer.stop! — Method
stop!(s::RunningServer; drain_seconds=0)

Shut down a server started with serve(...; blocking=false). From the start new ModelInfer calls are refused with UNAVAILABLE, and ServerReady, ModelReady, RepositoryIndex and /readyz report the worker not ready; with drain_seconds > 0 the calls already admitted (queued or running) first get up to drain_seconds to finish. Then it stops the model-directory watcher (if running), closes the gRPC server, halts the scheduler's dispatch loop, closes the metrics endpoint (if running), and tears down any registered shared-memory regions. Returns nothing.

source