Core API

Every exported name in ReactantServerCore that carries a docstring: the shared substrate the other packages build on: dtypes, protobuf messages, boundary types, the manifest, configuration, the codec, shared memory, the weight store and the staging BufferPool. ReactantServer re-exports all of it. See the API overview for the other packages and the unexported internals the guides refer to.

ReactantServerCore.HOST_CACHE_HALFLIFE_SECONDS — Constant
HOST_CACHE_HALFLIFE_SECONDS

How long a load keeps counting toward a host copy's place in the cache: each load of a model onto a device adds one to its count, and the count halves every this many seconds.

source
ReactantServerCore.SLOT_CONTROL_REGION_PARAM — Constant
SLOT_CONTROL_REGION_PARAM, SLOT_CONTROL_OFFSET_PARAM, SLOT_CONTROL_TOKEN_PARAM

Request-level parameters (ModelInferRequest.parameters) carrying a request's slot control: the registered region holding the control word (stringparam), the word's byte offset within that region's registered window (int64param, 8-byte aligned), and the attempt's token (int64_param). All three or none. A request with shared-memory outputs and slot control must keep those outputs in the control word's region.

source
ReactantServerCore.SLOT_RETIRE_SECONDS — Constant
SLOT_RETIRE_SECONDS

How long (seconds) a released slot may stay under a live writer before the client checks whether that writer still exists: a writer that died is reclaimed at once, one that is alive but stuck has its slot retired (never reused while it may still write) until it finishes.

source
ReactantServerCore.TIMEOUT_NS_PARAM — Constant
TIMEOUT_NS_PARAM

Key of the request-level KV parameter carrying the caller's REMAINING budget in nanoseconds (relative, not an absolute timestamp). Like the shared-memory region parameters, this is an extension to KServe V2 passed through ModelInferRequest.parameters. It is relative so each hop converts it to its own local absolute deadline (time_ns() + budget), which makes it robust to cross-process monotonic-clock differences and lets it ride unchanged through the gateway's raw-byte request forwarding. See deadline_params.

source
ReactantServerCore.BatchSizeMode — Type
BatchSizeMode

Which of a bundle's compiled batch sizes the worker loads (the runtime.batch_sizes knob). BATCH_SIZES_ALL (the default) loads every model.b{N}.mlir the manifest declares, so the scheduler can pick the largest size a dispatch can fill. BATCH_SIZES_LARGEST loads only the largest declared size (per input-shape variant) and never reads the others, so every dispatch runs the same compiled program, padded with zero rows when fewer requests are queued. That makes a row's result independent of how many other requests happened to be coalesced with it, at the cost of paying the full batch's compute on every dispatch. Unbatched bundles (model.mlir) are unaffected.

source
ReactantServerCore.BatchingSpec — Type
BatchingSpec

The set of batch sizes a bundle was compiled for (compiled_batch_sizes). At inference the request's size along the batch axis must equal one of these; the scheduler coalesces requests up to a compiled size and selects the matching executable.

source
ReactantServerCore.BufferPool — Method
BufferPool(n_bytes; n_slots=8, use_shm=true, name="reactant_server_pool", key=nothing)

Allocate a staging pool of n_bytes divided into n_slots equal slots. slot_bytes is fixed at construction, not recomputed per request, so the allocator can hand disjoint slots to concurrent callers: n_bytes ÷ n_slots rounded down to a multiple of the page size when that is at least a page, otherwise to a multiple of 64 bytes (8 below that; a single slot keeps its size), so every slot boundary keeps any element type aligned. A SHM-backed pool can be registered with a server; an inline pool (use_shm=false) is the fallback transport. A SHM pool's object name is minted per pool from name with a fresh random token (two pools never share a name, even one after the other in one process), unless an explicit key is given.

A SHM pool's object also holds the slot control area after the data slots (page aligned, one 128-byte word per slot), so its length, pool_region_bytes, is what a client registers. retire_after is SLOT_RETIRE_SECONDS unless a test shortens it.

Every page of a SHM pool's object is allocated when the pool is created, so staging into it, or a server writing outputs into it, can never find /dev/shm full (CORE-4). When /dev/shm cannot hold pool_region_bytes the constructor throws ShmSpaceError and leaves no object behind.

source
ReactantServerCore.DType — Type
DType

Canonical element-type enumeration shared across the server, the single source of truth for dtype translation. The companion maps convert between three representations: the manifest token form (e.g. "f32", "bf16"), the Julia element type (e.g. Float32, BFloat16), and the KServe V2 wire datatype string (e.g. "FP32", "BF16").

The DType to XLA primitive-type mapping deliberately lives in the Reactant backend, not here, so this layer carries no Reactant dependency.

FP8 (F8E5M2, F8E4M3) has no standard KServe wire datatype, so those two variants are intentionally absent from the wire mapping and may appear only on executable-internal tensors, never on client-facing inputs or outputs. Conversions are performed by dtype_from_token, dtype_token, julia_type, dtype_of, dtype_size, kserve_string, and dtype_from_kserve.

source
ReactantServerCore.DeadlineExceeded — Type
DeadlineExceeded(model_name)

Raised when a request's deadline has already passed at dispatch admission: the scheduler refuses to begin GPU work that is already expired, and a meta orchestration refuses to issue a further sub-call once its budget is gone. It never interrupts a running PJRT/GPU call; it only declines to start new work. The gRPC layer maps it to DEADLINE_EXCEEDED.

source
ReactantServerCore.Dim — Type
Dim

A single axis of a tensor shape. kind is one of FIXED, BATCH, or VARIABLE; size is meaningful only when kind == FIXED (it is 0 otherwise). A FIXED dim has a concrete size, a BATCH dim is the batch axis (from the reserved n/b shape letters), and a VARIABLE dim (a -1 in the manifest dims map) is a dynamic non-batch axis.

source
ReactantServerCore.EndpointsConfig — Type
EndpointsConfig

The listen addresses (the endpoints: config block): host, the gRPC port, the optional metrics_port for the Prometheus exposition endpoint (0 = disabled), and max_concurrent_requests, the cap on simultaneously-handled RPCs (0 = uncapped). For a worker fronted by the gateway, bind host to all interfaces (0.0.0.0) so the gateway and Prometheus can reach it; the gRPC port is usually derived from the node file's base_port and the metrics port from metrics_base_port.

max_concurrent_requests is a worker-level overload backstop: past the cap, new requests are shed immediately with RESOURCE_EXHAUSTED rather than queued. Keep it strictly above the gateway's per-worker outbound stream limit so it never sheds traffic the gateway has already admitted (and so it never rejects a meta-model's loopback sub-call); in single-worker mode (no gateway, clients hit the worker directly) it is the only inbound admission control.

source
ReactantServerCore.ErrorDetail — Type
ErrorDetail

How much an INTERNAL error tells the client: the grpc.error_detail key of the worker and the gateway. ERROR_DETAIL_REDACTED (redacted, the default) sends only a generic message naming a request id and the server, and logs the full exception and backtrace under that same id, so a production client learns nothing about the server's internals but an operator can still find the failure. ERROR_DETAIL_FULL (full) also sends the exception text (capped at 1 KiB), for development and testing. Every other status code always carries its explanation.

source
ReactantServerCore.GrpcConfig — Type
GrpcConfig

gRPC transport settings for a worker's server. max_recv_msg_bytes / max_send_msg_bytes bound a single gRPC message in each direction. The server allocates a received message's full declared length as soon as its length prefix arrives, so the receive cap also bounds that buffer per request. error_detail (an ErrorDetail) is how much an INTERNAL error tells the client. rapid_reset_protection (default false) keeps the HTTP/2 Rapid-Reset (CVE-2023-44487) protection of the worker's listener: a client connection may then have 100 streams open at once, and the worker ends it once the client has reset more than 200 streams over the connection's life, which a client does each time it gives up on a call. Off, a client is never disconnected for giving up on calls, and endpoints.max_concurrent_requests is what bounds one that resets streams. Configured under the grpc: block of a worker (or node global:) config, with INFERENCE_SERVER_GRPC_MAX_RECV_MSG_BYTES / INFERENCE_SERVER_GRPC_MAX_SEND_MSG_BYTES / INFERENCE_SERVER_GRPC_ERROR_DETAIL / INFERENCE_SERVER_GRPC_RAPID_RESET_PROTECTION environment overrides. Defaults: DEFAULT_GRPC_MSG_BYTES (512 MiB), redacted and false.

source
ReactantServerCore.InferRequest — Type
InferRequest

A decoded inference request, the scheduler's unit of work. It names the target model (model_name), the requested_outputs the caller wants returned, and the input tensors (inputs, a Vector{NamedTensor}). deadline_ns is an absolute local time_ns() deadline (0 means none): a remaining-budget timeout carried over the wire is converted to this local absolute form at decode, so cross-process monotonic-clock differences never matter. The codec produces it from a wire ModelInferRequest; the scheduler and runtime consume only this transport-agnostic form. slot_control is the client's shared-memory slot control for the request (see SlotControl), or nothing; the scheduler consults it at dispatch through request_cancelled.

source
ReactantServerCore.Manifest — Type
Manifest

The parsed and validated manifest.yaml of a model bundle. It records the format_version, the bundle name and description, the executable input/output specs (executable_inputs/executable_outputs), the optional client-facing specs (client_inputs/client_outputs, present only when a model.jl transforms the I/O), the BatchingSpec, provenance metadata, and the derived 0-based input_batch_dim. Tensor specs are TensorSpec values; see TensorSpec and Dim for the einsum-style shape encoding.

source
ReactantServerCore.ModelControlMode — Type
ModelControlMode

How a worker manages the set of loaded models over its lifetime (mirrors NVIDIA Triton's model-control-mode). STATIC loads and compiles every bundle once at startup and never changes the set. DYNAMIC (the default) additionally runs a filesystem watcher that polls the model repository every model_poll_seconds and hot-swaps bundles as they are added, changed, or removed on disk. EXPLICIT cedes authority to an upstream control plane (the externally-managed residency behavior): no autonomous watcher, and non-resident models are not served until the control plane pins them.

source
ReactantServerCore.ModelSchedConfig — Type
ModelSchedConfig

Per-model scheduler overrides (an entry under scheduler.models). weight is the model's relative compute share, a positive finite number (default 1.0, so all-default weights yield uniform shares; consulted only by the fair discipline). residency is the model's initial residency floor (see ResidencyState); nothing means unspecified, which the server resolves at startup to UNPINNED (swapped on demand, with host RAM copies within runtime.host_weight_budget), or to PINNED_SYSTEM when the on-demand weight cache is enabled and the budget is unlimited. max_batch_size caps how many rows the scheduler coalesces into one dispatch of this model; nothing means uncapped. The cap does not change compiled shapes: a partial fill still pads up to the smallest compiled batch size, and a single request larger than the cap is still served (requests are never split).

source
ReactantServerCore.NamedTensor — Type
NamedTensor(name, dtype, shape, data)
NamedTensor(name, data)

A named host tensor carried across the transport boundary as both an input and an output. It pairs a tensor name with its DType, its shape (Julia column-major dimensions), and the backing data array. The two-argument form derives dtype and shape from a typed host Array.

source
ReactantServerCore.NumericsMode — Type
NumericsMode

How f32 matmul/convolution precision is resolved at model compile time (the runtime.numerics knob). NUMERICS_AUTO (the default) is hardware-adaptive: TF32 is used where the GPU supports it (compute capability >= 8.0) and explicit TF32 algorithms are stripped where it does not, so the same bundle compiles everywhere but its numerics follow the hardware. NUMERICS_F32 pins full f32 everywhere: TF32 DotAlgorithms are rewritten to f32 and every algorithm-free f32 dot_general/convolution gets precision_config = HIGHEST, so numerics are identical across GPU generations (the mode for validated deployments; costs tensor-core throughput on TF32-capable GPUs). NUMERICS_TF32 compiles exactly like auto (TF32 permitted; kernel choice stays with XLA/cuBLAS, and StableHLO cannot force TF32 for convolutions at all) but makes the hardware requirement a guarantee: the worker fails startup on hardware that cannot run TF32, rather than silently degrading per worker in a mixed fleet.

source
ReactantServerCore.PoolAcquireTimeout — Type
PoolAcquireTimeout(span, waited_ns)

Raised by acquire_slot! when a deadline_ns was supplied and passed before span contiguous slots became free. The waiter is dequeued before this is thrown, so it never stalls the line. Callers that carry a request deadline (e.g. a meta model's fan-out) translate this into their own deadline-exceeded error.

source
ReactantServerCore.RequestCancelled — Type
RequestCancelled(model_name, reason)

The request was given up by its client: at dispatch its slot control word no longer read OWNED, or before writing its outputs the server could not claim the slot. The transport maps it to CANCELLED.

source
ReactantServerCore.ResidencyMode — Type
ResidencyMode

Who owns device residency on a worker, fixed at startup. SELF_MANAGED lets the worker autonomously transfer and evict weights above each model's floor within the device budget; EXTERNALLY_MANAGED makes a control plane authoritative (no autonomous eviction, non-resident models are not served until pinned). This is no longer configured directly: it is derived from ModelControlMode (explicit ⇒ EXTERNALLY_MANAGED, otherwise SELF_MANAGED).

source
ReactantServerCore.ResidencyState — Type
ResidencyState

The residency floor an operator (self-managed) or control plane (externally-managed) sets for a model's weights. UNPINNED keeps no guaranteed residency (loaded from the mmap on demand); PINNED_SYSTEM guarantees the weights stay resident in host RAM (and must be transferred to the device before execution); PINNED_DEVICE guarantees them resident on the GPU for the server's lifetime (exempt from eviction).

source
ReactantServerCore.RuntimeConfig — Type
RuntimeConfig

Runtime and device settings (the runtime: config block). backend selects CPU or CUDA execution; device_ordinal picks the GPU among several visible ones; device_memory_fraction is the fraction of device memory claimed for the pool; preallocate claims that pool up front; allow_cpu_fallback permits falling back to CPU when the device is unavailable; device_weight_fraction is the share of the BFC arena (device_memory_fraction * device memory) that holds resident model weights, device-pinned and on-demand together, in (0, 1] (1.0, the default, uses the whole arena minus measured scratch and headroom; resolved to a byte budget at startup, GPU only; models that do not fit in it are swapped on and off the device on demand); device_headroom_fraction is the fraction of the arena kept free as anti-fragmentation slack and feeds the startup auto-sizing (the worker probes each model's peak device usage and sizes the cache so pinned weights + on-demand + scratch + this slack fit); residency_mode selects self-managed or externally-managed residency; shared_host_weights opts the node into the shared-memory host-weight store so same-node workers share one host copy of each system-pinned model; and shared_host_weights_mode sets the permission bits (an octal string, default "666") for those shared regions and their lock files. The "666" default keeps cross-UID container setups working but is world-writable; "660" is recommended for production and multi-user systems. autotune (default true) enables XLA's GPU compile autotuner; set it false to compile with xla_gpu_autotune_level=0 (default gemm/conv algorithm selection, no timing trials), which removes autotuning's run-to-run non-determinism and its device scratch that otherwise inflates the startup memory probe (_probe_max_scratch!) on the first, un-cached start. autotune_cache (default nothing, meaning inherit Reactant's LocalPreferences.toml) toggles the persistent per-fusion autotune cache, and autotune_cache_dir (default "", inherit) sets its directory; both are applied to Reactant's compile cache at worker startup, so a container can drive them by env. numerics (default auto) sets the f32 matmul/convolution precision policy; see NumericsMode. executable_cache (default true) stores each compiled program under the bundle's .cache/ directory and loads it on later starts instead of recompiling, keyed by the Reactantjll build, the device, and the MLIR content (a changed model*.mlir invalidates its programs; a weights-only update does not); it needs a Reactant that exposes executable serialization and is otherwise a no-op. profile (default default) applies a named set of defaults; see RuntimeProfile. `batchsizes(defaultall, or the profile's) selects which compiled batch sizes are loaded; see [BatchSizeMode](@ref).xlaflags(default empty, plus the profile's) is a mapping of XLADebugOptionsfield names to values, passed to every compile, for examplexlagpuexcludenondeterministicops: true; names and value types are checked against the linked XLA at startup, and the flags are part of the executable cache key.weightsmigration(defaultoff) converts legacy-order weights files at load; see [WeightsMigration](@ref).hostweightbudgetcaps the host RAM held by weight copies kept between GPU loads, in bytes (see [parsebytesize](@ref) for the accepted spellings). The default0keeps no copies;nothing(unlimitedin YAML) keeps every model's copy with no cap. Withsharedhostweightsthe budget is one node-wide pool shared by every worker; without it each worker getsbudget ÷ hostweightbudget_workers` (default 1, which the node supervisor sets to its worker count).

source
ReactantServerCore.RuntimeProfile — Type
RuntimeProfile

A named set of runtime defaults (the runtime.profile knob). PROFILE_DEFAULT changes nothing. PROFILE_REGULATED is for validated deployments that need reproducible results: it defaults batch_sizes to largest, adds xla_gpu_exclude_nondeterministic_ops: true to xla_flags, and turns scheduler.coalescing off. Anything set explicitly in the config wins over the profile, so each piece can be turned back off. The profile does not change numerics; instead the worker logs a prominent warning at startup naming the f32 matmul/convolution precision actually in effect unless numerics is f32.

source
ReactantServerCore.SchedulerConfig — Type
SchedulerConfig

Global scheduler settings (the scheduler: config block). ema_halflife_seconds is the half-life of the per-model recent-compute moving average the worker reports, which no discipline schedules by; recency_penalty_cap and coalescing_discount are accepted and validated for compatibility but have no effect, since the fair discipline charges measured device time rather than ranking by a clamped recency deficit over a discounted cost; cost_ema_alpha is the weight a new measurement gets in the learned cost of each compiled program (shape variant and batch size) when it follows the previous one closely (an older estimate gives way faster); max_queue_depth caps each model's queue independently (a full model rejects new requests without affecting admission for the others); dispatch_timeout_seconds is the per-request execution timeout, how long a request may wait for its result from submission (queueing and execution together) before it is withdrawn and fails with DEADLINE_EXCEEDED (0 disables it); discipline selects the inter-model ordering (see SchedulingDiscipline); compaction_interval runs worker-local memory compaction every N on-demand weight-cache loads (0 disables), the standalone (no-gateway) trigger, off by default so a gateway-fronted worker never self-compacts; and models holds the per-model ModelSchedConfig overrides. coalescing (default true, false under the regulated profile) lets the scheduler merge several queued requests to one model into a single dispatch; when it is false every dispatch serves exactly one request, padded to its compiled batch size, so a request's rows always occupy the same batch slots whatever else is queued. weight_read_stall_seconds (default 20, 0 disables it) is how long a background read of a model's weights into host memory may go without progress before the worker abandons it: the model's queued requests fail with UNAVAILABLE, and its read slot passes to the next model. Progress is bytes arriving from the weights file, so storage that is slow but moving is never cut off; a wait for another worker filling the same shared host-weight copy counts that worker's progress, and any other wait on a lock counts as none.

source
ReactantServerCore.SchedulingDiscipline — Type
SchedulingDiscipline

The inter-model dispatch ordering. FAIR shares the device by per-model weight, charging each dispatch the device time it measured (start-time fair queueing); FIFO serves in global arrival order; EDF (earliest-deadline-first) serves the model whose most-urgent queued request has the soonest deadline. Coalescing runs underneath all three.

Guidance: FAIR is for deployments where models share this worker with no upstream placement authority, a single-GPU worker or a multi-GPU fleet behind the round-robin gateway, where the worker itself must stop one model from crowding out the rest. Under the gateway's lpt_packing scheduling the gateway is the fairness authority (placement concentration plus the per-worker share cap), and workers must run FIFO or EDF so the two do not fight; lpt_packing supersedes the role FAIR played on manually-assigned multi-GPU fleets.

EDF is for deadline-sensitive deployments where requests carry a remaining-budget timeout (see the request-level timeout parameter). It degrades to FIFO whenever queued requests share the same deadline, so its only divergence from FIFO is to promote requests with less budget left, in practice the in-flight meta-model sub-calls that have already spent part of their budget on an earlier stage. It also sheds work that cannot finish within its learned compute cost (laxity), so it trades some throughput (batch fragmentation, and no per-model fairness) for hitting more deadlines under load. NOTE: EDF derives urgency purely from the deadline, so issuing different per-client deadlines for the same model reorders that model's service and therefore affects fairness across clients; uniform deadlines keep it behaving like FIFO.

source
ReactantServerCore.ServerConfig — Type
ServerConfig

The fully resolved configuration for a single worker process, frozen for the process lifetime. It is produced from a node file (see node.jl) with environment-variable overrides applied, then checked by validate_config. Fields: model_dirs (bundle search paths), cache_dir, the RuntimeConfig, SchedulerConfig, and EndpointsConfig sub-configs, models_include (an allowlist of model names to load; empty means load all), model_poll_seconds (the dynamic-mode interval at which the worker re-scans its model_dirs for added, changed, or removed bundles and hot-swaps them), model_control_mode (see ModelControlMode: static, dynamic, or explicit), and the GrpcConfig grpc sub-config (gRPC message-size limits). ReactantServer.serve also accepts a ServerConfig directly.

source
ReactantServerCore.SharedMemoryRegistry — Type
SharedMemoryRegistry(; reap_interval = SHM_REAP_INTERVAL_SECONDS,
                       reserved_prefixes = [SHARED_WEIGHT_PREFIX])

The registry of client shared-memory regions. Once something is registered, a timer re-checks every reap_interval seconds that each registration's object still exists under its key and drops those that do not (see shm_reap!); shm_teardown! stops it. A non-positive interval disables the timer (registrations still sweep).

A key named <prefix>-... for any of reserved_prefixes is never registered (WRK-9): by default that covers the regions and lock files of the node's shared host-weight store (SharedWeightStore, /dev/shm/rsw-...), which a client could otherwise register and point an output at, writing into the weights every worker on the node serves.

source
ReactantServerCore.SharedStoreFull — Type
SharedStoreFull

Raised by materialize_host_weights! on a SharedWeightStore when the node cannot back a new region: /dev/shm is full or the memory limit refuses its pages. Nothing is left behind (the partial region is unlinked); the caller can fall back to a private copy.

source
ReactantServerCore.SharedWeightStore — Type
SharedWeightStore(; mode=0o666, budget=nothing, prefix=SHARED_WEIGHT_PREFIX)

Opt-in store backing each model's host weights with a node-level POSIX shared-memory region so same-node workers share one copy. See the file header for the flock-coordinated protocol. mode sets the permission bits for the regions and their lock files. The default 0o666 lets workers running as unrelated UIDs (for example different containers) share the regions, but is world-writable; 0o660 is recommended for production and multi-user systems.

budget (bytes) caps the regions in /dev/shm across every worker on the node; see "Node budget" in the file header. nothing (the default) leaves them unbudgeted. prefix names the regions and lock files (/dev/shm/<prefix>-...); stores with different prefixes share neither regions nor a budget. Every worker on a node must use the same one, which the server does. A SharedMemoryRegistry serving clients on the same node should refuse registrations under it (its reserved_prefixes), which it does by default for SHARED_WEIGHT_PREFIX.

source
ReactantServerCore.ShmRegionExistsError — Type
ShmRegionExistsError(name, key)

Raised by shm_register! when name is already registered for a different shared-memory object that still exists (key is the key it is registered with). The transport maps it to ALREADY_EXISTS, as Triton does; nothing about the existing registration changes.

source
ReactantServerCore.ShmRegionShrunkError — Type
ShmRegionShrunkError(name, key, size, needed)

Raised before a copy into or out of the registered region name (or a touch of a slot control word in it) when its shared-memory object key is now size bytes, shorter than the needed bytes the access reaches: the client shrank the object after registering it, and touching the missing pages would kill the worker with SIGBUS (CORE-4). The transport maps it to FAILED_PRECONDITION, the code of a registration that no longer holds; registering the object again is refused while it is too small for the window, so a client that re-registers on that code cannot loop.

source
ReactantServerCore.ShmSpaceError — Type
ShmSpaceError(name, bytes, errno)

The pages of bytes bytes of the shared-memory object or region name could not be allocated: /dev/shm is full (ENOSPC) or the memory limit refused them (ENOMEM). Raised where touching an unallocated page would otherwise kill the process with SIGBUS: when a client's BufferPool object is created (nothing is left behind), and before a server copies into or out of a registered region (the transport maps it to RESOURCE_EXHAUSTED).

source
ReactantServerCore.SlotControl — Type
SlotControl(mapping, offset, token)

A request's slot control as decoded: the registration its control word lives in (bound at decode, like an output target), the word's offset within that registration's window, and the attempt's token.

source
ReactantServerCore.WeightsFileId — Type
WeightsFileId

Which on-disk version of a weights file a process read: its device, inode, size and mtime. Take it from the open file (weights_file_id(stat(io))) at the moment the file is mapped, so it names the bytes actually mapped even if the path is replaced afterwards. NO_WEIGHTS_FILE stands for no file (hand-built entries).

source
ReactantServerCore.WeightsMigration — Type
WeightsMigration

What the worker does with a bundle whose weights.safetensors is in the legacy axis order (the runtime.weights_migration knob). MIGRATE_OFF (the default) loads it as it is, permuting every tensor on each load. MIGRATE_COPY writes a verified copy in the program's axis order, weights.T.safetensors, next to it and points the manifest's weights_file at the copy; the original is untouched and servers that predate the copy keep loading it. MIGRATE_REPLACE does the same, then, after proving the copy again, renames it over weights.safetensors and marks the manifest format_version: "2.1". Use replace only once every server reading the directory understands that version. Migration never fails a model: on any error the bundle loads as it was.

source
ReactantServerCore.acquire_slot! — Function
acquire_slot!(pool, span=1; deadline_ns=0) -> PoolSlot

Block until span physically contiguous slots are free, then return them as one slot whose [offset, offset + span*slot_bytes) range no other in-flight slot overlaps. Throws an ArgumentError immediately if span exceeds the pool's total slot count, since such a request could never be satisfied and would otherwise deadlock. Waiters are served in FIFO order. Pair with release_slot!, which frees the whole run.

deadline_ns is an absolute time_ns() deadline (0 = wait indefinitely, the default and prior behavior). When set, a waiter that has not acquired by the deadline throws [PoolAcquireTimeout] instead of parking past it. A one-shot timer wakes the waiter at the deadline even if no slot is released in the meantime, so a starved waiter fails fast rather than burning a request's whole budget in the park.

source
ReactantServerCore.bump_load_rank — Function
bump_load_rank(rank, now=time(); halflife=HOST_CACHE_HALFLIFE_SECONDS) -> Float64

Record one load in a host copy's load rank and return the new rank.

The rank holds a decayed load count, count(t) = 2^((rank - t) / halflife), as a single timestamp. A copy is worth keeping in host RAM in proportion to how often a device loads it (each load it serves saves a request the wait for a read from disk) times the read it saves per byte, and with one disk the read time grows with the bytes, so the worth comes down to the load rate. Ranks order copies by their counts at any shared now, so comparing ranks needs no clock, and a rank fits wherever a timestamp does (the shared store keeps it as the region lock file's mtime). A rank of 0 means no loads. With a vanishing halflife this is least recently loaded; a model loaded often ranks ahead of one loaded once a moment ago, and one a device keeps resident (no loads) decays out.

source
ReactantServerCore.cached_region_exists — Method
cached_region_exists(store, key, digest) -> Bool

Whether the node cache holds a region for key at digest, ready or being filled. A snapshot: any worker can evict a holder-free region the next moment, so this estimates what a reload will find rather than guaranteeing it.

source
ReactantServerCore.deadline_params — Method
deadline_params(budget_ns) -> Dict{String,InferParameter}
deadline_params(PB::Module, budget_ns) -> Dict{String,InferParameter}

Build the request-level parameters map carrying a remaining-budget timeout of budget_ns nanoseconds (see TIMEOUT_NS_PARAM). A non-positive budget_ns yields an empty map (no deadline). Merge the result into a ModelInferRequest's parameters. The PB-first form builds the map from pb module PB (consumer packages pass their own generated module; the single-argument form uses this package's inference module).

source
ReactantServerCore.decode_infer_request — Function
decode_infer_request(msg, registry=nothing) -> DecodedRequest
decode_infer_request(PB::Module, msg, registry=nothing) -> DecodedRequest

Translate a decoded ModelInferRequest message into the boundary InferRequest. The transport (gRPC) hands us the already-decoded protobuf message, so the codec never touches wire bytes. Input tensor data is read from a registered shared-memory region (preferred when the tensor declares one), otherwise from rawinputcontents, otherwise from the typed contents field. The PB-first form accepts a message from pb module PB (consumer packages pass their own generated module; the shorter form uses this package's inference module).

source
ReactantServerCore.decode_infer_response — Method
decode_infer_response(msg) -> Vector{NamedTensor}
decode_infer_response(PB::Module, msg) -> Vector{NamedTensor}

Translate a ModelInferResponse into boundary NamedTensor outputs. Data is read from rawoutputcontents when present, otherwise from the typed contents field. Shared-memory-backed outputs are not supported on this path (the caller never requests them). The PB-first form accepts a message from pb module PB (consumer packages pass their own generated module).

source
ReactantServerCore.encode_infer_request — Method
encode_infer_request(model_name, inputs; requested_outputs=String[], id="") -> ModelInferRequest
encode_infer_request(PB::Module, model_name, inputs; requested_outputs=String[], id="") -> ModelInferRequest

Build a ModelInferRequest from boundary NamedTensor inputs, with tensor data inline in rawinputcontents. requested_outputs, when non-empty, names the outputs to return. The PB-first form builds the message from pb module PB (consumer packages pass their own generated module).

source
ReactantServerCore.encode_infer_request_shm — Method
encode_infer_request_shm(model_name, inputs, region, offsets; requested_outputs, id)
encode_infer_request_shm(PB::Module, model_name, inputs, region, offsets; requested_outputs, id)

Encode a request whose inputs are ALL staged in shared-memory region at the given byte offsets (parallel to inputs); no raw_input_contents (the receiver reads each tensor via shm_read). The receiver must have region registered. This is all-or-nothing per request: the decode path treats raw_input_contents as parallel-to-inputs, so a request never mixes raw and SHM inputs. The PB-first form builds the message from pb module PB.

source
ReactantServerCore.encode_infer_response — Method
encode_infer_response(model_name, id, outputs) -> ModelInferResponse
encode_infer_response(PB::Module, model_name, id, outputs) -> ModelInferResponse

Build the response message with outputs entirely inline (rawoutputcontents). The transport serializes the returned message. The PB-first form builds the message from pb module PB (consumer packages pass their own generated module; the shorter form uses this package's inference module).

source
ReactantServerCore.encode_infer_response — Method
encode_infer_response(model_name, decoded, outputs, registry) -> ModelInferResponse
encode_infer_response(PB::Module, model_name, decoded, outputs, registry) -> ModelInferResponse

Build the response message, writing any output whose requested entry named a shared-memory region into that region (and referencing it in the response) instead of inline. rawoutputcontents holds one entry per output in outputs order, as the KServe proto requires: the bytes of an inline output, or an empty placeholder for a shared-memory output, so a client that indexes it by output position (as Triton clients do) reads the right tensor. The PB-first form builds the message from pb module PB (consumer packages pass their own generated module).

source
ReactantServerCore.encode_model_description — Method
encode_model_description(name, manifest; max_batch_size = 0) -> ModelDescription
encode_model_description(PB::Module, name, manifest; max_batch_size = 0) -> ModelDescription

Build the control plane's ModelDescription for a served model from its manifest: the client-facing tensors with the kind of every axis, the compiled batch sizes and input-shape variants, and a meta model's sub-models. max_batch_size is the worker's effective coalescing limit, which the manifest alone does not know. The PB-first form builds the message from pb module PB.

source
ReactantServerCore.encode_model_metadata — Method
encode_model_metadata(name, manifest, platform) -> ModelMetadataResponse
encode_model_metadata(PB::Module, name, manifest, platform) -> ModelMetadataResponse

Build a ModelMetadataResponse message from the manifest's client-facing I/O spec. The PB-first form builds the message from pb module PB (consumer packages pass their own generated module).

source
ReactantServerCore.encode_repository_index — Method
encode_repository_index(names) -> RepositoryIndexResponse
encode_repository_index(entries::AbstractVector{<:Pair}) -> RepositoryIndexResponse
encode_repository_index(PB::Module, names) -> RepositoryIndexResponse
encode_repository_index(PB::Module, entries::AbstractVector{<:Pair}) -> RepositoryIndexResponse

Build a RepositoryIndexResponse. The first form lists every model as READY (direct-client introspection). The second takes name => ready::Bool pairs and reports READY or UNAVAILABLE per model, so the gateway can discover which replicas actually serve a model (readiness reflects residency on the worker). The PB-first forms build the message from pb module PB.

source
ReactantServerCore.error_request_id — Function
error_request_id(id="") -> String

The request id an INTERNAL error is reported and logged under: the client's own KServe request id when it sent one (sanitized and trimmed, at most 64 characters), otherwise 8 random hex digits.

source
ReactantServerCore.load_manifest — Method
load_manifest(path) -> Manifest

Parse a manifest YAML file at path into a Manifest. This runs the structural parsing and validation of parse_manifest but not the bundle-directory checks in validate_manifest, so it is usable wherever only the manifest's contents are needed, for example a client deriving a model's I/O spec offline.

source
ReactantServerCore.materialize_host_weights! — Method
materialize_host_weights!(store, key, digest, specs, fill!) -> Vector{Any}

Return a model's host weight Arrays (in weight order), populating them via fill!(arrays) when this worker is the one that must materialize them. specs is a vector of (eltype, dims) per tensor. For PrivateWeightStore the arrays are freshly allocated; for SharedWeightStore they alias a node-shared region.

source
ReactantServerCore.materialize_node! — Method
materialize_node!(node, devices; cpu_workers=1) -> Vector{Union{String,Nothing}}

Prepare a raw node config for supervised single-container deployment: assign one visible device per worker and return the per-worker device selectors (in worker declaration order; nothing means the worker gets no CUDA_VISIBLE_DEVICES of its own, the CPU case).

With no workers: list, one is synthesized: worker0..workerN-1, one per device (or cpu_workers workers when devices is empty). An explicit workers: list wins: each worker is assigned devices[i] positionally, or devices[gpu+1] when the entry carries a gpu: key. The gpu: key is consumed here (deleted after assignment) so each child process, seeing a single device through CUDA_VISIBLE_DEVICES, resolves device ordinal 0. Assigning one device to two workers, or having more workers than devices, is a ConfigError. Call validate_node on the result before use.

source
ReactantServerCore.monotonic_time — Method
monotonic_time() -> Float64

Seconds on the host's monotonic clock: time_ns() (libuv's uv_hrtime, CLOCK_MONOTONIC on Linux) as a Float64. Use it instead of time() for anything taken as a difference (an elapsed time, a timeout, a deadline, an age), since a correction of the wall clock never moves it.

Its zero is arbitrary (on Linux, the host's boot), so a value means something only next to another monotonic_time() from the same host. It is the same clock in every process and container sharing the host's kernel, but never compare one with time() or a file's timestamps, show it as a date, persist it, or send it to another host.

source
ReactantServerCore.node_gpus — Method
node_gpus(node) -> :auto | Int | Vector{String}

Parse the optional top-level gpus: key: auto (the default when absent) asks the supervisor to enumerate visible devices; an integer is a device count (expanded to ordinals 0..n-1); a list gives explicit device selectors (ordinals or GPU UUIDs, passed to CUDA_VISIBLE_DEVICES verbatim).

source
ReactantServerCore.node_server_config — Method
node_server_config(node, worker) -> (ServerConfig, applied_overrides, worker_name)

Resolve the ServerConfig for one worker of a node. worker may be nothing when the node has exactly one worker (it defaults to that sole entry); otherwise it must name a defined worker. Environment overrides (INFERENCE_SERVER_*) are applied on top, as for any server config. Does not validate; call validate_config on the result.

source
ReactantServerCore.parse_byte_size — Method
parse_byte_size(s) -> Int

Parse a byte count such as 0, 0b, 512m, 5g or 1.5TiB: a non-negative number with an optional unit, case insensitive. Every unit is binary (5g and 5GB are both 5 GiB, as in Docker), so no two spellings disagree. An integer is taken as bytes.

source
ReactantServerCore.pool_region_bytes — Method
pool_region_bytes(pool) -> Int

The whole length of a SHM pool's object: the data slots and then the slot control area. This, not sizeof(pool) (the data bytes), is the window a client registers, so a server can reach the control words. Equal to sizeof(pool) for an inline pool.

source
ReactantServerCore.release_host_weights! — Method
release_host_weights!(store, key) -> nothing

Release a model's host weights. For SharedWeightStore this detaches the region and, if this was the last holder on the node (a non-blocking upgrade to an exclusive flock succeeds), unlinks the region and its lock file. A no-op for the private store. The caller must drop all references to the arrays first.

source
ReactantServerCore.release_slot! — Method
release_slot!(slot)

Return a slot acquired with acquire_slot! to the pool's free set, freeing every physical slot in its span. Releasing a derived subslot (index == 0) is a no-op. Throws if any slot in the span is already free (double release).

A slot armed for the slot control protocol (slot_arm!) first has its control word moved out of OWNED, so no server can begin writing it any more. If a server is writing it at that moment (WRITING), the run is quarantined instead of freed: it rejoins the free set once the writer is done, or once the writer is found dead after SLOT_RETIRE_SECONDS; a live writer stuck past that keeps it retired until it finishes. Nothing ever waits here.

source
ReactantServerCore.request_cancelled — Method
request_cancelled(req::InferRequest) -> Bool

Whether req's client has given up on it, as its shared-memory slot control word shows (CORE-3): the predicate the scheduler checks before starting a request's work. Never waits; false for a request without slot control.

source
ReactantServerCore.request_drain! — Method
request_drain!(seconds) -> Bool

Ask every blocking serve / serve_gateway in this process to drain: stop taking new requests, let the admitted ones finish for at most seconds, then shut down and return. The first request wins and later ones are ignored. Returns whether a blocking server is serving and will act on it; a request made while none is (for instance during startup) is still recorded and is acted on as soon as one begins serving.

source
ReactantServerCore.same_ipc_namespace — Method
same_ipc_namespace(name) -> Bool

Return whether the POSIX shared-memory object name is visible in this process's IPC namespace. The client passes the name of an object it created; we answer true only if we can open that same object ourselves, which is what determines whether system shared-memory transport can work between the two processes. The open is read-only and detached immediately; nothing is registered or kept mapped. Any failure (object absent because we are in a different namespace, permission error, malformed name) is reported as false.

source
ReactantServerCore.sanitize_status_message — Method
sanitize_status_message(msg) -> String

msg made safe to send as a gRPC status message: grpc-status: and grpc-message: (in any case) become grpc-status= and grpc-message=, so no client can read them as its call's status; control characters become spaces, since a header value is one line; and bytes that are not valid UTF-8 become U+FFFD. Every status message the worker and the gateway send passes through it.

source
ReactantServerCore.scratch — Method
scratch(slot, dims, T) -> Array{T}
scratch(slot, [dims1 => T1, dims2 => T2, ...]) -> Vector{Array}

Carve one typed buffer per dims => T spec from slot in a single call, advancing the slot's cursor so the buffers occupy disjoint, contiguous byte ranges. Each buffer is an Array{T} aliasing the pool's backing (via pool_view; zero-copy, and uniform across SHM- and Memory-backed pools since it is just pool_base + offset); write into the returned arrays directly. dims is a shape tuple (or a bare integer for a vector).

This is the buffer-request interface shared by the meta-model call.scratch and the client driver: ask for ALL buffers up front in one call. The carved buffers' lifetime is bounded by slot (and by the pool, which the caller keeps alive); they become invalid once it is released.

Every buffer starts at an offset aligned for its element type. Within one call the buffers are laid out by element alignment, largest first (the returned order is the requested one): each buffer's size is a multiple of its element size, so that order leaves no padding between them and the call uses exactly the sum of the buffer sizes, which is what a caller sizes the slot for.

source
ReactantServerCore.shm_read — Method
shm_read(registry, name, offset, byte_size) -> Vector{UInt8}
shm_read(alloc, registry, name, offset, byte_size) -> Array

Copy a range of the named region into a fresh byte vector, or into an array the caller allocates. The range is [offset, offset+byte_size); the array alloc(byte_size) returns must hold exactly byte_size bytes of a bits type (a decoded tensor is read straight into its final array this way, with no intermediate copy). The copy is done while the region's lock is held, so a concurrent shm_unregister! of the same region cannot detach the mapping underneath the read. The object must still cover the range (ShmRegionShrunkError otherwise), and its pages are allocated first (ShmSpaceError when /dev/shm has no room), so the copy cannot raise SIGBUS. The destination is allocated only after the range has passed both checks.

source
ReactantServerCore.shm_reap! — Method
shm_reap!(registry) -> Vector{String}

Drop every registration whose key no longer resolves to the object the worker mapped (the client unlinked it, or unlinked it and created another under the same key), and return the names dropped. A normal client exit unlinks its objects, so this is what eventually releases their mappings; an object a crashed client left linked cannot be told from a live one and stays registered. Runs at every registration and on the registry's timer.

source
ReactantServerCore.shm_register! — Method
shm_register!(registry, name, key, offset, byte_size)

Attach the existing POSIX shared-memory object key read-write and register it under name. The worker opens key, records the (device, inode) of the object through the descriptor it then maps, and keeps that descriptor. When name is already registered:

  • the same object (equal device and inode): idempotent success; the registration is kept, or replaced by a fresh mapping of the same object when offset or byte_size changed;
  • the registered key no longer resolves to the registered object (it was unlinked, or unlinked and created again): the registration is stale and is replaced;
  • a different object that still exists: ShmRegionExistsError (ALREADY_EXISTS), and the existing registration is untouched.

A key under one of the registry's reserved prefixes (by default the shared host-weight store's) is refused with an ArgumentError (INVALIDARGUMENT) before it is opened. Every registration also sweeps the registry for stale entries ([`shmreap!`](@ref)).

source
ReactantServerCore.shm_unregister! — Method
shm_unregister!(registry, name)

Unregister and detach a region, or all regions when name is empty. Idempotent: unregistering a name that is not registered is a successful no-op (it matches KServe semantics and lets the gateway fan-out and its rollback unregister succeed without a spurious error). Registration, by contrast, fails loudly (see shm_register!) so a bad region surfaces at register time rather than during inference.

source
ReactantServerCore.shm_write! — Method
shm_write!(registry, name, offset, bytes)

Copy bytes into [offset, offset+length(bytes)) of the named region. The copy is done while the region's lock is held, so a concurrent shm_unregister! of the same region cannot detach the mapping underneath the write. As for shm_read, the object must still cover the range and its pages are allocated first.

source
ReactantServerCore.slot_arm! — Method
slot_arm!(slot) -> UInt64

Start an attempt on slot (a top-level slot of a pool with has_slot_control): move its control word to OWNED(t) under a fresh token t, record t on the slot, and return it, for the request to carry. Arming again for a retry first closes the previous attempt, so its server can no longer claim it; if that server is already writing, throws SlotWriteInProgress and the slot keeps its previous attempt (release it as usual).

source
ReactantServerCore.slot_control_offset — Method
slot_control_offset(slot) -> Int

Byte offset, in the pool's object (and so in a window registered at offset 0), of the control word of slot's run (the word of its first slot).

source
ReactantServerCore.slot_seal! — Method
slot_seal!(slot) -> Bool

After a successful response, decide whether the outputs in slot are final and may be read: true when the server finished writing them under the protocol (DONE(t)), or when no server claimed the attempt (a server that does not run the protocol wrote them before replying), in which case the word is closed first so nothing can begin writing during the read. false means a writer claimed the attempt and is still copying, so the bytes cannot be trusted. Always true for an unarmed slot.

source
ReactantServerCore.subslot — Method
subslot(parent, n_bytes; align = 1) -> PoolSlot

Carve a child slot of n_bytes from the next cursor position of parent, first moving the cursor forward to a multiple of align bytes from the pool's base (a power of two, at most 64). The parent's cursor advances; the child starts with a fresh cursor at zero and index 0 (not separately releasable, its lifetime is bounded by the parent's). A typed view of the child (pool_view(child, T, ...)) needs align to be at least T's alignment; scratch carves that way.

source
ReactantServerCore.trim_host_cache! — Method
trim_host_cache!(store) -> Int

Evict holder-free regions until the node fits the store's budget (a lowered budget, or regions an earlier run left behind). Returns the number evicted; a no-op without a budget.

source
ReactantServerCore.try_acquire_slot! — Function
try_acquire_slot!(pool, span=1) -> Union{PoolSlot, Nothing}

Like acquire_slot! without waiting: the slot when span contiguous slots are free right now, otherwise nothing. It never passes a caller parked in acquire_slot!, so it returns nothing while anyone waits. Throws PoolClosedError on a closed pool and ArgumentError for a span the pool can never satisfy.

source
ReactantServerCore.validate_node — Method
validate_node(node)

Structural validation of a parsed node config. Raises ConfigError on a malformed file: missing model_repo, duplicate worker names, colliding ports, or a models: entry that targets an undefined worker. The models: map is optional for any node: when omitted, every worker loads every bundle in the repo; when present, it is a per-model override that pins the named models to device memory on the listed workers (see worker_raw_config).

source
ReactantServerCore.weights_digest — Method
weights_digest(key, specs; content=0) -> UInt64

Digest over a model's identity, weight layout (name, per-tensor dtype/size/shape, a format version), and a content token identifying the weight file's on-disk version (see weights_file_token). Two workers computing this for the same model and the same file agree on the same region key. Without the content token a weights-only update (same name, same tensor layout) would collide with the previous version's region and silently serve stale weights, including across server restarts (regions in /dev/shm outlive the process by design).

source
ReactantServerCore.weights_file_token — Method
weights_file_token(id_or_path) -> UInt64

Identity token for one on-disk version of a weights file, folded into weights_digest via its content keyword. Derived from the device, inode, size and mtime: cheap (no content read, so the shared store's zero-copy attach path stays zero-read) and identical across same-node workers that mapped the same file. The inode makes an atomic replacement (write a new file, rename it over the old one) a new version even when size and mtime match. Pass the WeightsFileId taken when the file was mapped; a path is stat'ed now, which can name a newer file than the one mapped. Returns 0 for no file (hand-built test entries), which reproduces the layout-only digest.

source
ReactantServerCore.wire_batch_spec — Method
wire_batch_spec(m::Manifest) -> Union{Nothing,Tuple{String,Int}}

The name and 1-based batch axis of the first WIRE-FACING input that declares one, or nothing when the model declares no batch axis at all (every request is then one item).

"Wire-facing" is client_inputs when the bundle has them and executable_inputs otherwise: client_inputs is present exactly when a model.jl intercepts the request (see validate_manifest), and without one the client sends the executable inputs directly.

This exists so a gateway can count the items in a request it never fully decodes. It cannot infer the axis: the position is per tensor and genuinely varies across bundles (whcn puts it last, nc first), the first input need not carry one at all (a cross-encoder's query is unbatched while its keys are batched), and it has moved between exports of the same model. Picking the first input that declares an axis mirrors _derive_input_batch_dim, and the name lets the reader match it against the request's own name-addressed tensors rather than trusting their order.

source
ReactantServerCore.with_cached_host_weights — Method
with_cached_host_weights(f, store, key, digest, specs, fill!) -> Union{Tuple{Bool, Any}, Nothing}

Run f(arrays) on a model's host weights from the node cache. The store must have a budget. The region is held only while f runs (an upload) and then detached without unlinking, so it stays cached for the next load by any worker. A missing region is admitted (evicting least recently loaded ones) and filled via fill!(arrays) first. Returns (hit, f(arrays)), where hit says the region already existed, or nothing when the region cannot fit the budget; the caller then loads without the cache.

source
ReactantServerCore.worker_raw_config — Method
worker_raw_config(node, name) -> Dict{String,Any}

Resolve a single worker's raw config dict from a (validated) node config: deep-merge global with the named worker's override blocks, then set model_dirs to the shared repo, the endpoint port from base_port, the runtime device ordinal from the worker's GPU, and the node-level shared_host_weights flag. Every worker loads every bundle in the repo; the top-level models: map is an optional per-model override that pins the named models to device memory on the listed workers (translated here into scheduler.models.<name>.residency: device). The result has the shape build_config consumes.

source