Gateway API

Every exported name in ReactantServerGateway that carries a docstring: the multi-GPU gateway: serve_gateway, its scheduling modes and its control plane. See the API overview for the other packages and the unexported internals the guides refer to.

ReactantServerGateway.probe_worker_ready — Function
probe_worker_ready(node_path, worker=nothing) -> Bool

Resolve a worker's port from the node file and call its KServe ServerReady on localhost, returning whether it reported ready. Used as the worker container's healthcheck (a Julia replacement for the former Go reactant-healthprobe). worker may be omitted when the node has a single worker.

source
ReactantServerGateway.serve_gateway — Function
serve_gateway(gateway_path=nothing; blocking=true) -> nothing | RunningGateway

Load gateway.yml (listen addresses and the worker endpoint list), build the worker client pool, start the admin HTTP server and the readiness/discovery prober (which probes each endpoint's ServerReady and RepositoryIndex and swaps in the discovered routing table), and serve the KServe gRPC proxy. When blocking is false the server runs in the background and a RunningGateway is returned (stop it with stop!). A blocking gateway serves until a drain is requested (request_drain!, or under the node supervisor end of file on stdin), then drains, shuts down and returns.

gateway_path may be omitted (or nothing) to configure the gateway from defaults and REACTANT_GATEWAY_* environment variables alone; the endpoint list then comes from REACTANT_GATEWAY_WORKERS. The node supervisor uses this to run an embedded gateway without a gateway.yml.

source