> ## Documentation Index > Fetch the complete documentation index at: https://docs.expanse.sh/llms.txt > Use this file to discover all available pages before exploring further. # Inference right-sizer > Tighten serving workload resource requests at submit time so more inference jobs pack per GPU. The inference right-sizer sits on the job submit path of a Kubernetes, SLURM, or Nomad compute. Before a model-serving workload reaches the scheduler, it predicts the resources the workload actually needs with a fast analytical model (no LLM, no network call) and tightens the request to the prediction: CPU, memory, GPU count, and optionally the GPU request itself down to a MIG slice. The scheduler gets tight, correct requirements and packs more serving jobs per accelerator. It is off by default and entirely opt-in per compute. ## What gets right-sized Only workloads the right-sizer positively recognises as model serving. It reads the launch command for **vLLM**, **SGLang**, and **TGI** flags (model, tensor parallel, sequence length, concurrency, dtype), and honours explicit `expanse.sh/*` annotations (`served-model`, `model-params-billions`, `model-precision`, `max-model-len`, `max-num-seqs`, `tensor-parallel-size`, `model-kv-group`, and related keys), which override anything parsed from the command. The memory model assumes grouped-query attention; a model with full multi-head attention should state `model-kv-group: 1` so its larger KV cache is sized correctly. Annotations live in pod annotations on Kubernetes, job/group/task `meta` on Nomad, and the job `--comment` as `;`-separated `key=value` pairs on SLURM. A workload with no recognised framework and no annotation is never touched. ## Safety model In active mode the right-sizer rewrites requests with no human in the loop, so it is built around three guarantees: * **Fail-open, always.** Any timeout, predictor error, low confidence, or unrecognised request forwards the original request untouched. Every decision is bounded by a deadline (default 200ms); on expiry the request passes through. If the right-sizer itself is down, submissions are unaffected: the Kubernetes webhook registers with `failurePolicy: Ignore`, the SLURM plugin fails open on any socket error, and the Nomad proxy forwards the original body on any parse failure. * **Shadow by default.** A new deployment records what it would have rewritten while forwarding every request unchanged. You flip to active deliberately, after reviewing the shadow decisions. * **Tighten, never inflate.** A rewrite only ever reduces a stated request or limit, or fills an unstated one. It never raises a value you set, so it cannot increase cost or topology beyond what you asked for. Active mode also rewrites only at or above a minimum prediction confidence (default `medium`). A request whose memory footprint would be fully guessed is passed through, not shrunk. ## Enabling it The Expanse Helm chart's `rightsizer` block deploys a mutating admission webhook (two replicas by default) that patches recognised serving pods at creation: ```yaml theme={"dark"} rightsizer: enabled: true mode: shadow tls: certManager: issuerRef: kind: ClusterIssuer name: your-issuer ``` cert-manager issues and rotates the webhook serving certificate by default; set `rightsizer.tls.certManager.enabled: false` and provide `tls.existingSecret` plus `tls.caBundle` to bring your own. Scope the webhook to specific namespaces with `rightsizer.webhook.namespaceSelector`; empty applies it everywhere except system namespaces. The SLURM bootstrap playbook installs a sidecar on the controller and a `job_submit.lua` plugin that calls it at submit. Set the flags in your inventory group vars, then run the playbook: ```yaml theme={"dark"} expanse_rightsizer_enabled: true expanse_rightsizer_mode: shadow expanse_rightsizer_download_url: "https://…/expanse-rightsizer" ``` ```bash theme={"dark"} ansible-playbook -i inventories//hosts.ini playbooks/rightsizer.yml ``` The playbook installs the sidecar systemd unit, emits a version-matched `job_submit.lua` from the binary itself, and sets `JobSubmitPlugins=lua` (restarting `slurmctld`). If `slurm.conf` already lists other job\_submit plugins without `lua`, it stops and asks you to merge the list by hand rather than replace it. Spec extraction reads the batch script and the job comment; the job's environment is not sent to the sidecar. Nomad has no native mutating admission, so the right-sizer runs as a stateless reverse proxy in front of the Nomad agent. Install the `expanse-rightsizer.service` systemd unit (shipped under `infrastructure/install/nomad/`) and configure `/etc/expanse/rightsizer.env`: ```bash theme={"dark"} EXPANSE_COMPUTE_TYPE=nomad EXPANSE_RIGHTSIZER_ENABLED=true EXPANSE_RIGHTSIZER_MODE=shadow ``` Then repoint job submission at the proxy, for example `NOMAD_ADDR=http://127.0.0.1:8646`. The proxy right-sizes the job register and plan calls and forwards everything else to the real agent unchanged. It is one in-cluster hop and horizontally scalable. ## Rolling out: shadow first Run in shadow through an initial data-collection window. Every decision is recorded (a structured log record plus a metric increment) while the original request is forwarded unchanged, so you can see exactly what active mode would have done at zero risk. Review the decision log and `expanse_rightsizer_decisions_total`, then flip `mode` to `active` when the would-be rewrites look right for your fleet. An automated shadow-versus-actuals comparison is upcoming; today the review is the decision log. ## MIG packing With `migMultiplexing` enabled on a MIG-partitioned cluster, a single whole-GPU serving request is rewritten to the smallest MIG slice whose usable memory holds the predicted footprint, so several servers share one accelerator. The rewrite applies only to single-GPU workloads. Prerequisites differ per scheduler: * **Kubernetes:** the NVIDIA device plugin must run the `mixed` MIG strategy so slices are schedulable as `nvidia.com/mig-*` resources. The right-sizer keeps its view of the cluster's MIG geometry current by watching GPU-labelled nodes; no static inventory is needed. * **SLURM:** MIG profiles must be declared as typed gres in `gres.conf`, and the same profiles declared in `expanse_rightsizer_mig_inventory`. The sidecar cross-checks the declared inventory against `gres.conf` and drops any profile the file does not declare, so a mismatch only loses a packing opportunity. A request already stated with a GPU type (such as `gres/gpu:a100:4`) is tightened count-only, never rewritten to a MIG profile. * **Nomad:** the right-sizer tightens GPU count, CPU, and memory but never rewrites the device name. MIG device names are fingerprinted by the Nomad device plugin, and a wrong name would make the allocation unplaceable, so slice placement stays with your device plugin configuration. ## Configuration reference On Kubernetes, set these through the Helm `rightsizer` values; on SLURM, through the `expanse_rightsizer_*` playbook vars; on Nomad, in the unit's environment file. | Variable | Default | Meaning | | ----------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------- | | `EXPANSE_COMPUTE_TYPE` | required | `kubernetes`, `slurm`, or `nomad`; selects the front-end | | `EXPANSE_RIGHTSIZER_ENABLED` | `false` | Master enable | | `EXPANSE_RIGHTSIZER_MODE` | `shadow` | `shadow` or `active` | | `EXPANSE_RIGHTSIZER_PREDICTOR` | `stats` | `stats` or `hydragt` | | `EXPANSE_RIGHTSIZER_MIN_CONFIDENCE` | `medium` | Minimum confidence at which active mode rewrites (`high`, `medium`, `low`) | | `EXPANSE_RIGHTSIZER_DEADLINE` | `200ms` | Decision deadline; on expiry the request passes through (the SLURM playbook sets `150ms`) | | `EXPANSE_RIGHTSIZER_MIG_MULTIPLEXING` | `false` | Rewrite a single-GPU request to the smallest fitting MIG slice | | `EXPANSE_RIGHTSIZER_MIG_INVENTORY` | empty | Static MIG inventory for SLURM and Nomad: `modelid:hbmGiB:profile,…;…` | | `EXPANSE_RIGHTSIZER_SLURM_GRES_CONF` | `/etc/slurm/gres.conf` | gres.conf the SLURM sidecar cross-checks the inventory against | | `EXPANSE_RIGHTSIZER_MIG_PROFILE_GEOMETRY` | empty | Override the built-in MIG profile geometry table | | `EXPANSE_RIGHTSIZER_MIG_USABLE_FRACTION` | `0.90` | Fraction of a slice's memory treated as usable by the fit | | `EXPANSE_RIGHTSIZER_METRICS_ADDR` | `:9095` | Prometheus metrics listener | | `EXPANSE_RIGHTSIZER_PROXY_ADDR` | `127.0.0.1:8646` | Nomad proxy listen address | | `EXPANSE_RIGHTSIZER_NOMAD_UPSTREAM` | `http://127.0.0.1:4646` | Real Nomad agent behind the proxy | | `EXPANSE_RIGHTSIZER_SLURM_SOCKET` | `/run/expanse/rightsizer.sock` | Unix socket the SLURM plugin dials | | `EXPANSE_RIGHTSIZER_HYDRAGT_URL` | empty | Self-hosted HydraGT endpoint (egress-guarded) | | `EXPANSE_AIR_GAPPED` | unset | Refuse any non-private predictor endpoint at startup | The `stats` predictor is analytical and runs entirely locally: with it, the right-sizer reaches no network at all. The `hydragt` learned predictor is not yet calibrated; selecting it today leaves the right-sizer inert (every request passes through) until calibration ships. ## Observability Every decision, in both modes, produces: * One structured log record (`event=rightsizer_decision`) carrying the action, mode, confidence, predictor engine, served model, original and proposed resources, the chosen MIG profile, and the decision latency. In shadow mode this record is the would-be right-size. * Prometheus metrics on the metrics listener: * `expanse_rightsizer_decisions_total`, a counter labelled by `action`, `mode`, `confidence`, and `engine`. * `expanse_rightsizer_decision_latency_seconds`, a histogram of per-decision wall-clock latency against the deadline.