Inference right-sizer

The inference right-sizer sits on the job submit path of a Kubernetes, SLURM, or Nomad compute. Before a model-serving workload reaches the scheduler, it predicts the resources the workload actually needs with a fast analytical model (no LLM, no network call) and tightens the request to the prediction: CPU, memory, GPU count, and optionally the GPU request itself down to a MIG slice. The scheduler gets tight, correct requirements and packs more serving jobs per accelerator. It is off by default and entirely opt-in per compute.

What gets right-sized

Only workloads the right-sizer positively recognises as model serving. It reads the launch command for vLLM, SGLang, and TGI flags (model, tensor parallel, sequence length, concurrency, dtype), and honours explicit expanse.sh/* annotations (served-model, model-params-billions, model-precision, max-model-len, max-num-seqs, tensor-parallel-size, model-kv-group, and related keys), which override anything parsed from the command. The memory model assumes grouped-query attention; a model with full multi-head attention should state model-kv-group: 1 so its larger KV cache is sized correctly. Annotations live in pod annotations on Kubernetes, job/group/task meta on Nomad, and the job --comment as ;-separated key=value pairs on SLURM. A workload with no recognised framework and no annotation is never touched.

Safety model

In active mode the right-sizer rewrites requests with no human in the loop, so it is built around three guarantees:

Fail-open, always. Any timeout, predictor error, low confidence, or unrecognised request forwards the original request untouched. Every decision is bounded by a deadline (default 200ms); on expiry the request passes through. If the right-sizer itself is down, submissions are unaffected: the Kubernetes webhook registers with failurePolicy: Ignore, the SLURM plugin fails open on any socket error, and the Nomad proxy forwards the original body on any parse failure.
Shadow by default. A new deployment records what it would have rewritten while forwarding every request unchanged. You flip to active deliberately, after reviewing the shadow decisions.
Tighten, never inflate. A rewrite only ever reduces a stated request or limit, or fills an unstated one. It never raises a value you set, so it cannot increase cost or topology beyond what you asked for.

Active mode also rewrites only at or above a minimum prediction confidence (default medium). A request whose memory footprint would be fully guessed is passed through, not shrunk.

Enabling it

Kubernetes
SLURM
Nomad

The Expanse Helm chart’s rightsizer block deploys a mutating admission webhook (two replicas by default) that patches recognised serving pods at creation:

rightsizer:
  enabled: true
  mode: shadow
  tls:
    certManager:
      issuerRef:
        kind: ClusterIssuer
        name: your-issuer

cert-manager issues and rotates the webhook serving certificate by default; set rightsizer.tls.certManager.enabled: false and provide tls.existingSecret plus tls.caBundle to bring your own. Scope the webhook to specific namespaces with rightsizer.webhook.namespaceSelector; empty applies it everywhere except system namespaces.

The SLURM bootstrap playbook installs a sidecar on the controller and a job_submit.lua plugin that calls it at submit. Set the flags in your inventory group vars, then run the playbook:

expanse_rightsizer_enabled: true
expanse_rightsizer_mode: shadow
expanse_rightsizer_download_url: "https://…/expanse-rightsizer"

ansible-playbook -i inventories/<env>/hosts.ini playbooks/rightsizer.yml

The playbook installs the sidecar systemd unit, emits a version-matched job_submit.lua from the binary itself, and sets JobSubmitPlugins=lua (restarting slurmctld). If slurm.conf already lists other job_submit plugins without lua, it stops and asks you to merge the list by hand rather than replace it. Spec extraction reads the batch script and the job comment; the job’s environment is not sent to the sidecar.

Nomad has no native mutating admission, so the right-sizer runs as a stateless reverse proxy in front of the Nomad agent. Install the expanse-rightsizer.service systemd unit (shipped under infrastructure/install/nomad/) and configure /etc/expanse/rightsizer.env:

EXPANSE_COMPUTE_TYPE=nomad
EXPANSE_RIGHTSIZER_ENABLED=true
EXPANSE_RIGHTSIZER_MODE=shadow

Then repoint job submission at the proxy, for example NOMAD_ADDR=http://127.0.0.1:8646. The proxy right-sizes the job register and plan calls and forwards everything else to the real agent unchanged. It is one in-cluster hop and horizontally scalable.

Rolling out: shadow first

Run in shadow through an initial data-collection window. Every decision is recorded (a structured log record plus a metric increment) while the original request is forwarded unchanged, so you can see exactly what active mode would have done at zero risk. Review the decision log and expanse_rightsizer_decisions_total, then flip mode to active when the would-be rewrites look right for your fleet. An automated shadow-versus-actuals comparison is upcoming; today the review is the decision log.

MIG packing

With migMultiplexing enabled on a MIG-partitioned cluster, a single whole-GPU serving request is rewritten to the smallest MIG slice whose usable memory holds the predicted footprint, so several servers share one accelerator. The rewrite applies only to single-GPU workloads. Prerequisites differ per scheduler:

Kubernetes: the NVIDIA device plugin must run the mixed MIG strategy so slices are schedulable as nvidia.com/mig-* resources. The right-sizer keeps its view of the cluster’s MIG geometry current by watching GPU-labelled nodes; no static inventory is needed.
SLURM: MIG profiles must be declared as typed gres in gres.conf, and the same profiles declared in expanse_rightsizer_mig_inventory. The sidecar cross-checks the declared inventory against gres.conf and drops any profile the file does not declare, so a mismatch only loses a packing opportunity. A request already stated with a GPU type (such as gres/gpu:a100:4) is tightened count-only, never rewritten to a MIG profile.
Nomad: the right-sizer tightens GPU count, CPU, and memory but never rewrites the device name. MIG device names are fingerprinted by the Nomad device plugin, and a wrong name would make the allocation unplaceable, so slice placement stays with your device plugin configuration.

Configuration reference

On Kubernetes, set these through the Helm rightsizer values; on SLURM, through the expanse_rightsizer_* playbook vars; on Nomad, in the unit’s environment file.

Variable	Default	Meaning
`EXPANSE_COMPUTE_TYPE`	required	`kubernetes`, `slurm`, or `nomad`; selects the front-end
`EXPANSE_RIGHTSIZER_ENABLED`	`false`	Master enable
`EXPANSE_RIGHTSIZER_MODE`	`shadow`	`shadow` or `active`
`EXPANSE_RIGHTSIZER_PREDICTOR`	`stats`	`stats` or `hydragt`
`EXPANSE_RIGHTSIZER_MIN_CONFIDENCE`	`medium`	Minimum confidence at which active mode rewrites (`high`, `medium`, `low`)
`EXPANSE_RIGHTSIZER_DEADLINE`	`200ms`	Decision deadline; on expiry the request passes through (the SLURM playbook sets `150ms`)
`EXPANSE_RIGHTSIZER_MIG_MULTIPLEXING`	`false`	Rewrite a single-GPU request to the smallest fitting MIG slice
`EXPANSE_RIGHTSIZER_MIG_INVENTORY`	empty	Static MIG inventory for SLURM and Nomad: `modelid:hbmGiB:profile,…;…`
`EXPANSE_RIGHTSIZER_SLURM_GRES_CONF`	`/etc/slurm/gres.conf`	gres.conf the SLURM sidecar cross-checks the inventory against
`EXPANSE_RIGHTSIZER_MIG_PROFILE_GEOMETRY`	empty	Override the built-in MIG profile geometry table
`EXPANSE_RIGHTSIZER_MIG_USABLE_FRACTION`	`0.90`	Fraction of a slice’s memory treated as usable by the fit
`EXPANSE_RIGHTSIZER_METRICS_ADDR`	`:9095`	Prometheus metrics listener
`EXPANSE_RIGHTSIZER_PROXY_ADDR`	`127.0.0.1:8646`	Nomad proxy listen address
`EXPANSE_RIGHTSIZER_NOMAD_UPSTREAM`	`http://127.0.0.1:4646`	Real Nomad agent behind the proxy
`EXPANSE_RIGHTSIZER_SLURM_SOCKET`	`/run/expanse/rightsizer.sock`	Unix socket the SLURM plugin dials
`EXPANSE_RIGHTSIZER_HYDRAGT_URL`	empty	Self-hosted HydraGT endpoint (egress-guarded)
`EXPANSE_AIR_GAPPED`	unset	Refuse any non-private predictor endpoint at startup

The stats predictor is analytical and runs entirely locally: with it, the right-sizer reaches no network at all. The hydragt learned predictor is not yet calibrated; selecting it today leaves the right-sizer inert (every request passes through) until calibration ships.

Observability

Every decision, in both modes, produces:

One structured log record (event=rightsizer_decision) carrying the action, mode, confidence, predictor engine, served model, original and proposed resources, the chosen MIG profile, and the decision latency. In shadow mode this record is the would-be right-size.
Prometheus metrics on the metrics listener:
- expanse_rightsizer_decisions_total, a counter labelled by action, mode, confidence, and engine.
- expanse_rightsizer_decision_latency_seconds, a histogram of per-decision wall-clock latency against the deadline.

Getting Started

Core Concepts

Deployment

Integrations

Inference right-sizer

What gets right-sized

Safety model

Enabling it

Rolling out: shadow first

MIG packing

Configuration reference

Observability

​What gets right-sized

​Safety model

​Enabling it

​Rolling out: shadow first

​MIG packing

​Configuration reference

​Observability

What gets right-sized

Safety model

Enabling it

Rolling out: shadow first

MIG packing

Configuration reference

Observability