Install on SLURM

A SLURM cluster is one compute in Expanse. One daemon runs on the controller (slurmctld host) and reports workload lifecycle, requested against used resources, GPU activity and source evidence. Workers run samplers without compute credentials and send samples through a shared filesystem. Only the controller connects to the control and data planes. The console’s Add compute panel shows your short-lived install token, control-plane and data-plane URLs, and pinned daemon version. Install the controller on the actual slurmctld host, which may differ from the login node. For AWS ParallelCluster, use the ParallelCluster guide.

Prefer to delegate? Copy this into an AI assistant with shell access on the controller and workers.

Prerequisites

For the fullest GPU diagnostics, configure both CUPTI and DCGM: CUPTI supplies detailed performance counters, while DCGM adds supported health, reliability and interconnect telemetry. Both are recommended on GPU nodes and remain optional; baseline GPU inventory and utilisation work with nvidia-smi alone.

Console access with permission to register a compute in the intended organisation.
Root or sudo on the controller and workers; systemd for managed services.
SLURM running, with scontrol, squeue and preferably sacct accessible.
The slurm OS account on every node, with consistent UID/GID across the shared mount. Use --user if your daemon account has another name.
A filesystem mounted at the same path on every node, such as /srv/expanse on NFS or Lustre. Single-node clusters can use local disk.
Working NVIDIA drivers and nvidia-smi on GPU nodes. Baseline metrics need neither a CUDA toolkit nor CUPTI.
Controller HTTPS access to your control and data planes and, for online installs, releases.expanse.sh or your release mirror.

Workers can fetch signed CUPTI releases during bootstrap and upload trace segments to your deployment’s object store. For workers without outbound access, stage CUPTI archives on the shared mount and use EXPANSE_SEGMENT_UPLOAD_MODE=relay as described under restricted networks. Both planes must be operational before installation. An HTTP response proves reachability; successful authentication and the first heartbeat prove enrolment.

1. Register the compute

Select the intended organisation in the console, open Compute → Add compute, choose SLURM, and generate an install token. Keep the panel open to check that this exact compute becomes active.

2. Install the CLI on the controller

command -v expanse >/dev/null 2>&1 || curl -fsSL https://expanse.sh/install | sh

Check expanse compute install --help against the flag reference when using an older CLI. CLI and daemon releases are versioned separately.

3. Install the controller daemon

Use the version and endpoints from Add compute. This Bash example keeps the token out of shell history and explicitly passes it through sudo:

(
  set -e
  read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN
  echo
  sudo EXPANSE_INSTALL_TOKEN="$EXPANSE_INSTALL_TOKEN" \
    expanse compute install --type slurm \
      --daemon-version <daemon-version> \
      --shared-prefix /srv/expanse \
      --control-plane <control-plane-url> \
      --data-plane <data-plane-url> \
      --patch-slurm-conf=true
)

Alternatively, run the sudo install command without the environment assignment and enter the token at its hidden prompt. An inherited EXPANSE_INSTALL_TOKEN takes precedence over that prompt. The installer verifies the release’s signed manifest, archive checksum and signature. It preflights paths, permissions, SLURM config and hook changes before exchanging the token. It installs the daemon, bundled py-spy, hooks and systemd unit, then waits for an authenticated heartbeat. A fresh install rolls back its managed changes on failure. --patch-slurm-conf=true authorises the planned Prolog, Epilog and TaskProlog changes. Without it, SLURM-owned paths remain read-only; your configuration management must apply the hook wiring. Foreign hooks may require explicit chaining. Use --slurm-conf /absolute/path/slurm.conf if discovery fails. The compute credential stays on the controller at /var/lib/slurm/.expanse/config.json, or your --config override. The unit pins that path through EXPANSE_DAEMON_CONFIG. Keep this file off the shared mount and do not edit its compute ID to change registration.

If installation reports an existing daemon config, decide whether you want an upgrade, a replacement credential or a new registration. --reinstall preserves the compute identity. On its own it skips install-token intake; with EXPANSE_INSTALL_TOKEN set it exchanges that token, which must be re-issued for this compute. A token typed into a previous failed invocation is not carried into the next invocation.

4. Start worker samplers

By default, the installed Prolog starts or updates a worker’s sampler when a job starts. Autoscaled workers need the shared mount and the daemon OS account; no per-worker token is used. For static workers that need telemetry before their first job, install the CLI on each worker and run:

sudo expanse compute install --type slurm --role sampler \
  --shared-prefix /srv/expanse

The worker uses the daemon binary already installed on the shared mount. Optional node-local settings and capability drop-ins must be provisioned on workers separately, including newly autoscaled nodes.

Skip the sampler on the controller, including single-node clusters. Its daemon already samples local GPUs. Running both roles on that node duplicates sampling.

5. Verify registration and telemetry

Check the services on their respective hosts:

# Controller
sudo systemctl status expanse-daemon --no-pager
sudo journalctl -u expanse-daemon --since "5 minutes ago" --no-pager

# Each worker
sudo systemctl status expanse-sampler --no-pager
sudo journalctl -u expanse-sampler --since "5 minutes ago" --no-pager
nvidia-smi --query-gpu=name,uuid,memory.total --format=csv

On a worker, test -f /srv/expanse/.expanse-preflight-sentinel && echo shared checks visibility of the controller’s preflight sentinel. Confirm it is the same shared mount, not an independently created local directory. In the console, check the intended organisation and compute ID, a fresh heartbeat, and the expected nodes and GPUs. Submit a job with sbatch and check its execution and metrics. An active heartbeat alone does not prove optional CUPTI, DCGM or Python stack collection is working; inspect the daemon diagnostics too. Idle GPUs can legitimately show zero utilisation.

Upgrade the existing compute

Reinstall with a new pinned daemon version and the same prefix/configuration:

sudo expanse compute install --type slurm --reinstall \
  --daemon-version <new-daemon-version> \
  --shared-prefix /srv/expanse \
  --patch-slurm-conf=true

This reuses the existing compute credential and saved plane endpoints. No new install token is needed; leave EXPANSE_INSTALL_TOKEN unset, because a set token is exchanged as a replacement credential. Repeat any non-default install flags, such as --user, --config, --slurm-conf or --profiling-agent=false. Workers converge to the new shared binary at their next job start. Re-run the sampler install on static workers to update immediately. Reinstall cleans the old managed installation before installing the replacement; if replacement fails, the identity remains but the old runtime is not automatically restored.

Replace a lost compute credential

If the controller is rebuilt or /var/lib/slurm/.expanse/config.json is lost, the compute keeps its identity and history. On the compute’s console page choose Re-issue install token. Then run the step 3 command on the controller with --reinstall added. Supply the re-issued token the same way:

(
  set -e
  read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN
  echo
  sudo EXPANSE_INSTALL_TOKEN="$EXPANSE_INSTALL_TOKEN" \
    expanse compute install --type slurm --reinstall \
      --daemon-version <daemon-version> \
      --shared-prefix /srv/expanse \
      --control-plane <control-plane-url> \
      --data-plane <data-plane-url> \
      --patch-slurm-conf=true
)

Same flags as the original install. Repeat --user, --config, --slurm-conf and the prefix. On AWS ParallelCluster omit --shared-prefix.
Keep --patch-slurm-conf=true. Reinstall cleanup removes the Expanse hook wiring and only that flag re-applies it. Without it the compute heartbeats but captures no jobs.
The token must match this compute. A token re-issued for a different compute is refused before anything on the node changes.
A rebuilt controller needs the CLI first (step 2). The plane URLs are required only when no saved config remains.
The previous credential stops at the exchange. A daemon still using it stops on its own. The token can be exchanged once.

The installer exchanges the token, writes the new credential, restarts the daemon and waits for a heartbeat carrying the new credential.

Reset and register a different compute

Uninstall is the explicit identity reset. It stops collection, removes managed files and hooks, and deprovisions the old compute. Run this in a maintenance window after draining affected jobs so their hooks and queued captures are not removed mid-run. Keep the same --user, --config and prefix overrides as the original installation.

# Each worker, before removing the shared controller artefacts
sudo expanse compute uninstall --type slurm --role sampler \
  --shared-prefix /srv/expanse

# Controller
sudo expanse compute uninstall --type slurm --role controller \
  --shared-prefix /srv/expanse --slurm-conf /etc/slurm/slurm.conf

If a sampler was also installed on the controller, uninstall that role there first. Stop on cleanup errors. Do not use --keep-identity for a registration reset, and do not delete config or lock files by hand. Review any operator-owned systemd drop-ins and /etc/expanse/daemon.env before reusing the host; uninstall is not a reset of your own configuration management. Return to step 1 in the intended organisation and perform a fresh install without --reinstall. --force is an alias for --reinstall, so it also preserves identity.

Optional: GPU performance counters with CUPTI

Baseline GPU metrics use nvidia-smi. CUPTI adds SM activity, tensor activity, DRAM throughput and other hardware performance counters. SLURM attempts CUPTI collection by default; its availability depends on the helper, driver and profiling permissions. Online controller installation selects a driver-compatible CUPTI variant from the signed release manifest and installs its helper, capture agent and matched libraries under <prefix>/lib/cupti-nodes/<hostname>/. Worker bootstrap selects for each worker’s driver independently. The released matrix includes CUDA 12.8 and 13.0 for amd64; select only a variant compatible with that node’s driver. Missing CUPTI does not block baseline telemetry or registration.

Allow restricted profiling counters

SLURM installs default to --cupti-mode=perfmon. The controller and sampler units grant CAP_PERFMON in both their bounding and ambient sets, including the unit distributed to new workers by the bootstrap. This requires Linux 5.8+ and NVIDIA R565+. Keep RmProfilingAdminOnly=1 on shared compute nodes. Use explicit --cupti-mode=sysadmin for older hosts, or --cupti-mode=none to grant neither capability. There is no automatic privilege fallback. The Ansible role forwards the same choice as expanse_cupti_mode. After installation, check the service on each GPU worker (use service=expanse-daemon on a controller that also runs GPU jobs):

service=expanse-sampler
sudo systemctl show "$service" -p User -p CapabilityBoundingSet -p AmbientCapabilities
sudo journalctl -u "$service" --since "2 minutes ago" --no-pager

A successful root-shell probe does not prove the service has permission. Verify PM sampling through the running service and its helper. Capability configuration alone does not prove driver/library compatibility. The in-job agent remains the job user and receives no service capability: restricted nodes permit Activity/NVTX capture, with PC sampling unavailable. Use --profiling-agent=false when the deployment excludes in-job injection. To disable CUPTI collection, set EXPANSE_CUPTI_ENABLE=0 in /etc/expanse/daemon.env on the relevant nodes and restart their Expanse service. The installer never changes the NVIDIA driver’s profiling policy.

Optional: DCGM enrichment

Expanse uses a DCGM host engine you already operate. It does not install DCGM or start nv-hostengine. Install your supported dcgmi client on GPU nodes and make the existing engine reachable, by default at localhost:5555. For a different engine or client location, add these runtime settings:

# /etc/expanse/daemon.env
EXPANSE_DCGM_HOST=127.0.0.1:5555
EXPANSE_DCGMI=/usr/bin/dcgmi

Restart the sampler on each affected worker, or the daemon on a GPU controller. Unsupported counters remain absent; baseline GPU metrics continue if DCGM is unavailable. On pre-Hopper GPUs, external DCGM profiling counter collection can contend with CUPTI for the performance monitor. Coordinate that collection with your existing monitoring stack; Expanse suppresses its own DCGM profiling groups while CUPTI owns the counters, while continuing compatible DCGM health collection.

Optional: Python stacks and deep capture

The daemon release bundles py-spy, which installs to <prefix>/bin/py-spy by default. You do not need to download it separately. Python stack attribution needs ptrace access to workload processes; the generated units grant CAP_SYS_PTRACE, but site security policy can still prevent collection. --bundled-pyspy=false skips the bundled copy; an operator-managed py-spy can still be selected through EXPANSE_PYSPY. --profiling-agent=true installs TaskProlog wiring for the in-process CUPTI capture agent when its library is available. This is separate from the CUPTI hardware-counter helper. Set --profiling-agent=false to skip this wiring; generated controller and sampler units also disable in-process capture and its permission check. CUPTI counter sampling remains enabled. Containers must see the shared prefix at the same path for the injected library to load. Set EXPANSE_PROFILING_ENABLE=0 at runtime to disable adaptive escalation. It does not disable baseline GPU metrics or the CUPTI Light counter set. To disable only in-process capture and its permission check, set EXPANSE_PROFILING_AGENT_ENABLE=0 in /etc/expanse/daemon.env on the controller and workers, then restart their Expanse services. Older units did not record --profiling-agent=false: preserve that opt-out when upgrading by reinstalling with the flag, or applying this runtime setting before restarting. The restarted service publishes a node-local disable marker that TaskProlog checks before exporting injection variables. This applies to newly launched tasks; running jobs retain their loaded library. Existing site TaskProlog output and operator-managed opt-outs are preserved. Older TaskProlog scripts need a reinstall with agent wiring enabled to gain this gate, or use their existing <control-dir>/disabled file on each worker. Jobs never read daemon.env. If the service cannot publish the disable marker, startup fails with the control path and error. Repair the path and restart to apply the opt-out. A failed Expanse service alone does not stop TaskProlog injection. With agent injection enabled, each running daemon or sampler checks the control directory and its ancestors, plus the shared-prefix root and spool, every 30 seconds. Missing other-execute (o+x) permission degrades profiling_permissions health with the exact path. The daemon restores only that bit in directories it owns inside /var/lib/expanse, the configured control root or the shared spool tree. The shared-prefix root and customer ancestors require operator repair. Worker failures reach the console through the controller’s heartbeat. Worker reports expire after two minutes, so an unavailable shared spool can prevent that relay.

Optional: restricted networks and air-gapped installs

Use --releases-url https://releases.internal.example to change the release manifest lookup host. Current manifests contain absolute archive URLs, which this flag does not rewrite: copying a manifest to an internal host alone does not keep downloads inside that mirror. For a site without release-host access, use the staged archive path below. Keep signatures with the archives; a mirror does not change the pinned verification key. For a staged install, replace --daemon-version with --daemon-binary /path/to/expanse-daemon_linux_amd64.tar.gz and keep its .sig beside it. Stage compatible CUPTI release archives and their .sig files beside the daemon archive for offline selection. Both planes must still be reachable at your deployment’s internal URLs. For workers without outbound access, make signed CUPTI archives available in <prefix>/lib/cupti-artifacts/ and configure the workers with:

# /etc/expanse/daemon.env
EXPANSE_SEGMENT_UPLOAD_MODE=relay

Provision this file on every node; it is node-local. The shared bootstrap renders services but does not distribute your environment file. For an HTTPS proxy or private CA, pass HTTPS_PROXY, NO_PROXY and SSL_CERT_FILE explicitly through sudo during installation. The installer seeds /etc/expanse/daemon.env with the network settings for the service. Preserve existing entries when adding runtime settings, and restart the relevant service after editing the file. Never store the install token there.

Troubleshooting

Symptom	What to check
`open install lock ... permission denied`	Re-run with sudo. A previous root install leaves an empty root-owned lock file; its presence does not mean an install is running. Do not delete an active install’s lock.
`existing daemon config found`	Choose upgrade, replacement credential or reset above. A token for another compute is refused; the saved identity stays.
Active compute, no GPUs on the page	Compare the daemon log’s compute ID with the console URL and selected organisation. Check `nvidia-smi`, the worker mount and sampler logs.
`expanse-cupti-helper not found`	Inspect the CUPTI selection message during install. Verify the installed helper path in `systemctl cat` and use a compatible, signed release. Installing a CUDA toolkit alone does not install Expanse’s helper.
CUPTI archive `unsupported type 49`, or rejected `lib/cupti/versions.tsv`	A release-packaging/extractor mismatch, observed with v1.8.0. Use CLI and daemon releases containing the fix when available; repeating the same install will not repair the archive. Baseline metrics remain available.
`CUPTI PM sampling unavailable`	Inspect per-device `pm_sampling_reason` from the installed helper’s `probe --json`. `CUPTI_ERROR_INSUFFICIENT_PRIVILEGES` requires the service capability above; older diagnostics can hide it behind the generic message. Root and service-user probes have different privileges.
`workers: 0 of 1 sampled worker nodes`	One worker supplies baseline samples; zero supplies the named optional capability. It does not mean all GPU telemetry is missing.
Profiling `permission denied` or `operation not permitted`	Inspect ownership and mount permissions at the exact reported path as the daemon user. Shared profiling spool and node-local control paths have different owners and purposes; do not apply a blanket recursive chmod.
`profiling_permissions` reports `missing o+x`	Inspect the named path and any node in the reason. For customer-managed ancestors, restore traversal through your configuration management when profiling is intended. Managed directories owned by the daemon are repaired automatically; failed repairs retain the reason.
Worker writes samples but console has none	Check controller logs for `sampler aggregator drained` and read/parse failures. Confirm matching shared prefix, hostname and UID/GID across nodes.

Install flag reference

SLURM is configured with CLI flags and node-local systemd settings. It has no Helm values.yaml. These are the controller/sampler install flags; use expanse compute install --help for the CLI version installed on your host.

Flag	Default	What it does
`--type`	unset	Set `slurm` for cluster installation; unset is config-only.
`--role`	`controller`	`sampler` installs the worker service without a token.
`--shared-prefix`	unset	Absolute path shared by every node; a validated platform may supply a default.
`--daemon-version`	unset	Pinned daemon release to download and verify on the controller.
`--daemon-binary`	unset	Staged daemon archive with adjacent `.sig`, or an explicit bare binary override. Mutually exclusive with `--daemon-version`; a bare binary has no archive signature verification.
`--control-plane`	discovered	Supply the Add compute URL for a fresh install or a rebuilt controller; otherwise reinstall uses the saved endpoint.
`--data-plane`	unset	Required for a fresh install or a rebuilt controller; otherwise reinstall reuses the saved endpoint.
`--config`	derived	Controller credential path; normally `/var/lib/slurm/.expanse/config.json`. Repeat overrides on upgrade/reset.
`--user`	`slurm`	Existing OS account for the daemon and sampler.
`--slurm-conf`	discovered	Override the controller’s authoritative SLURM config path.
`--patch-slurm-conf`	`false`	Apply planned hook changes; otherwise leave SLURM-owned paths read-only.
`--reconfigure`	`true`	Reconfigure SLURM after applicable hook changes; false prints the command.
`--systemd`	`true`	Install/start the service. False prints the unit for manual setup and does not wait for a heartbeat.
`--heartbeat-timeout`	`1m`	Controller first-heartbeat timeout with managed systemd.
`--sampler-bootstrap`	`prolog`	Start/update workers from Prolog; `off` requires explicit worker installs.
`--cupti-variant`	automatic	Pin a compatible variant, e.g. `cuda1300`, with the controller’s daemon release.
`--cupti-mode`	`perfmon`	Counter capability: `perfmon` (Linux 5.8+, NVIDIA R565+), explicit `sysadmin`, or `none`.
`--profiling-agent`	`true`	Install TaskProlog and available in-process capture agent; false skips wiring and disables Phase B and its permission check in generated units.
`--bundled-pyspy`	`true`	Install the py-spy shipped beside the resolved daemon binary.
`--releases-url`	`https://releases.expanse.sh`	Release mirror base URL; signatures still required.
`--cosign-key`	embedded key	Override the verification PEM for an agreed key rotation.
`--reinstall`, `--force`	`false`	Clean/reinstall keeping the compute identity. Reads `EXPANSE_INSTALL_TOKEN` only to exchange a token re-issued for this compute; never prompts.
`--write-config-only`	`false`	Exchange/write config without installing a cluster runtime.
`--verbose`	`false`	Print install and rollback diagnostics.

Runtime configuration reference

Add settings to the existing /etc/expanse/daemon.env on each affected node, preserving proxy/CA entries. Restart that node’s Expanse service to apply them. An export in your interactive shell does not configure a systemd service.

Setting	Default	What it does
`EXPANSE_CUPTI_ENABLE`	`1`	`0` disables CUPTI counter sampling before helper discovery.
`EXPANSE_CUPTI_HELPER`	discovered/pinned	Explicit helper executable with its matched sibling libraries; normally managed by the installer.
`EXPANSE_PROFILING_ENABLE`	enabled	`0` disables adaptive profiling escalation.
`EXPANSE_PROFILING_AGENT_ENABLE`	install flag, otherwise `1`	`0` disables in-process capture and its traversal permission check while retaining Phase A counter escalation.
`EXPANSE_PYSPY`	bundled/PATH	Override the Python stack sampler executable.
`EXPANSE_DCGMI`	PATH	Override the existing `dcgmi` client executable.
`EXPANSE_DCGM_HOST`	`localhost:5555`	Your existing DCGM host-engine address.
`EXPANSE_SEGMENT_UPLOAD_MODE`	`auto`	Chooses a transport from available grants; `relay` sends worker trace segments through the controller for upload.
`HTTPS_PROXY`, `NO_PROXY`, `SSL_CERT_FILE`	environment	Proxy routing and private CA settings.

Reusable controller install script

Save this as install-expanse.sh, replace the four deployment values, and run bash install-expanse.sh for a fresh registration. It contains no credential. Use the upgrade command for an existing compute. Apply optional runtime settings and capability drop-ins separately on each node.

#!/usr/bin/env bash
set -euo pipefail

daemon_version='<daemon-version>'
shared_prefix='/srv/expanse'
control_plane='<control-plane-url>'
data_plane='<data-plane-url>'

read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN
echo
trap 'unset EXPANSE_INSTALL_TOKEN' EXIT
sudo EXPANSE_INSTALL_TOKEN="$EXPANSE_INSTALL_TOKEN" \
  expanse compute install \
    --type slurm \
    --role controller \
    --daemon-version "$daemon_version" \
    --shared-prefix "$shared_prefix" \
    --control-plane "$control_plane" \
    --data-plane "$data_plane" \
    --patch-slurm-conf=true \
    --reconfigure=true \
    --systemd=true \
    --user slurm \
    --sampler-bootstrap prolog \
    --bundled-pyspy=true \
    --profiling-agent=true \
    --heartbeat-timeout 1m

Getting Started

Installation

Core Concepts

Deployment

Integrations

Install on SLURM

Prerequisites

1. Register the compute

2. Install the CLI on the controller

3. Install the controller daemon

4. Start worker samplers

5. Verify registration and telemetry

Upgrade the existing compute

Replace a lost compute credential

Reset and register a different compute

Optional: GPU performance counters with CUPTI

Allow restricted profiling counters

Optional: DCGM enrichment

Optional: Python stacks and deep capture

Optional: restricted networks and air-gapped installs

Troubleshooting

Install flag reference

Runtime configuration reference

Reusable controller install script

Getting Started

Installation

Core Concepts

Deployment

Integrations

​Prerequisites

​1. Register the compute

​2. Install the CLI on the controller

​3. Install the controller daemon

​4. Start worker samplers

​5. Verify registration and telemetry

​Upgrade the existing compute

​Replace a lost compute credential

​Reset and register a different compute

​Optional: GPU performance counters with CUPTI

​Allow restricted profiling counters

​Optional: DCGM enrichment

​Optional: Python stacks and deep capture

​Optional: restricted networks and air-gapped installs

​Troubleshooting

​Install flag reference

​Runtime configuration reference

​Reusable controller install script

Prerequisites

1. Register the compute

2. Install the CLI on the controller

3. Install the controller daemon

4. Start worker samplers

5. Verify registration and telemetry

Upgrade the existing compute

Replace a lost compute credential

Reset and register a different compute

Optional: GPU performance counters with CUPTI

Allow restricted profiling counters

Optional: DCGM enrichment

Optional: Python stacks and deep capture

Optional: restricted networks and air-gapped installs

Troubleshooting

Install flag reference

Runtime configuration reference

Reusable controller install script