# Architecture Source: https://docs.expanse.sh/architecture How Expanse captures workload evidence, learns from it, and manages your data. Expanse runs its own models for resource sizing, failure diagnosis, and optimisation suggestions, and grounds every answer in evidence from your own compute. Those models improve through the Expanse data flywheel: every captured workload adds evidence about what ran, what resources it used, how it finished, and which recommendation helped next time. The daemon captures workload and compute telemetry. Evidence stays inside the configured deployment. Every captured workload sharpens resource sizing, diagnosis, and optimisation. Analyse and diagnose recommendations shape the next workload run. Each run feeds the next one, meaning the models get even better over time, and better answers produce better future runs. ## Planes | Plane | Role | | ---------------------- | ------------------------------------------------------------------------------------------ | | **Data plane** | Stores evidence inside the configured deployment. | | **Intelligence plane** | Runs the models behind analyse, diagnose, and optimisation. | | **Control plane** | Handles identity, organisations, endpoint discovery, registration, and licence validation. | ## Data boundaries The daemon runs on your compute and captures very granular data. It is a collector, not a plane. For privacy-concerned enterprises, the data plane and intelligence plane can run on your network. If you prefer Expanse-managed infrastructure, they can run on ours. It is your choice. The hosted Console at [console.expanse.sh](https://console.expanse.sh) is the user-facing view of your compute and intelligence workflows, not where telemetry is stored. The control plane handles identity, registration, endpoint discovery, and licence validation. It does **not** receive any form of workload data we collect. NOT A SINGLE BYTE. ## Trust Expanse is SOC 2 Type II compliant. # Changelog Source: https://docs.expanse.sh/changelog Product updates, enhancements, and fixes for the Expanse platform. The latest updates, improvements, and fixes to the Expanse platform. Platform (CLI, daemon, control+data planes) refinements Platform refinements and SOC 2 compliance hardening. Initial release. # CLI Reference Source: https://docs.expanse.sh/cli/overview Every command the Expanse CLI exposes. The `expanse` CLI is the primary way to register compute, run workloads, and ask intelligence questions about them. ## Installation ```bash theme={"dark"} curl -fsSL https://expanse.sh/install | sh ``` *** ## Auth ### `expanse login` Store your personal `exp_user_*` API key locally and mint a session. Pass the key with `--api-key` or set `EXPANSE_API_KEY` (the key is created from the Console settings page after you sign in through your organisation's SSO). ```bash theme={"dark"} # provide your exp_user_* key with --api-key (or set EXPANSE_API_KEY): expanse login --api-key exp_user_… ``` | Flag | What it does | | ------------------- | ------------------------------------------------------------------------------------------------------- | | `--api-key` | User API key (defaults to `EXPANSE_API_KEY`) | | `--control-plane` | Control-plane URL, for self-hosted deployments (defaults to `EXPANSE_API_URL` or your existing session) | | `--organisation-id` | Organisation to mint the session for | ### `expanse status` Show which identity the CLI is acting under. ```bash theme={"dark"} expanse status ``` ### `expanse refresh` Mint and cache a fresh short-lived access token for the current session. The CLI does this automatically; use `refresh` to force one, or to switch organisation with `--organisation-id`. ```bash theme={"dark"} expanse refresh ``` *** ## Compute ### `expanse compute register` Register a new compute with your organisation. The command asks what kind of compute you're registering (or pass `--type`) and prints a single-use install token plus the install command to run on the target. ```bash theme={"dark"} expanse compute register ``` | Flag | What it does | | ----------- | -------------------------------------------------------------------- | | `--type` | Compute type (`slurm`, `kubernetes`, `nomad`) | | `--name` | Compute name (default: derived from the type and timestamp) | | `--team-id` | Team to bind the compute to (team IDs are on the Console Teams page) | ### `expanse compute install` Run on the target machine. Exchanges the install token (set `EXPANSE_INSTALL_TOKEN`) for the compute's stable credential and writes the daemon config. `expanse compute register` prints the exact command to run, including `--control-plane` and `--data-plane`. ```bash theme={"dark"} EXPANSE_INSTALL_TOKEN=… expanse compute install \ --control-plane https://… --data-plane https://… \ --type slurm --daemon-version v1.2.1 ``` With `--type slurm` the installer downloads the daemon release named by `--daemon-version`, verifies its checksum and cosign signature against the pinned Expanse release signing key, places the daemon and scheduler hooks, and enables the systemd unit. `--daemon-binary` installs a pre-staged binary or release archive instead (the air-gapped path; archives are verified against the same key). `--write-config-only` skips cluster orchestration and only writes the config file. *** ## Workloads ### `expanse executions` List recent workload executions for your organisation, newest first. The quickest way to find an execution ID for `expanse diagnose` or `expanse metrics` without opening the Console. ```bash theme={"dark"} expanse executions # 25 most recent expanse executions --status failed # only failed runs expanse executions --outcome out_of_memory # only OOM-killed runs expanse executions --mine # runs you submitted ``` The table shows the execution ID, scheduler-native job ID, submitting user, status, outcome, start time, and a Console link. | Flag | What it does | | ----------- | -------------------------------------------------------------------------------------------- | | `--limit` | Maximum executions returned (default 25) | | `--status` | Filter by status (`queued`, `running`, `succeeded`, `failed`, `cancelled`, `unknown`) | | `--outcome` | Filter by outcome (`success`, `failure`, `cancelled`, `timeout`, `out_of_memory`, `unknown`) | | `--user` | Filter by submitting user | | `--mine` | Filter to executions you submitted | | `--compute` | Target compute ID | | `--json` | Print the raw JSON response | The submitting user comes from the scheduler and is best-effort: executions whose telemetry carried no user stay unattributed and are not matched by `--user` or `--mine`. ### `expanse metrics` Export metric series for one execution as a summary table, CSV, or JSON. ```bash theme={"dark"} expanse metrics gpu-util gpu-memory-used --csv > gpu.csv expanse metrics cpu memory --interval 30s --agg max expanse metrics --job 41982 --source-type slurm gpu-util # resolve by SLURM job ID expanse metrics --dcgm --all --csv # every DCGM counter ``` Name at least one metric: `gpu-util`, `gpu-memory-used`, `gpu-memory-total`, `gpu-power`, `gpu-temp`, `gpu-clock`, `cpu`, `memory`. Any other name is passed through raw, so CUPTI PM sampling and DCGM counters keep their native names. | Flag | What it does | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `--csv` | Long-format CSV to stdout, one row per point: `timestamp,metric,source,gpu_index,gpu_uuid,value,min,max,avg,sample_count,unit` | | `--json` | Raw JSON responses to stdout | | `--interval` | Aggregation bucket width in whole seconds, for example `5s` or `1m` | | `--agg` | Bucket aggregation: `avg` (default), `min`, `max`, `sum`, `p95`, `last` | | `--since` | Window relative to now, for example `2h` | | `--from` / `--to` | Window start and end (RFC 3339); intersected with the execution's own lifecycle window | | `--gpu` | Restrict to one GPU index | | `--allow-fallback` | Widen to compute scope when the execution has no scoped data (may include other jobs' GPUs) | | `--cupti` / `--dcgm` | Only CUPTI PM sampling / only DCGM series | | `--all` | Discover and export every metric for the selected source (requires `--cupti` or `--dcgm`) | | `--job` / `--source-type` | Resolve the execution by scheduler-native job ID (`slurm`, `kubernetes`, `nomad`) | | `--max-points` | Maximum points per series (default 5000) | `--csv` and `--json` write data to stdout only; progress and warnings go to stderr, so piped output stays machine-clean. When a series hits `--max-points` the CLI warns on stderr and suggests a wider `--interval`. *** ## Intelligence ### `expanse analyse` Recommend resources **before** a workload runs. Accepts a SLURM batch script, a source file, a Kubernetes workload manifest, or a Nomad jobspec. ```bash theme={"dark"} expanse analyse train.slurm # SLURM batch script expanse analyse train.py # source file expanse analyse job.yaml # Kubernetes workload manifest expanse analyse job.nomad # Nomad jobspec ``` The output is `RESOURCE RECOMMENDATION` (walltime, CPU, host memory, GPU count and type, the measured GPU peak and utilisation of earlier runs, and `completion_probability`, the probability a run completes at the recommended resources computed from every recorded run of the same source), `FAILURE RISK` at the resources you requested, `OPTIMISATION` (the resource change proposed, as a class such as `memory_lower+walltime_raise`), followed by the executions on your compute that anchored the numbers. Units live in the row name (`_s`, `_gb`, `_pct`); the one `confidence` label is `low`, `medium` or `high`, with the fit score beside it when recorded runs anchor the result. Analyse proposes resource changes only, never code changes. When the recommendation changes your submission, analyse produces a patch artefact and, on an interactive terminal, asks `Apply the resource recommendations automatically? [y/n]`: `y` shows the diff and applies it, `n` leaves these commands for later: ```bash theme={"dark"} expanse analyse diff --artefact # inspect the patch expanse analyse apply --artefact # apply it locally ``` | Flag | What it does | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `--compute` | Target compute ID | | `--cpu` / `--memory` / `--gpu` / `--walltime` | Requested resources, for example `--memory 16GiB --walltime 2h` | | `--json` | Print the raw JSON response | | `--no-cache` | Force a fresh analysis instead of returning the cached run | | `--no-patch` | Skip patch artefacts; return the recommendation numbers only (`OPTIMISATION` reads `none`) | | `--yes` | Apply the resource patch without asking (never code patches; not with `--json`, `--no-patch` or `--queue`) | | `--queue` | Queue analyse runs instead of waiting for recommendations | `analyze` works as a spelling alias. ### `expanse diagnose` Root cause of a failed execution, with cited evidence and a prompt for your coding agent. The output opens with `PIPELINE DIAGNOSE` (what evidence each stage found), then the execution's identity, `RESOURCE SNAPSHOT` (same names and units as `expanse analyse`, GPU memory per device) and, for a failed run, `FAILURE DETECTION`: the failure pattern, the confidence label, `recurrence` (the share of this workload's earlier recorded runs that failed) and the root cause. ```bash theme={"dark"} expanse diagnose expanse diagnose 41982 --source-type slurm # scheduler-native job ID ``` `` is the Expanse execution ID (find it with `expanse executions`). If you only have the scheduler-native ID, pass it with `--source-type` (`slurm`, `kubernetes`, `nomad`). The diagnosis cites the evidence it used: telemetry, logs, the captured source bundle, and similar executions on your compute. Diagnose never proposes or applies a fix. A failed execution carries a prompt for your own coding agent, which owns the change. On an interactive terminal diagnose offers it (`Suggested prompt for your agent? [y/n]`); `--prompt` prints it bare for piping into a clipboard tool: ```bash theme={"dark"} expanse diagnose --prompt | pbcopy # xclip or wl-copy on Linux ``` | Flag | What it does | | --------------- | --------------------------------------------------------------------------- | | `--source-type` | Treat the argument as a scheduler-native job ID from this source | | `--source` | Source file to include as evidence (repeatable) | | `--compute` | Target compute ID | | `--prompt` | Print only the prompt for your coding agent, for piping to a clipboard tool | | `--json` | Print the raw JSON response | | `--no-cache` | Force a fresh diagnosis instead of returning the cached run | *** ## Other ### `expanse version` Print the CLI version. ```bash theme={"dark"} expanse version ``` *** ## Environment variables | Variable | What it does | | ----------------------- | --------------------------------------------------------------------------- | | `EXPANSE_API_KEY` | Personal `exp_user_*` key, read by `expanse login` | | `EXPANSE_API_URL` | Override the control-plane URL (defaults to your organisation's deployment) | | `EXPANSE_INSTALL_TOKEN` | Single-use install token consumed by `expanse compute install` | | `EXPANSE_CONSOLE_URL` | Console base URL for the links printed by `expanse executions` | # Computes Source: https://docs.expanse.sh/concepts/computes What a compute is and what Expanse supports. A **compute** is any environment running `expanse-daemon`. A SLURM cluster, a Nomad cluster, and a Kubernetes cluster are all the same thing from Expanse's point of view: one row on the Compute page in the Console. ## Supported environments | Type | Where the daemon runs | | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `slurm` | The controller node of a SLURM cluster. Every job (`sbatch`, `srun`, `salloc`) is captured automatically via scheduler prolog/epilog hooks. | | [`nomad`](/integrations/nomad) | A Nomad client node with access to the Nomad HTTP API. Captures allocation and task lifecycle telemetry automatically. | | `kubernetes` | Installed via Helm. Watches all non-system namespaces by default (or a configured namespace list) and captures every pod in the watched namespaces, including pods owned by Volcano, Kueue, Argo Workflows, Flux, Ray, and Flyte. | ## Registering a compute ```bash theme={"dark"} expanse compute register ``` The command asks what kind of compute you're registering and prints the install command or bootstrap snippet to run on the target. The compute appears in the Console within a minute of the daemon starting. Per-provider walkthroughs: [SLURM](/installation/slurm), [AWS ParallelCluster](/installation/parallelcluster), [Kubernetes](/installation/kubernetes), [Nomad](/installation/nomad), and [VM / generic hosts](/installation/vm). ## Compute states Every compute is in one of these states: | State | Meaning | | ---------- | -------------------------------------------------------------------- | | `pending` | Registered but the daemon has not connected yet. | | `active` | Daemon is heartbeating. | | `degraded` | Heartbeat has been missing for a short window. | | `offline` | Heartbeat has been missing long enough to consider the compute gone. | | `revoked` | The compute key was revoked or the compute was removed. | Historical data for an offline or revoked compute stays queryable in the Console. # Intelligence Source: https://docs.expanse.sh/concepts/intelligence Resource recommendations and failure diagnosis grounded in your compute's own history. Expanse exposes two intelligence commands. Both reason over evidence from your own compute. **Before** submission. Recommends CPU, memory, GPU and walltime, with the probability the run completes. **After** failure. Root cause with cited evidence, plus a prompt for your coding agent. *** ## Resource recommendation `expanse analyse` recommends what a workload will need before you submit it. It accepts a SLURM batch script, a source file, a Kubernetes workload manifest, or a Nomad jobspec. ```bash theme={"dark"} expanse analyse train.slurm ``` ```text theme={"dark"} EXPANSE ANALYSE target train.slurm job_name train run 8b2f4c1e-… status run_status_succeeded RESOURCE RECOMMENDATION walltime_s 9000 gpu_type NVIDIA H100 80GB HBM3 gpu_count 1 gpu_memory_available_gb 79.6 gpu_memory_peak_gb 31.2 gpu_utilisation_pct 91 cpu_cores 16 host_ram_gb 15 gpu_memory_floor_gb 29.8 host_ram_requested_gb 128 completion_probability mean=0.90 sigma=0.12 p10=0.73 p90=1.00 confidence high (0.83, 4 of 4 measured runs fit) OPTIMISATION class memory_lower+walltime_raise review_required yes source_file train.slurm EVIDENCE successful_runs 4 (SLURM jobs 4401, 4402, 4403, 4404; similarity 0.98) Apply the resource recommendations automatically? [y/n] ``` ```bash theme={"dark"} expanse analyse train.slurm --json ``` ```json theme={"dark"} { "run_id": "8b2f4c1e-…", "status": "RUN_STATUS_SUCCEEDED", "result": { "recommended_resources": { "cpu_cores": 16, "memory_bytes": "16106127360", "gpu_count": 1, "walltime_seconds": "9000", "rationale": "…", "gpu_model": "NVIDIA H100 80GB HBM3", "gpu_memory_available_bytes": "85520809984", "gpu_memory_peak_bytes": "33500000000", "gpu_utilisation_pct": 91, "gpu_utilisation_known": true }, "confidence": "CONFIDENCE_LABEL_HIGH", "confidence_score": 0.83, "confidence_score_fit": 4, "confidence_score_runs": 4, "completion_probability": { "mean": 0.9, "sigma": 0.12, "p10": 0.73, "p90": 1, "success_count": 4, "basis": "exact_bundle" }, "residual_failure_risk": { "probability": { "mean": 0.1, "sigma": 0.12, "p10": 0, "p90": 0.27, "success_count": 4, "basis": "exact_bundle" }, "class": "none" }, "right_size": { "class": "memory_lower+walltime_raise", "source_file": "train.slurm", "artifact_id": "…", "review_required": true }, "artifacts": [ { "artifact_id": "…", "artifact_type": "ARTIFACT_TYPE_UNIFIED_DIFF", "content_hash": "sha256:…" } ] } } ``` Every row is a limit to request, with the unit in its name: `_s` is seconds, `_gb` is memory in GiB (2^30 bytes, so an 80 GB card reads 79.6), `_pct` is percent. `gpu_memory_peak_gb` and `gpu_utilisation_pct` are measured on earlier runs of the same source on your compute; `gpu_memory_floor_gb` is the per-GPU memory the workload needs; `host_ram_requested_gb` is your own request, so the change is visible beside the recommendation. `completion_probability` is the probability that a run of this workload completes at the recommended resources. It is computed from every recorded run of the same source on your compute, one run one observation: `mean` is the estimate, `sigma` its spread, `p10` and `p90` the range it most likely falls in. Four recorded successes read `mean=0.90` with a wide spread; the spread narrows as your history grows. With fewer than two recorded runs the row reads `unknown` rather than a guess. `FAILURE RISK` lists the risks at the resources you requested (none here, because every recorded run succeeded); the risk that remains once you adopt it is in the JSON as `residual_failure_risk`. `confidence` is the one confidence label on a result: `low`, `medium` or `high`. Runs of the same source on your compute make it `high`, and the score beside it is the probability that the next run fits the recommendation, counted from those runs: four runs that all fit read `0.83`, ten read `0.92`. `EVIDENCE` names the runs the numbers came from. `OPTIMISATION` names the resource change analyse proposes as a class (`memory_lower+walltime_raise`); on an interactive terminal analyse then asks `Apply the resource recommendations automatically? [y/n]` and shows the diff as it applies. Analyse proposes resource changes only; it never suggests changing your code. For a workload with no history there is nothing to cite, so the recommendation comes from your source alone, flagged `low` confidence with explicit missing-evidence warnings rather than a fabricated history. Patches are review-first: analyse never touches your files on its own. On an interactive terminal it asks `Apply the resource recommendations automatically? [y/n]`; `y` shows the diff and applies it, `n` leaves the `expanse analyse diff` and `expanse analyse apply` commands for later (`--yes` skips the confirmation). A patch that lowers a directive is marked `review_required yes`. *** ## Failure diagnosis `expanse diagnose` explains why an execution failed. It returns the root cause with the evidence it used (telemetry, logs, the captured source bundle, and similar executions of the same workload) and a prompt you paste into your own coding agent to make the change. ```bash theme={"dark"} expanse diagnose 7f3e9a2b-… ``` ```text theme={"dark"} PIPELINE DIAGNOSE evidence done execution record, scheduler status similar done 5 similar executions sources done 3 captured files, 2 fetched attribution done 4 regions, 12 frames metrics done 6 series, snapshot runbooks done host_out_of_memory synthesis done deterministic agent-prompt done 11k chars EXPANSE DIAGNOSE target 7f3e9a2b-… job_name train state failed failure_class host_out_of_memory compute hpc-01 RESOURCE SNAPSHOT runtime_s 2531 gpu_type NVIDIA H100 80GB HBM3 cuda_version 12.4 gpu_count 1 gpu_utilisation_pct 91 gpu_memory_available_gb 79.6 gpu_memory_observed_gb 31.2 cpu_requested 16 host_ram_observed_peak_gb 24 host_ram_requested_gb 24 FAILURE DETECTION pattern host_out_of_memory confidence high recurrence mean=0.05 sigma=0.07 p10=0.00 p90=0.14 root_cause What happened: the execution was killed at its 24.0 GiB host (CPU) RAM limit (--mem) by the out-of-memory handler. The sampled peak of 24.0 GiB is a floor, not the demand: the kill at the limit proves true demand exceeds 24.0 GiB by an unknown margin. A completed similar run measured a 36.2 GiB peak host RSS, the best available estimate of this workload's true demand. Why: terminal status or logs report a host out-of-memory kill with no device-memory signal. ``` ```bash theme={"dark"} expanse diagnose 7f3e9a2b-… --json ``` ```json theme={"dark"} { "run_id": "d41c8a7e-…", "status": "RUN_STATUS_SUCCEEDED", "result": { "root_cause": "What happened: the execution was killed at its 24.0 GiB host (CPU) RAM limit (--mem) by the out-of-memory handler. The sampled peak of 24.0 GiB is a floor, not the demand: the kill at the limit proves true demand exceeds 24.0 GiB by an unknown margin. A completed similar run measured a 36.2 GiB peak host RSS, the best available estimate of this workload's true demand. Why: terminal status or logs report a host out-of-memory kill with no device-memory signal.", "confidence": "CONFIDENCE_LABEL_HIGH", "evidence": [ { "execution_id": "5b8e1f04-…", "summary": "9 similar workloads succeeded with 36G or more" } ], "display": { "target": "7f3e9a2b-…", "job_name": "train", "state": "failed", "failure_class": "host_out_of_memory", "compute": "hpc-01", "resource_snapshot": { "runtime_seconds": "2531", "gpu_type": "NVIDIA H100 80GB HBM3", "cuda_version": "12.4", "gpu_count": 1, "gpu_utilisation_pct": 91, "gpu_memory_available_bytes": "85520809984", "gpu_memory_observed_bytes": "33500000000", "cpu_requested": 16, "host_ram_observed_peak_bytes": "25769803776", "host_ram_requested_bytes": "25769803776" }, "field_status": { "failure_class": "DIAGNOSE_FIELD_STATUS_ERROR", "gpu_utilisation_pct": "DIAGNOSE_FIELD_STATUS_OK", "gpu_memory_available_gb": "DIAGNOSE_FIELD_STATUS_OK", "gpu_memory_observed_gb": "DIAGNOSE_FIELD_STATUS_OK", "host_ram_observed_peak_gb": "DIAGNOSE_FIELD_STATUS_ERROR", "host_ram_requested_gb": "DIAGNOSE_FIELD_STATUS_ERROR" }, "recurrence": { "mean": 0.05, "sigma": 0.07, "p10": 0, "p90": 0.14, "success_count": 9, "basis": "exact_bundle" } }, "pipeline": [ { "name": "evidence", "status": "PIPELINE_STAGE_STATUS_OK", "summary": "execution record, scheduler status" }, { "name": "runbooks", "status": "PIPELINE_STAGE_STATUS_OK", "summary": "host_out_of_memory" } ], "agent_prompt": "…" } } ``` `PIPELINE DIAGNOSE` shows what evidence the diagnosis stands on: one row per stage with `done`, `degraded` or `skipped` and a short note of what it found. `RESOURCE SNAPSHOT` uses the same names and units as `expanse analyse`; the GPU memory figures are per GPU, so the observed peak reads against one device's capacity. `FAILURE DETECTION` carries the failure pattern, the confidence label, the root cause, and `recurrence`: the share of this workload's recorded runs that failed, excluding this one, computed from every recorded run of the same source. With fewer than two other recorded runs the row is omitted rather than guessed. Diagnose never proposes or applies a fix. The prompt is for your own coding agent, which owns the change: it carries the job, the diagnosis, the resource snapshot, telemetry peaks, code hotspots, source paths, a bounded log tail, and similar executions that succeeded, and it asks the agent to verify the root cause against your repository before changing anything. The question appears only on an interactive terminal; the JSON carries the same prompt as `result.agent_prompt`. To get the prompt bare, for piping into a clipboard tool or over SSH: ```bash theme={"dark"} expanse diagnose --prompt | pbcopy # macOS expanse diagnose --prompt | xclip -selection clipboard # X11 expanse diagnose --prompt | wl-copy # Wayland ``` In the console, the failed-run card has a "Copy prompt for agent" action that puts the same prompt on your clipboard. # Telemetry Source: https://docs.expanse.sh/concepts/passive-telemetry Every scheduled workload on your compute, captured automatically. Once the daemon is registered, Expanse captures **every workload the scheduler sees**, with nothing for users to install, import, or remember. Complete coverage gives you accurate waste numbers, sharper recommendations, and an honest view of how the compute estate is being used. ## What gets captured For every workload, Expanse captures three views: * **Before:** what the scheduler asked for, the submitting user, the queue, the requested resources. * **During:** live metrics streamed throughout the run including CPU, memory, and GPU utilisation, memory, power, and clocks. * **After:** real runtime, peak memory, real utilisation, exit status, and error context for failures. Together, those views feed the Console's waste dashboards and the [intelligence layer](/concepts/intelligence). **Low overhead, never blocking.** Baseline capture observes the scheduler from the outside; optional GPU profiling runs in short, bounded windows so the amortised cost stays small. Telemetry never blocks workload execution: if the daemon or the data plane is unreachable, jobs run normally and telemetry is buffered locally until it can be delivered. ## Compute coverage Expanse captures workloads automatically where the scheduler can be observed: * **SLURM:** captures every job (`sbatch`, `srun`, `salloc`) via scheduler prolog/epilog hooks, including user, account, partition, requested CPU, memory, GPU, walltime, job state, runtime, exit code, and scheduler diagnostics. * **Kubernetes:** captures pods across the cluster's non-system namespaces (or a configured namespace list for scoped installs), including owner references for Volcano, Kueue, Argo Workflows, Flux, Ray, and Flyte, plus requested resources, limits, node placement, status, runtime, and log tails. * **Nomad:** captures allocations placed on the cluster, including job, group, task, namespace, and node context, requested CPU, memory, and device resources, allocation and task state, runtime, exit status, and failure context. See [Computes](/concepts/computes) for how these environments map to compute types. If you already run Prometheus, Grafana, OpenTelemetry, or another telemetry pipeline, [reach out](mailto:contact@expanse.sh) and we'll scope the integration. # Teams & roles Source: https://docs.expanse.sh/concepts/teams-and-roles How organisations, teams, and roles control access in Expanse. Every Expanse user belongs to an **organisation**. Within an organisation, **teams** group members and control which computes each member sees. Membership and role assignment are managed from the [Teams page](https://console.expanse.sh/teams) in the Console. An organisation can define its own roles and permission sets from **Security → Roles**. A user has one organisation role. Team membership is a separate scope and continues to use the fixed Owner, Admin, and Member team roles. ## Roles & what they unlock ### Owner role Every organisation has a protected Owner role with full authority. It cannot be edited or removed, and the organisation must retain at least one owner. ### Presets and custom roles New organisations start with editable Admin and Member presets. They can be renamed, have their permissions changed, or be removed when they are not the default and have no assigned members or pending invitations. Organisations can also create custom roles and choose which non-owner role new members receive by default. Role names are descriptive. The selected permissions determine what a member can do, including managing members, teams, security settings, billing, computes, or organisation-wide execution reads. ## Assigning roles A member who can manage organisation members may assign only roles whose permissions are a subset of their own. Only an owner or Expanse staff can grant or remove the protected Owner role. Some roles require a team placement when invited because they do not have organisation-wide execution access. The Console indicates when a team is required. If an invited email does not have an account yet, Expanse sends an invitation and adds them to the organisation with the selected role after they sign up with that email. # Authentication Source: https://docs.expanse.sh/deployment/authentication How Expanse authenticates users. Expanse authenticates users with a personal API key, created from the Console settings page after you sign in through your organisation's SSO. Your API key is the only long-lived credential you hold; compute registration hands you a single-use, short-lived install token that the installer exchanges for the compute's own credential. ## Sign in ```bash theme={"dark"} # provide your exp_user_* key with --api-key (or set EXPANSE_API_KEY): expanse login --api-key exp_user_… ``` Provide your personal `exp_user_*` API key (via `--api-key` or the `EXPANSE_API_KEY` environment variable); the CLI mints and stores a local session. The key itself is created in the Console, not by the `login` command. Confirm it worked: ```bash theme={"dark"} expanse status ``` ## Personal API key Your personal `exp_user_*` key is scoped to the teams and computes you can access. It authenticates your CLI session; behind the scenes the CLI exchanges it for short-lived signed tokens automatically, so the key itself never travels beyond sign-in. Console sessions use your SSO sign-in, not the API key. If you suspect a key is compromised, revoke it from [console.expanse.sh/settings](https://console.expanse.sh/settings) and sign in again. ## Compute credentials When a compute is registered, the daemon receives its own credential automatically: the installer exchanges the single-use install token for it and writes it straight into the daemon config. You never need to read or type it; on Kubernetes you mount the generated config file into the chart as a Secret. To replace a compute credential, re-register the compute with `expanse compute register` and run the printed install command again. ## Privacy-aware deployments Expanse runs hosted by default. Privacy-conscious enterprises can instead run the data plane and intelligence plane inside their own infrastructure, so workload telemetry, source bundles, and intelligence reasoning stay on your network; in that mode only identity (SSO, organisation membership, licence validation) goes through the Expanse control plane. SSO ties into your existing IdP (Okta, Entra, Google Workspace, or any OIDC-compatible provider). [Reach out](mailto:contact@expanse.sh) for a deployment walkthrough. # Inference right-sizer Source: https://docs.expanse.sh/deployment/inference-rightsizer Tighten serving workload resource requests at submit time so more inference jobs pack per GPU. The inference right-sizer sits on the job submit path of a Kubernetes, SLURM, or Nomad compute. Before a model-serving workload reaches the scheduler, it predicts the resources the workload actually needs with a fast analytical model (no LLM, no network call) and tightens the request to the prediction: CPU, memory, GPU count, and optionally the GPU request itself down to a MIG slice. The scheduler gets tight, correct requirements and packs more serving jobs per accelerator. It is off by default and entirely opt-in per compute. ## What gets right-sized Only workloads the right-sizer positively recognises as model serving. It reads the launch command for **vLLM**, **SGLang**, and **TGI** flags (model, tensor parallel, sequence length, concurrency, dtype), and honours explicit `expanse.sh/*` annotations (`served-model`, `model-params-billions`, `model-precision`, `max-model-len`, `max-num-seqs`, `tensor-parallel-size`, `model-kv-group`, and related keys), which override anything parsed from the command. The memory model assumes grouped-query attention; a model with full multi-head attention should state `model-kv-group: 1` so its larger KV cache is sized correctly. Annotations live in pod annotations on Kubernetes, job/group/task `meta` on Nomad, and the job `--comment` as `;`-separated `key=value` pairs on SLURM. A workload with no recognised framework and no annotation is never touched. ## Safety model In active mode the right-sizer rewrites requests with no human in the loop, so it is built around three guarantees: * **Fail-open, always.** Any timeout, predictor error, low confidence, or unrecognised request forwards the original request untouched. Every decision is bounded by a deadline (default 200ms); on expiry the request passes through. If the right-sizer itself is down, submissions are unaffected: the Kubernetes webhook registers with `failurePolicy: Ignore`, the SLURM plugin fails open on any socket error, and the Nomad proxy forwards the original body on any parse failure. * **Shadow by default.** A new deployment records what it would have rewritten while forwarding every request unchanged. You flip to active deliberately, after reviewing the shadow decisions. * **Tighten, never inflate.** A rewrite only ever reduces a stated request or limit, or fills an unstated one. It never raises a value you set, so it cannot increase cost or topology beyond what you asked for. Active mode also rewrites only at or above a minimum prediction confidence (default `medium`). A request whose memory footprint would be fully guessed is passed through, not shrunk. ## Enabling it The Expanse Helm chart's `rightsizer` block deploys a mutating admission webhook (two replicas by default) that patches recognised serving pods at creation: ```yaml theme={"dark"} rightsizer: enabled: true mode: shadow tls: certManager: issuerRef: kind: ClusterIssuer name: your-issuer ``` cert-manager issues and rotates the webhook serving certificate by default; set `rightsizer.tls.certManager.enabled: false` and provide `tls.existingSecret` plus `tls.caBundle` to bring your own. Scope the webhook to specific namespaces with `rightsizer.webhook.namespaceSelector`; empty applies it everywhere except system namespaces. The SLURM bootstrap playbook installs a sidecar on the controller and a `job_submit.lua` plugin that calls it at submit. Set the flags in your inventory group vars, then run the playbook: ```yaml theme={"dark"} expanse_rightsizer_enabled: true expanse_rightsizer_mode: shadow expanse_rightsizer_download_url: "https://…/expanse-rightsizer" ``` ```bash theme={"dark"} ansible-playbook -i inventories//hosts.ini playbooks/rightsizer.yml ``` The playbook installs the sidecar systemd unit, emits a version-matched `job_submit.lua` from the binary itself, and sets `JobSubmitPlugins=lua` (restarting `slurmctld`). If `slurm.conf` already lists other job\_submit plugins without `lua`, it stops and asks you to merge the list by hand rather than replace it. Spec extraction reads the batch script and the job comment; the job's environment is not sent to the sidecar. Nomad has no native mutating admission, so the right-sizer runs as a stateless reverse proxy in front of the Nomad agent. Install the `expanse-rightsizer.service` systemd unit (shipped under `infrastructure/install/nomad/`) and configure `/etc/expanse/rightsizer.env`: ```bash theme={"dark"} EXPANSE_COMPUTE_TYPE=nomad EXPANSE_RIGHTSIZER_ENABLED=true EXPANSE_RIGHTSIZER_MODE=shadow ``` Then repoint job submission at the proxy, for example `NOMAD_ADDR=http://127.0.0.1:8646`. The proxy right-sizes the job register and plan calls and forwards everything else to the real agent unchanged. It is one in-cluster hop and horizontally scalable. ## Rolling out: shadow first Run in shadow through an initial data-collection window. Every decision is recorded (a structured log record plus a metric increment) while the original request is forwarded unchanged, so you can see exactly what active mode would have done at zero risk. Review the decision log and `expanse_rightsizer_decisions_total`, then flip `mode` to `active` when the would-be rewrites look right for your fleet. An automated shadow-versus-actuals comparison is upcoming; today the review is the decision log. ## MIG packing With `migMultiplexing` enabled on a MIG-partitioned cluster, a single whole-GPU serving request is rewritten to the smallest MIG slice whose usable memory holds the predicted footprint, so several servers share one accelerator. The rewrite applies only to single-GPU workloads. Prerequisites differ per scheduler: * **Kubernetes:** the NVIDIA device plugin must run the `mixed` MIG strategy so slices are schedulable as `nvidia.com/mig-*` resources. The right-sizer keeps its view of the cluster's MIG geometry current by watching GPU-labelled nodes; no static inventory is needed. * **SLURM:** MIG profiles must be declared as typed gres in `gres.conf`, and the same profiles declared in `expanse_rightsizer_mig_inventory`. The sidecar cross-checks the declared inventory against `gres.conf` and drops any profile the file does not declare, so a mismatch only loses a packing opportunity. A request already stated with a GPU type (such as `gres/gpu:a100:4`) is tightened count-only, never rewritten to a MIG profile. * **Nomad:** the right-sizer tightens GPU count, CPU, and memory but never rewrites the device name. MIG device names are fingerprinted by the Nomad device plugin, and a wrong name would make the allocation unplaceable, so slice placement stays with your device plugin configuration. ## Configuration reference On Kubernetes, set these through the Helm `rightsizer` values; on SLURM, through the `expanse_rightsizer_*` playbook vars; on Nomad, in the unit's environment file. | Variable | Default | Meaning | | ----------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------- | | `EXPANSE_COMPUTE_TYPE` | required | `kubernetes`, `slurm`, or `nomad`; selects the front-end | | `EXPANSE_RIGHTSIZER_ENABLED` | `false` | Master enable | | `EXPANSE_RIGHTSIZER_MODE` | `shadow` | `shadow` or `active` | | `EXPANSE_RIGHTSIZER_PREDICTOR` | `stats` | `stats` or `hydragt` | | `EXPANSE_RIGHTSIZER_MIN_CONFIDENCE` | `medium` | Minimum confidence at which active mode rewrites (`high`, `medium`, `low`) | | `EXPANSE_RIGHTSIZER_DEADLINE` | `200ms` | Decision deadline; on expiry the request passes through (the SLURM playbook sets `150ms`) | | `EXPANSE_RIGHTSIZER_MIG_MULTIPLEXING` | `false` | Rewrite a single-GPU request to the smallest fitting MIG slice | | `EXPANSE_RIGHTSIZER_MIG_INVENTORY` | empty | Static MIG inventory for SLURM and Nomad: `modelid:hbmGiB:profile,…;…` | | `EXPANSE_RIGHTSIZER_SLURM_GRES_CONF` | `/etc/slurm/gres.conf` | gres.conf the SLURM sidecar cross-checks the inventory against | | `EXPANSE_RIGHTSIZER_MIG_PROFILE_GEOMETRY` | empty | Override the built-in MIG profile geometry table | | `EXPANSE_RIGHTSIZER_MIG_USABLE_FRACTION` | `0.90` | Fraction of a slice's memory treated as usable by the fit | | `EXPANSE_RIGHTSIZER_METRICS_ADDR` | `:9095` | Prometheus metrics listener | | `EXPANSE_RIGHTSIZER_PROXY_ADDR` | `127.0.0.1:8646` | Nomad proxy listen address | | `EXPANSE_RIGHTSIZER_NOMAD_UPSTREAM` | `http://127.0.0.1:4646` | Real Nomad agent behind the proxy | | `EXPANSE_RIGHTSIZER_SLURM_SOCKET` | `/run/expanse/rightsizer.sock` | Unix socket the SLURM plugin dials | | `EXPANSE_RIGHTSIZER_HYDRAGT_URL` | empty | Self-hosted HydraGT endpoint (egress-guarded) | | `EXPANSE_AIR_GAPPED` | unset | Refuse any non-private predictor endpoint at startup | The `stats` predictor is analytical and runs entirely locally: with it, the right-sizer reaches no network at all. The `hydragt` learned predictor is not yet calibrated; selecting it today leaves the right-sizer inert (every request passes through) until calibration ships. ## Observability Every decision, in both modes, produces: * One structured log record (`event=rightsizer_decision`) carrying the action, mode, confidence, predictor engine, served model, original and proposed resources, the chosen MIG profile, and the decision latency. In shadow mode this record is the would-be right-size. * Prometheus metrics on the metrics listener: * `expanse_rightsizer_decisions_total`, a counter labelled by `action`, `mode`, `confidence`, and `engine`. * `expanse_rightsizer_decision_latency_seconds`, a histogram of per-decision wall-clock latency against the deadline. # Introduction Source: https://docs.expanse.sh/index Expanse is the intelligence layer for HPC compute. Resource predictions, failure diagnostics, and waste visibility for the workloads your compute estate runs. ## What is Expanse? Expanse helps platform teams run HPC compute with more capacity and fewer failed jobs. It captures every workload on your computes (SLURM, Kubernetes, Nomad) without asking users to change how they submit, predicts resource needs before submission, and explains failures with a root cause and a prompt for your coding agent. Predict runtime, memory, CPU, and GPU before you submit. Right-size every workload from day one. Root cause with cited evidence for failed jobs, plus a prompt for your coding agent, grounded in your compute's own history. See exactly how much compute is wasted, per user, per team, per compute. Turn guesswork into data. ## Get started Install the CLI, register a compute, and run your first analyse in 5 minutes. How Expanse captures evidence and improves its models over time. Resource predictions and failure diagnosis. Every command and flag. How signing in works. # Install on Kubernetes Source: https://docs.expanse.sh/installation/kubernetes Connect a Kubernetes cluster to Expanse: exchange the install token from a workstation, then install the Helm chart. A Kubernetes cluster is one **compute** in Expanse. One Helm chart installs a daemon that watches pods and jobs (and the batch systems built on them: Volcano, Kueue, Argo Workflows, Flux, Ray, Flyte) and reports each run to the console: lifecycle, requested against used resources, GPU activity and, when enabled, where in the code a run spends its time. Workloads are untouched. The console's **Add compute** panel shows the short-lived install token and your control-plane and data-plane URLs. The token is prompted for during install and never goes on a command line or into Helm values. You are connecting this Kubernetes cluster to Expanse telemetry. Fetch and follow the current install guide at [https://docs.expanse.sh/installation/kubernetes](https://docs.expanse.sh/installation/kubernetes) - the guide is the source of truth; if anything I say conflicts with it, the guide wins. I will provide the values the guide's commands take: a short-lived install token, the control-plane and data-plane URLs, and the pinned daemon version, all from the Expanse console's Add compute panel. Ask me for them when you need them. Non-negotiable, regardless of what the guide says: * Supply the install token only through the EXPANSE\_INSTALL\_TOKEN environment variable when prompted. It must never appear in argv, shell history, Helm values, or a file other than the config the installer writes. * Before installing the chart, tell me which namespaces it will watch and what RBAC scope it gets, and let me confirm. Finish by confirming the daemon pod is running and the compute is active with a heartbeat in the console. ## Prerequisites * Console access in your Expanse organisation (any member can register). * A workstation with `kubectl` access to the cluster and Helm 3. * Outbound HTTPS from the daemon pod to your Expanse control and data planes. Both planes must be up before the pod starts: token exchange needs the data plane, discovery and the first heartbeat need both, and the compute stays pending until that heartbeat. ## 1. Register the compute In the console, open **Compute → Add compute**, pick **Kubernetes**, and generate an install token. ## 2. Exchange the token On your workstation, with the URLs from the Add compute panel: ```bash theme={"dark"} command -v expanse >/dev/null 2>&1 || curl -fsSL https://expanse.sh/install | sh read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN; echo export EXPANSE_INSTALL_TOKEN expanse compute install --type kubernetes \ --control-plane --data-plane unset EXPANSE_INSTALL_TOKEN ``` This writes the compute's long-lived credential to `~/.expanse/config.json`. If the install fails before the file is written, rerun it with the same token; retry stops working at the compute's first heartbeat or when the token expires. ## 3. Create the namespace and the config Secret The chart mounts an existing Secret and never creates one. Keep the Secret and the release in the same namespace; this guide uses `expanse-system`. ```bash theme={"dark"} kubectl create namespace expanse-system --dry-run=client -o yaml | kubectl apply -f - kubectl -n expanse-system create secret generic expanse-daemon-config \ --from-file=config.json="$HOME/.expanse/config.json" \ --dry-run=client -o yaml | kubectl apply -f - ``` ## 4. Install the chart Two values are required: the pinned daemon release from the Add compute panel as `image.tag`, and the Secret name. ```yaml theme={"dark"} # values.yaml image: tag: config: existingSecret: expanse-daemon-config ``` The Expanse mirror is private. Generate pull credentials in the console under **Platform → Registry**, switch to the **Kubernetes** tab, run the `kubectl create secret docker-registry` command it shows in `expanse-system`, and add the Secret to your values: ```yaml theme={"dark"} imagePullSecrets: - name: expanse-registry ``` The credential in that Secret is valid for 12 hours and the command is safe to re-run, so re-issue it before a `helm upgrade`, a node replacement or any other fresh pull. Clusters that pull unattended mirror the images and set `image.repository`; a mirror needs no expiring Secret. ```bash theme={"dark"} helm repo add expanse https://charts.expanse.sh --force-update && helm repo update helm upgrade --install expanse-daemon expanse/expanse-daemon \ -n expanse-system -f values.yaml ``` Air-gapped clusters install from a chart directory or `.tgz`, pulled with `helm pull expanse/expanse-daemon --version ` on a connected machine or taken from your release bundle mirror: ```bash theme={"dark"} helm upgrade --install expanse-daemon ./expanse-daemon \ -n expanse-system -f values.yaml ``` Images then come from your registry mirror: set `image.repository` and, if the mirror needs credentials, `imagePullSecrets` (the console's **Platform → Registry** page issues them for the Expanse mirror). Everything else has a default. `helm show values expanse/expanse-daemon` prints every value with its comment, and the [values reference](#values-reference) at the end of this page lists them all. In production set `state.persistence.enabled=true` so queued source uploads and the daemon's cached endpoints and identity survive pod replacement. ## 5. Verify The compute becomes active after its first authenticated heartbeat, normally within a minute of the pod starting. If nothing arrives, read `kubectl -n expanse-system logs deploy/expanse-daemon`. ## Upgrade Upgrade the release in place, pinning the daemon release you are moving to: ```bash theme={"dark"} helm upgrade expanse-daemon expanse/expanse-daemon \ -n expanse-system -f values.yaml --set image.tag= ``` `image.tag` is required on every install and upgrade so all pods pull one pinned version; take it from the release notes of the chart version you install. Re-issue the pull Secret from the Registry page first if the cluster pulls from the Expanse mirror; its credential lasts 12 hours. The observer restarts once. With `state.persistence.enabled=false` its queued source uploads sit on an `emptyDir` and do not survive that restart, so enable persistence before the first production upgrade. Turning `nodeSampler.cupti`, `nodeSampler.dcgm` or `nodeSampler.codeAttribution` on or off is a rolling restart of the node sampler DaemonSet, one GPU node at a time; the workloads on those nodes are not touched. The liveness probe (`probes.enabled`) is on by default and needs `image.tag` v1.5.0 or newer, which carries the `healthcheck` subcommand it runs. On an older image every probe fails and the kubelet restarts the pods, so set `probes.enabled=false` there. ## Optional: GPU metrics The default install observes workload lifecycle only. On GPU clusters, enable the node sampler DaemonSet: it ships node and per-GPU metrics and attributes each GPU to the pod holding it, a path the chart runs as root on the node. ```yaml theme={"dark"} nodeSampler: enabled: true # Only where the NVIDIA runtime is a RuntimeClass (k3s, RKE2 with the toolkit). # A node without that runtime cannot start the pod, so pair it with a selector. runtimeClassName: nvidia nodeSelector: nvidia.com/gpu.present: "true" ``` The sampler runs on every node by default. CPU-only nodes still get host and per-pod usage and Python stacks; they report no GPU chips, so the GPU health summary counts GPU nodes only. Set `nodeSelector` to save the pods on large CPU pools or when a RuntimeClass is required. The sampler reads the host `/proc` and `/sys`, so its namespace must allow privileged pods: ```bash theme={"dark"} kubectl label ns expanse-system pod-security.kubernetes.io/enforce=privileged --overwrite ``` Set `nodeSampler.podResources.enabled=false` to keep the sampler non-root and accept node-scope GPU metrics without per-pod attribution or the per-pod CPU and memory series. ## Optional: GPU profiling (CUPTI and DCGM) Both live in the node sampler and are off by default. Nothing is installed on the node, and Expanse never runs a DCGM host engine of its own. ```yaml theme={"dark"} nodeSampler: # GPU performance counters. Adds the compute driver capability and CAP_SYS_ADMIN; # needs an NVIDIA driver of the r580 branch or newer. cupti: enabled: true # DCGM from the host engine you already run: the GPU Operator's nvidia-dcgm, or a # host nvidia-dcgm.service bound to the node IP. dcgm: hostEngine: "$(NODE_IP):5555" ``` Without a reachable engine the DCGM chip on the compute page reads disabled with the reason and the standard GPU metrics keep flowing. To keep CAP\_SYS\_ADMIN off the sampler, set `nodeSampler.cupti.sysAdmin=false` and allow GPU profiling for all users on the nodes (`NVreg_RestrictProfilingToAdminUsers=0`). ## Optional: Python stack attribution `nodeSampler.codeAttribution.enabled=true` samples the Python processes of each node's pods, so an execution page shows where in the code a run spends its time. Python only; compiled workloads get no stacks. The sampler then needs the host PID namespace, `CAP_SYS_PTRACE`, root and read-only `pods` access, which the chart applies. System namespaces and the release's own pods are never sampled, and `watch.namespaces` scopes it like the observer. GPU profiling and stack attribution need daemon v1.5.0 or newer; pin `image.tag` to a release the chart's `appVersion` matches or exceeds. ## Optional: namespace-scoped install Set `watch.namespaces` and `rbac.clusterWide=false`: the chart grants namespace permissions only where it watches and renders nothing cluster-scoped. The queue view then reports degraded, because queue and capacity snapshots need one cluster-scoped read, `list nodes`; grant it with `rbac.listNodes=true` when an administrator allows. Per-pod GPU attribution and stack attribution also read pods cluster-wide; leave both off for a strictly namespace-scoped install. ## Optional: network policy The daemon dials only your control and data planes, the API server and, when configured, the DCGM host engine. Turn on the chart's NetworkPolicy and narrow the plane addresses: ```yaml theme={"dark"} networkPolicy: enabled: true egress: planes: - cidr: port: 443 - cidr: port: 443 ``` On EKS with Pod Identity add `networkPolicy.extraEgress` for `169.254.170.23/32` on TCP 80. ## Optional: pin the image digest Set `image.digest` to the `sha256:...` from the release manifest alongside `image.tag`; a re-pushed tag can then never change what the cluster runs. ## Optional: OpenShift ```bash theme={"dark"} oc adm policy add-scc-to-user restricted-v2 -z expanse-daemon -n expanse-system oc adm policy add-scc-to-user privileged -z expanse-daemon-node-sampler -n expanse-system ``` Set `podSecurityContext: null` so the SCC's uid range applies to the observer. The second line is needed only with the node sampler. ## Values reference Every chart value with its default. | Value | Default | What it does | | ------------------------------------------------------ | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `replicaCount` | `1` | Must stay 1 until the Kubernetes watcher has leader election. | | `image.repository` | `742550615465.dkr.ecr.eu-west-2.amazonaws.com/expanse-daemon` | Expanse release registry with signed multi-arch images per daemon release. Point at your registry mirror on air-gapped clusters. | | `image.tag` | `unset` | Empty on purpose: every install passes an explicit daemon-v\* semver. A "latest" default would let a pod restart drift off the cluster's version. | | `image.digest` | `unset` | Optional image digest (sha256:...) from the release manifest. Set it so a re-pushed tag can never change what the cluster runs. | | `image.pullPolicy` | `IfNotPresent` | Standard Kubernetes field, passed through. | | `imagePullSecrets` | `unset` | Pull secrets for image.repository when your registry mirror needs credentials (the console Platform > Registry page issues them for the Expanse mirror). | | `nameOverride` | `unset` | Replaces the chart name in resource names. | | `fullnameOverride` | `unset` | Replaces the whole release-qualified resource name. | | `serviceAccount.create` | `true` | Create the observer ServiceAccount; false reuses serviceAccount.name. | | `serviceAccount.annotations` | `unset` | Observer SA annotations for a cloud identity (IRSA role ARN, Workload Identity). | | `serviceAccount.name` | `unset` | Observer ServiceAccount name; empty derives it from the release name. | | `serviceAccount.automountToken` | `true` | Mount the observer's API token; the watcher needs it to list and watch pods. | | `rbac.create` | `true` | Create the watcher's (Cluster)Role and binding; false expects them to exist. | | `rbac.clusterWide` | `true` | true watches every namespace with cluster-wide list/watch permissions. false creates Role/RoleBinding objects only for watch.namespaces. | | `rbac.listNodes` | `null` | Cluster-scoped `list nodes` for queue and topology snapshots. Unset follows rbac.clusterWide; scoped installs set true to un-degrade the queue view. | | `config.existingSecret` | `expanse-daemon-config` | Installer-created Secret containing the exchanged daemon config. | | `config.key` | `config.json` | Secret key that holds the exchanged config. | | `config.mountPath` | `/expanse-home/.expanse` | Directory the config Secret is projected into; the daemon reads HOME/.expanse. | | `config.fileName` | `config.json` | File name of the projected config inside mountPath. | | `config.defaultMode` | `416` | 0640 (decimal 416): fsGroup below makes the root-owned Secret file readable by the nonroot daemon; the daemon refuses anything broader than 0640. | | `home` | `/expanse-home` | HOME of the daemon process; the config mount lives under it. | | `state.mountPath` | `/expanse-state` | Writable state mount: queued source uploads, caches and health markers. | | `state.persistence.enabled` | `false` | emptyDir suits KIND and local dev. Enable in production so queued source uploads survive Pod replacement and node drain. | | `state.persistence.existingClaim` | `unset` | Bring your own PersistentVolumeClaim instead of the chart-created one. | | `state.persistence.storageClassName` | `unset` | StorageClass for the chart-created claim; empty uses the cluster default. | | `state.persistence.size` | `1Gi` | Size of the chart-created claim; queued source uploads are small. | | `state.persistence.annotations` | `unset` | Standard Kubernetes field, passed through. | | `state.persistence.labels` | `unset` | Standard Kubernetes field, passed through. | | `watch.namespaces` | `unset` | Empty means all namespaces. Set a namespace list for scoped installs. | | `podAnnotations` | `unset` | Standard Kubernetes field, passed through. | | `podLabels` | `unset` | Standard Kubernetes field, passed through. | | **`podSecurityContext`** | | The release image runs as uid 65532; without fsGroup the projected Secret is root-owned 0600 and the observer exits at "load config". | | `podSecurityContext.fsGroup` | `65532` | Only the group: the uid comes from the image. Under an OpenShift SCC or a mustRunAsRange policy set podSecurityContext=null and let the range apply. | | `podSecurityContext.fsGroupChangePolicy` | `OnRootMismatch` | Skip the kubelet's ownership pass once the state PVC root carries the group; the pass ORs group rw onto files and the daemon's 0600 config cache must stay tight. | | `podSecurityContext.runAsNonRoot` | `true` | Standard Kubernetes field, passed through. | | `podSecurityContext.seccompProfile.type` | `RuntimeDefault` | Standard Kubernetes field, passed through. | | **`securityContext`** | | The observer writes only under the state mount, so it satisfies the Pod Security restricted profile at defaults. | | `securityContext.allowPrivilegeEscalation` | `false` | Standard Kubernetes field, passed through. | | `securityContext.readOnlyRootFilesystem` | `true` | Standard Kubernetes field, passed through. | | `securityContext.capabilities.drop` | `["ALL"]` | Standard Kubernetes field, passed through. | | **`resources`** | | Observer memory scales with the pod informer cache; 1Gi covers a few thousand pods. Raise on very large clusters. No cpu limit, as in the umbrella chart. | | `resources.requests.cpu` | `100m` | Standard Kubernetes field, passed through. | | `resources.requests.memory` | `256Mi` | Standard Kubernetes field, passed through. | | `resources.limits.memory` | `1Gi` | Standard Kubernetes field, passed through. | | `nodeSelector` | `unset` | Standard Kubernetes field, passed through. | | `tolerations` | `unset` | Standard Kubernetes field, passed through. | | `affinity` | `unset` | Standard Kubernetes field, passed through. | | `priorityClassName` | `unset` | Standard Kubernetes field, passed through. | | `runtimeClassName` | `unset` | Standard Kubernetes field, passed through. | | `extraEnv` | `unset` | Extra env for both workloads, e.g. SSL\_CERT\_FILE for a corporate CA bundle. | | `extraVolumes` | `unset` | Appended after the chart's fixed volumes/mounts in both workloads. Intended for mounting a corporate CA bundle, paired with SSL\_CERT\_FILE via extraEnv. | | `extraVolumeMounts` | `unset` | Standard Kubernetes field, passed through. | | **`controller`** | | Gates the cluster-wide workload-observer Deployment. Disable for a GPU-sampler-only install on a cluster whose workloads are observed elsewhere. | | `controller.enabled` | `true` | Render the cluster-wide observer Deployment. | | **`probes`** | | Exec probes on both workloads via `expanse-daemon healthcheck`, which reads marker files the heartbeat loop touches. | | `probes.enabled` | `true` | Needs image.tag v1.5.0 or newer (the healthcheck subcommand); on an older image every probe fails and the kubelet restart-loops the pods, so set false. | | **`probes.liveness`** | | Liveness follows the loop marker, not heartbeat success, so a plane outage never restarts pods. The heartbeat interval clamps at 5m; stay above it. | | `probes.liveness.maxAge` | `20m` | Go duration, e.g. 20m; oldest acceptable loop marker, passed as healthcheck --max-age. | | **`probes.readiness`** | | Readiness follows the last accepted heartbeat. Off by default so DaemonSet rollouts stay independent of plane availability; on, helm --wait proves a heartbeat. | | `probes.readiness.enabled` | `false` | Add a readiness probe on the heartbeat marker; needs probes.enabled. | | `probes.readiness.maxAge` | `15m` | Go duration, e.g. 15m; oldest acceptable accepted-heartbeat marker. | | **`networkPolicy`** | | Opt-in NetworkPolicy per workload: ingress denied, egress limited to DNS, the planes, the API server and (sampler) the DCGM host engine. Off: many CNIs do not enforce. | | `networkPolicy.enabled` | `false` | Render one NetworkPolicy per workload. | | `networkPolicy.dns.kubeSystemOnly` | `false` | Restrict port 53 to kube-system instead of any destination. | | **`networkPolicy.egress.planes`** | | Control and data plane destinations; narrow the CIDRs to your plane addresses. | | **`networkPolicy.egress.apiServer`** | | API server endpoint. kind and k3s serve 6443, managed clusters mostly 443. | | `networkPolicy.egress.dcgmPort` | `5555` | DCGM host engine port and the node ranges it is reached on, used only when nodeSampler.dcgm.hostEngine is set. | | `networkPolicy.extraEgress` | `unset` | Extra NetworkPolicyEgressRule items appended to both policies, e.g. EKS Pod Identity: \[\{to: \[\{ipBlock: \{cidr: 169.254.170.23/32}}], ports: \[\{protocol: TCP, port: 80}]}]. | | **`nodeSampler`** | | Per-node GPU metric sampler DaemonSet. The observer Deployment does NOT sample GPUs. | | `nodeSampler.enabled` | `false` | Render the per-node GPU sampler DaemonSet. | | **`nodeSampler.serviceAccount`** | | Dedicated SA bound to NO RBAC. On serviceAccount.create=false you must name an SA that also carries no RBAC; the render fails loudly if it is unset. | | `nodeSampler.serviceAccount.name` | `unset` | Sampler ServiceAccount name; empty derives it from the release name. | | `nodeSampler.serviceAccount.annotations` | `unset` | Node-sampler SA only, never inherited from serviceAccount.annotations, so the sampler cannot pick up the observer's IRSA or Workload Identity role. | | **`nodeSampler.podSecurityContext`** | | fsGroup lets the nonroot daemon read the root-owned config Secret at 0640. podResources.enabled=true overrides this to root; see the install doc. | | `nodeSampler.podSecurityContext.runAsUser` | `65532` | Standard Kubernetes field, passed through. | | `nodeSampler.podSecurityContext.runAsGroup` | `65532` | Standard Kubernetes field, passed through. | | `nodeSampler.podSecurityContext.fsGroup` | `65532` | Standard Kubernetes field, passed through. | | `nodeSampler.podSecurityContext.runAsNonRoot` | `true` | Standard Kubernetes field, passed through. | | `nodeSampler.podSecurityContext.seccompProfile.type` | `RuntimeDefault` | Standard Kubernetes field, passed through. | | **`nodeSampler.securityContext`** | | Node-sampler container only, never inherited. The chart adds SYS\_PTRACE and SYS\_ADMIN on top of drop ALL only when the matching feature is on. | | `nodeSampler.securityContext.allowPrivilegeEscalation` | `false` | Standard Kubernetes field, passed through. | | `nodeSampler.securityContext.readOnlyRootFilesystem` | `false` | false until verified on a GPU node with the release image: the NVIDIA hook bind-mounts driver libraries into the rootfs. | | `nodeSampler.securityContext.capabilities.drop` | `["ALL"]` | Standard Kubernetes field, passed through. | | `nodeSampler.securityContextOverridden` | `false` | Set true to keep full control of nodeSampler.podSecurityContext instead of the chart forcing root for podResources, codeAttribution or cupti.sysAdmin. | | `nodeSampler.config.defaultMode` | `416` | Projected config Secret mode for the node sampler. 0640 (decimal 416) so the fsGroup can read it; the daemon refuses anything broader than 0640. | | **`nodeSampler.podResources`** | | Per-pod GPU attribution via the kubelet pod-resources socket plus read-only pods access. Disabled ships node-scope metrics only, with no RBAC or mount. | | `nodeSampler.podResources.enabled` | `true` | Mount the kubelet pod-resources socket and grant read-only pods access. | | `nodeSampler.podResources.hostPath` | `/var/lib/kubelet/pod-resources` | Host directory containing kubelet.sock. Override only on clusters that relocate the kubelet root; k3s uses this stock path. | | **`nodeSampler.nvidia`** | | All GPUs visible read-only, so the sampler sees every GPU on the node without taking a schedulable GPU slot. | | `nodeSampler.nvidia.visibleDevices` | `"all"` | NVIDIA\_VISIBLE\_DEVICES / NVIDIA\_DRIVER\_CAPABILITIES for the sampler container: all GPUs, utility libraries only (nvidia-smi, no CUDA). | | `nodeSampler.nvidia.driverCapabilities` | `"utility"` | NVIDIA\_DRIVER\_CAPABILITIES; cupti.enabled appends compute when missing. | | **`nodeSampler.cupti`** | | GPU performance-counter sampling (CUPTI) from the sampler pod; needs an r580 or newer NVIDIA driver. | | `nodeSampler.cupti.enabled` | `false` | Starts CUPTI sampling and adds the compute driver capability. Needs a daemon image v1.5.0 or newer. | | `nodeSampler.cupti.sysAdmin` | `true` | CAP\_SYS\_ADMIN lets the sampler read GPU performance counters on nodes that keep the driver default RmProfilingAdminOnly=1. Needs the root pod. | | **`nodeSampler.profiling`** | | Adaptive profiling on each GPU node: rule engine plus Phase A CUPTI widening when cupti.enabled is on. false pins Light-only sampling. | | `nodeSampler.profiling.enabled` | `true` | Sets EXPANSE\_PROFILING\_ENABLE; false pins Light-only sampling. | | **`nodeSampler.dcgm`** | | DCGM enrichment from a host engine you already run, passed as dcgmi --host. "\$(NODE\_IP):5555" reaches a GPU Operator nvidia-dcgm or a host nvidia-dcgm.service. | | `nodeSampler.dcgm.hostEngine` | `unset` | Host engine address (dcgmi --host); empty leaves the DCGM chip disabled. Needs daemon v1.5.0 or newer. | | **`nodeSampler.codeAttribution`** | | Python stack attribution for the pods on this node. Needs hostPID, SYS\_PTRACE, root and read-only pod access, so off by default. | | `nodeSampler.codeAttribution.enabled` | `false` | Needs daemon v1.5.0 or newer. | | `nodeSampler.runtimeClassName` | `unset` | Set to "nvidia" only if the cluster needs an explicit RuntimeClass. Most GPU AMIs already default the container runtime to nvidia. | | `nodeSampler.nodeSelector` | `unset` | Restrict to GPU nodes, e.g. \{"nvidia.com/gpu.present": "true"}. Empty runs everywhere: CPU-only nodes ship host and per-pod usage without GPU chips. Required when runtimeClassName is set. | | **`nodeSampler.tolerations`** | | Tolerates the conventional GPU node taint (eksctl and the GPU operator both apply nvidia.com/gpu=present:NoSchedule). | | `nodeSampler.affinity` | `unset` | Node-sampler placement and pod metadata, passed through unchanged. | | **`nodeSampler.resources`** | | The CUPTI helper holds \~165MB per node on the bundled cuda1300 build plus a \~16MiB ring per GPU; 2Gi covers 8 GPUs. Raise for denser nodes. | | `nodeSampler.resources.requests.cpu` | `100m` | Standard Kubernetes field, passed through. | | `nodeSampler.resources.requests.memory` | `256Mi` | Standard Kubernetes field, passed through. | | `nodeSampler.resources.limits.memory` | `2Gi` | Standard Kubernetes field, passed through. | | `nodeSampler.priorityClassName` | `unset` | Pre-existing PriorityClass so the sampler survives node pressure. The chart creates none: PriorityClass is cluster-scoped and scoped installs avoid those. | | `nodeSampler.podAnnotations` | `unset` | Standard Kubernetes field, passed through. | | `nodeSampler.podLabels` | `unset` | Standard Kubernetes field, passed through. | # Install on Nomad Source: https://docs.expanse.sh/installation/nomad Connect a Nomad cluster to Expanse with the one-line systemd installer on a client node. A Nomad client node runs one Expanse daemon that watches allocations through the local Nomad HTTP API and reports task lifecycle telemetry. Jobs are untouched. The console's **Add compute** panel and [`expanse compute register`](/cli/overview#expanse-compute-register) show the short-lived install token plus your deployment's control-plane and data-plane URLs, which this guide's installer takes. The token is prompted for during install; it never goes on a command line. You are installing Expanse telemetry on this Nomad client node. Fetch and follow the current install guide at [https://docs.expanse.sh/installation/nomad](https://docs.expanse.sh/installation/nomad) - the guide is the source of truth; if anything I say conflicts with it, the guide wins. I will provide the values the guide's installer takes: a short-lived install token and the control-plane and data-plane URLs, from the Expanse console's Add compute panel. Ask me for them when you need them, and check with me whether the local Nomad agent is on a non-default address or has ACLs enabled before you run the installer. Non-negotiable, regardless of what the guide says: * Supply the install token only through the EXPANSE\_INSTALL\_TOKEN environment variable when prompted. It must never appear in argv, shell history, or a systemd unit. Finish by confirming the expanse-daemon service is running and the compute is active with a heartbeat in the console. ## Prerequisites * A Nomad client node with `systemd`, `curl`, `tar`, and `sha256sum` or `shasum`, plus `sudo`. * The daemon talks to the local Nomad agent; the installer defaults `EXPANSE_NOMAD_ADDR` to `http://127.0.0.1:4646`. Override it (and set `EXPANSE_NOMAD_TOKEN` for ACL-enabled clusters) before install if needed. * The control plane and data plane must be running and reachable from the client node before the daemon starts. Token exchange needs the data plane; endpoint discovery and the first heartbeat need both planes. A DNS, TCP, or generic HTTP response is not enough. The compute stays pending until its first authenticated heartbeat. ## 1. Register the compute In the console, open **Compute → Add compute**, pick **Nomad**, and generate an install token. ## 2. Run the installer on the client node On the client node, using the control-plane and data-plane URLs from the Add compute panel: ```bash theme={"dark"} read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN; echo export EXPANSE_INSTALL_TOKEN export EXPANSE_CONTROL_PLANE_URL= EXPANSE_DATA_PLANE_URL= curl -fsSL https://releases.expanse.sh/infrastructure/install/nomad/install.sh | sudo -E bash unset EXPANSE_INSTALL_TOKEN ``` Air-gapped deployment? Run the same script from your release bundle mirror host instead of `releases.expanse.sh`. The installer downloads and verifies the daemon binary, exchanges the token for the compute's long-lived credential, writes `/etc/expanse/config.json`, installs the `expanse-daemon` systemd unit, starts it, and waits for the first heartbeat (default 60 seconds; set `EXPANSE_HEARTBEAT_TIMEOUT_SECONDS` to adjust). ## 3. Verify ```bash theme={"dark"} systemctl status expanse-daemon journalctl -u expanse-daemon -n 100 --no-pager ``` The compute appears active in the console within one heartbeat interval. Allocation telemetry follows as the daemon observes running allocations. For what gets captured, see [Nomad integration](/integrations/nomad). # Install on AWS ParallelCluster Source: https://docs.expanse.sh/installation/parallelcluster Connect an AWS ParallelCluster to Expanse. The install detects ParallelCluster and picks the shared prefix for you. AWS ParallelCluster is SLURM underneath, so the flow is the [SLURM install](/installation/slurm) with one difference: the install detects ParallelCluster and defaults the shared prefix to `/opt/parallelcluster/shared/expanse`, so you never pass `--shared-prefix`. Before starting the daemon, the control plane and data plane must be running and reachable from the head node. Token exchange needs the data plane; endpoint discovery and the first heartbeat need both planes. A DNS, TCP, or generic HTTP response is not enough. The compute stays pending until its first authenticated heartbeat. Start in the console's **Add compute** panel (or with [`expanse compute register`](/cli/overview#expanse-compute-register)): it shows the short-lived install token plus your deployment's control-plane and data-plane URLs and the pinned daemon version, which this guide's commands take. The token is prompted for during install; it never goes on a command line. You are installing Expanse telemetry on the head node of this AWS ParallelCluster. Fetch and follow the current install guide at [https://docs.expanse.sh/installation/parallelcluster](https://docs.expanse.sh/installation/parallelcluster) - the guide is the source of truth; if anything I say conflicts with it, the guide wins. I will provide the values the guide's commands take: a short-lived install token, the control-plane and data-plane URLs, and the pinned daemon version, all from the Expanse console's Add compute panel. Ask me for them when you need them. Non-negotiable, regardless of what the guide says: * Supply the install token only through the EXPANSE\_INSTALL\_TOKEN environment variable when prompted. It must never appear in argv, shell history, or a file. * Show me the installer's preflight plan, and stop and report before the token is exchanged if preflight fails. * Do not hand-patch around a failed install; it rolls back and is safe to rerun once the cause is fixed. Finish by confirming the compute is active with a heartbeat in the console and the compute-node samplers are installed. ## 1. Register the compute In the console, open **Compute → Add compute**, pick **SLURM**, and generate an install token. Keep the panel open: it polls registration state live and shows the endpoint URLs used below. ## 2. Install the CLI on the head node SSH to the cluster's head node (it runs `slurmctld`) and install the CLI if it is missing: ```bash theme={"dark"} command -v expanse >/dev/null 2>&1 || curl -fsSL https://expanse.sh/install | sh ``` ## 3. Install the controller daemon on the head node Using the control-plane and data-plane URLs and the daemon version from the Add compute panel: ```bash theme={"dark"} read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN; echo export EXPANSE_INSTALL_TOKEN expanse compute install --type slurm --daemon-version \ --control-plane --data-plane \ --patch-slurm-conf=true unset EXPANSE_INSTALL_TOKEN ``` `--daemon-version` downloads the daemon release and verifies its signed manifest, checksum, and cosign signature before installing. The preflight recognises ParallelCluster, resolves the shared prefix to `/opt/parallelcluster/shared/expanse`, and plans hook wiring against ParallelCluster's managed SLURM config using drop-in directories where they exist. Preflight runs **before** the token is exchanged, so a failure never burns the short-lived token, and a failed install rolls itself back. ## 4. Samplers on compute nodes Each compute node runs a credential-less sampler: ```bash theme={"dark"} expanse compute install --type slurm --role sampler ``` Static nodes: run it once per node. Dynamically scaled queues: run it at node boot, for example from an `OnNodeConfigured` custom bootstrap action or a custom AMI, so freshly launched instances join automatically. The binary and spool live on the shared ParallelCluster mount, so the per-node step only installs and enables the sampler systemd unit. ## 5. Verify The compute becomes active after its first authenticated heartbeat. Job capture needs no change to how anyone submits. # Install on SLURM Source: https://docs.expanse.sh/installation/slurm Connect a SLURM cluster to Expanse, configure optional GPU profiling, and manage upgrades and worker samplers. A SLURM cluster is one **compute** in Expanse. One daemon runs on the controller (`slurmctld` host) and reports workload lifecycle, requested against used resources, GPU activity and source evidence. Workers run samplers without compute credentials and send samples through a shared filesystem. Only the controller connects to the control and data planes. The console's **Add compute** panel shows your short-lived install token, control-plane and data-plane URLs, and pinned daemon version. Install the controller on the actual `slurmctld` host, which may differ from the login node. For AWS ParallelCluster, use the [ParallelCluster guide](/installation/parallelcluster). You are connecting this SLURM cluster to Expanse telemetry. Fetch and follow [https://docs.expanse.sh/installation/slurm](https://docs.expanse.sh/installation/slurm). I will provide the intended organisation, install token, plane URLs and pinned daemon version from Add compute. Confirm the controller host, worker nodes and shared mount before installing. Use sudo for system paths. Supply the token only through hidden input or EXPANSE\_INSTALL\_TOKEN; never put its literal value in a command, shell history or file. If an existing daemon config is found, report its compute ID and stop before changing identity. Do not use --reinstall to enrol into a different compute. Review the preflight output; do not exchange a token after a failed preflight. Treat CAP\_SYS\_ADMIN for CUPTI as an explicit operator choice. Finish by verifying the intended compute ID and organisation, a fresh heartbeat, GPU samples on each GPU node, and the health of any optional collectors enabled. ## Prerequisites For the fullest GPU diagnostics, configure both [CUPTI](#optional-gpu-performance-counters-with-cupti) and [DCGM](#optional-dcgm-enrichment): CUPTI supplies detailed performance counters, while DCGM adds supported health, reliability and interconnect telemetry. Both are recommended on GPU nodes and remain optional; baseline GPU inventory and utilisation work with `nvidia-smi` alone. * Console access with permission to register a compute in the intended organisation. * Root or sudo on the controller and workers; systemd for managed services. * SLURM running, with `scontrol`, `squeue` and preferably `sacct` accessible. * The `slurm` OS account on every node, with consistent UID/GID across the shared mount. Use `--user` if your daemon account has another name. * A filesystem mounted at the same path on every node, such as `/srv/expanse` on NFS or Lustre. Single-node clusters can use local disk. * Working NVIDIA drivers and `nvidia-smi` on GPU nodes. Baseline metrics need neither a CUDA toolkit nor CUPTI. * Controller HTTPS access to your control and data planes and, for online installs, `releases.expanse.sh` or your release mirror. Workers can fetch signed CUPTI releases during bootstrap and upload trace segments to your deployment's object store. For workers without outbound access, stage CUPTI archives on the shared mount and use `EXPANSE_SEGMENT_UPLOAD_MODE=relay` as described under [restricted networks](#optional-restricted-networks-and-air-gapped-installs). Both planes must be operational before installation. An HTTP response proves reachability; successful authentication and the first heartbeat prove enrolment. ## 1. Register the compute Select the intended organisation in the console, open **Compute → Add compute**, choose **SLURM**, and generate an install token. Keep the panel open to check that this exact compute becomes active. ## 2. Install the CLI on the controller ```bash theme={"dark"} command -v expanse >/dev/null 2>&1 || curl -fsSL https://expanse.sh/install | sh ``` Check `expanse compute install --help` against the [flag reference](#install-flag-reference) when using an older CLI. CLI and daemon releases are versioned separately. ## 3. Install the controller daemon Use the version and endpoints from Add compute. This Bash example keeps the token out of shell history and explicitly passes it through sudo: ```bash theme={"dark"} ( set -e read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN echo sudo EXPANSE_INSTALL_TOKEN="$EXPANSE_INSTALL_TOKEN" \ expanse compute install --type slurm \ --daemon-version \ --shared-prefix /srv/expanse \ --control-plane \ --data-plane \ --patch-slurm-conf=true ) ``` Alternatively, run the sudo install command without the environment assignment and enter the token at its hidden prompt. An inherited `EXPANSE_INSTALL_TOKEN` takes precedence over that prompt. The installer verifies the release's signed manifest, archive checksum and signature. It preflights paths, permissions, SLURM config and hook changes before exchanging the token. It installs the daemon, bundled py-spy, hooks and systemd unit, then waits for an authenticated heartbeat. A fresh install rolls back its managed changes on failure. `--patch-slurm-conf=true` authorises the planned Prolog, Epilog and TaskProlog changes. Without it, SLURM-owned paths remain read-only; your configuration management must apply the hook wiring. Foreign hooks may require explicit chaining. Use `--slurm-conf /absolute/path/slurm.conf` if discovery fails. The compute credential stays on the controller at `/var/lib/slurm/.expanse/config.json`, or your `--config` override. The unit pins that path through `EXPANSE_DAEMON_CONFIG`. Keep this file off the shared mount and do not edit its compute ID to change registration. If installation reports an existing daemon config, decide whether you want an [upgrade](#upgrade-the-existing-compute) or a [new registration](#reset-and-register-a-different-compute). `--reinstall` preserves the old compute identity and skips install-token intake, including an exported token. A token typed into a previous failed invocation is not carried into the next invocation. ## 4. Start worker samplers By default, the installed Prolog starts or updates a worker's sampler when a job starts. Autoscaled workers need the shared mount and the daemon OS account; no per-worker token is used. For static workers that need telemetry before their first job, install the CLI on each worker and run: ```bash theme={"dark"} sudo expanse compute install --type slurm --role sampler \ --shared-prefix /srv/expanse ``` The worker uses the daemon binary already installed on the shared mount. Optional node-local settings and capability drop-ins must be provisioned on workers separately, including newly autoscaled nodes. Skip the sampler on the controller, including single-node clusters. Its daemon already samples local GPUs. Running both roles on that node duplicates sampling. ## 5. Verify registration and telemetry Check the services on their respective hosts: ```bash theme={"dark"} # Controller sudo systemctl status expanse-daemon --no-pager sudo journalctl -u expanse-daemon --since "5 minutes ago" --no-pager # Each worker sudo systemctl status expanse-sampler --no-pager sudo journalctl -u expanse-sampler --since "5 minutes ago" --no-pager nvidia-smi --query-gpu=name,uuid,memory.total --format=csv ``` On a worker, `test -f /srv/expanse/.expanse-preflight-sentinel && echo shared` checks visibility of the controller's preflight sentinel. Confirm it is the same shared mount, not an independently created local directory. In the console, check the intended organisation and compute ID, a fresh heartbeat, and the expected nodes and GPUs. Submit a job with `sbatch` and check its execution and metrics. An active heartbeat alone does not prove optional CUPTI, DCGM or Python stack collection is working; inspect the daemon diagnostics too. Idle GPUs can legitimately show zero utilisation. ## Upgrade the existing compute Reinstall with a new pinned daemon version and the same prefix/configuration: ```bash theme={"dark"} sudo expanse compute install --type slurm --reinstall \ --daemon-version \ --shared-prefix /srv/expanse \ --patch-slurm-conf=true ``` This reuses the existing compute credential and saved plane endpoints. No new install token is needed or read. Repeat any non-default install flags, such as `--user`, `--config`, `--slurm-conf` or `--profiling-agent=false`. Workers converge to the new shared binary at their next job start. Re-run the sampler install on static workers to update immediately. Reinstall cleans the old managed installation before installing the replacement; if replacement fails, the identity remains but the old runtime is not automatically restored. ## Reset and register a different compute Uninstall is the explicit identity reset. It stops collection, removes managed files and hooks, and deprovisions the old compute. Run this in a maintenance window after draining affected jobs so their hooks and queued captures are not removed mid-run. Keep the same `--user`, `--config` and prefix overrides as the original installation. ```bash theme={"dark"} # Each worker, before removing the shared controller artefacts sudo expanse compute uninstall --type slurm --role sampler \ --shared-prefix /srv/expanse # Controller sudo expanse compute uninstall --type slurm --role controller \ --shared-prefix /srv/expanse --slurm-conf /etc/slurm/slurm.conf ``` If a sampler was also installed on the controller, uninstall that role there first. Stop on cleanup errors. Do not use `--keep-identity` for a registration reset, and do not delete config or lock files by hand. Review any operator-owned systemd drop-ins and `/etc/expanse/daemon.env` before reusing the host; uninstall is not a reset of your own configuration management. Return to step 1 in the intended organisation and perform a fresh install without `--reinstall`. `--force` is an alias for `--reinstall`, so it also preserves identity. ## Optional: GPU performance counters with CUPTI Baseline GPU metrics use `nvidia-smi`. CUPTI adds SM activity, tensor activity, DRAM throughput and other hardware performance counters. SLURM attempts CUPTI collection by default; its availability depends on the helper, driver and profiling permissions. Online controller installation selects a driver-compatible CUPTI variant from the signed release manifest and installs its helper, capture agent and matched libraries under `/lib/cupti-nodes//`. Worker bootstrap selects for each worker's driver independently. The released matrix includes CUDA 12.8 and 13.0 for amd64; select only a variant compatible with that node's driver. Missing CUPTI does not block baseline telemetry or registration. ### Allow restricted profiling counters If the NVIDIA driver restricts profiling to administrators, the service needs `CAP_SYS_ADMIN`. Generated SLURM units do not grant it automatically. This is a broad Linux capability; opt in only on GPU nodes where CUPTI is wanted. On a worker, run the following after installing its sampler. On a controller that also runs GPU jobs, set `service=expanse-daemon` instead: ```bash theme={"dark"} service=expanse-sampler sudo mkdir -p "/etc/systemd/system/$service.service.d" sudo tee "/etc/systemd/system/$service.service.d/profiling-permissions.conf" >/dev/null <<'UNIT' [Service] CapabilityBoundingSet=CAP_SYS_ADMIN AmbientCapabilities=CAP_SYS_ADMIN UNIT sudo systemctl daemon-reload sudo systemctl restart "$service" sudo systemctl show "$service" -p User -p CapabilityBoundingSet -p AmbientCapabilities sudo journalctl -u "$service" --since "2 minutes ago" --no-pager ``` These assignments add to the unit's existing capability sets. Provision the drop-in through your worker image or configuration management for autoscaled nodes. A successful manual probe as root does not prove the service running as `slurm` has permission; verify the running service's diagnostics. To disable CUPTI collection, set `EXPANSE_CUPTI_ENABLE=0` in `/etc/expanse/daemon.env` on the relevant nodes and restart their Expanse service. The installer never changes the NVIDIA driver's profiling policy. ## Optional: DCGM enrichment Expanse uses a DCGM host engine you already operate. It does not install DCGM or start `nv-hostengine`. Install your supported `dcgmi` client on GPU nodes and make the existing engine reachable, by default at `localhost:5555`. For a different engine or client location, add these runtime settings: ```ini theme={"dark"} # /etc/expanse/daemon.env EXPANSE_DCGM_HOST=127.0.0.1:5555 EXPANSE_DCGMI=/usr/bin/dcgmi ``` Restart the sampler on each affected worker, or the daemon on a GPU controller. Unsupported counters remain absent; baseline GPU metrics continue if DCGM is unavailable. On pre-Hopper GPUs, external DCGM profiling counter collection can contend with CUPTI for the performance monitor. Coordinate that collection with your existing monitoring stack; Expanse suppresses its own DCGM profiling groups while CUPTI owns the counters, while continuing compatible DCGM health collection. ## Optional: Python stacks and deep capture The daemon release bundles py-spy, which installs to `/bin/py-spy` by default. You do not need to download it separately. Python stack attribution needs ptrace access to workload processes; the generated units grant `CAP_SYS_PTRACE`, but site security policy can still prevent collection. `--bundled-pyspy=false` skips the bundled copy; an operator-managed py-spy can still be selected through `EXPANSE_PYSPY`. `--profiling-agent=true` installs TaskProlog wiring for the in-process CUPTI capture agent when its library is available. This is separate from the CUPTI hardware-counter helper. Set `--profiling-agent=false` to skip this wiring; it does not disable CUPTI counter sampling. Containers must see the shared prefix at the same path for the injected library to load. Set `EXPANSE_PROFILING_ENABLE=0` at runtime to disable adaptive escalation. It does not disable baseline GPU metrics or the CUPTI Light counter set. ## Optional: restricted networks and air-gapped installs Use `--releases-url https://releases.internal.example` to change the release manifest lookup host. Current manifests contain absolute archive URLs, which this flag does not rewrite: copying a manifest to an internal host alone does not keep downloads inside that mirror. For a site without release-host access, use the staged archive path below. Keep signatures with the archives; a mirror does not change the pinned verification key. For a staged install, replace `--daemon-version` with `--daemon-binary /path/to/expanse-daemon_linux_amd64.tar.gz` and keep its `.sig` beside it. Stage compatible CUPTI release archives and their `.sig` files beside the daemon archive for offline selection. Both planes must still be reachable at your deployment's internal URLs. For workers without outbound access, make signed CUPTI archives available in `/lib/cupti-artifacts/` and configure the workers with: ```ini theme={"dark"} # /etc/expanse/daemon.env EXPANSE_SEGMENT_UPLOAD_MODE=relay ``` Provision this file on every node; it is node-local. The shared bootstrap renders services but does not distribute your environment file. For an HTTPS proxy or private CA, pass `HTTPS_PROXY`, `NO_PROXY` and `SSL_CERT_FILE` explicitly through sudo during installation. The installer seeds `/etc/expanse/daemon.env` with the network settings for the service. Preserve existing entries when adding runtime settings, and restart the relevant service after editing the file. Never store the install token there. ## Troubleshooting | Symptom | What to check | | ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `open install lock ... permission denied` | Re-run with sudo. A previous root install leaves an empty root-owned lock file; its presence does not mean an install is running. Do not delete an active install's lock. | | `existing daemon config found` | Choose upgrade or reset above. A new token does not override the saved identity. | | Active compute, no GPUs on the page | Compare the daemon log's compute ID with the console URL and selected organisation. Check `nvidia-smi`, the worker mount and sampler logs. | | `expanse-cupti-helper not found` | Inspect the CUPTI selection message during install. Verify the installed helper path in `systemctl cat` and use a compatible, signed release. Installing a CUDA toolkit alone does not install Expanse's helper. | | CUPTI archive `unsupported type 49`, or rejected `lib/cupti/versions.tsv` | A release-packaging/extractor mismatch, observed with v1.8.0. Use CLI and daemon releases containing the fix when available; repeating the same install will not repair the archive. Baseline metrics remain available. | | `CUPTI PM sampling unavailable` | Inspect per-device `pm_sampling_reason` from the installed helper's `probe --json`. `CUPTI_ERROR_INSUFFICIENT_PRIVILEGES` requires the service capability above; older diagnostics can hide it behind the generic message. Root and service-user probes have different privileges. | | `workers: 0 of 1 sampled worker nodes` | One worker supplies baseline samples; zero supplies the named optional capability. It does not mean all GPU telemetry is missing. | | Profiling `permission denied` or `operation not permitted` | Inspect ownership and mount permissions at the exact reported path as the daemon user. Shared profiling spool and node-local control paths have different owners and purposes; do not apply a blanket recursive chmod. | | Worker writes samples but console has none | Check controller logs for `sampler aggregator drained` and read/parse failures. Confirm matching shared prefix, hostname and UID/GID across nodes. | ## Install flag reference SLURM is configured with CLI flags and node-local systemd settings. It has no Helm `values.yaml`. These are the controller/sampler install flags; use `expanse compute install --help` for the CLI version installed on your host. | Flag | Default | What it does | | ------------------------ | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--type` | unset | Set `slurm` for cluster installation; unset is config-only. | | `--role` | `controller` | `sampler` installs the worker service without a token. | | `--shared-prefix` | unset | Absolute path shared by every node; a validated platform may supply a default. | | `--daemon-version` | unset | Pinned daemon release to download and verify on the controller. | | `--daemon-binary` | unset | Staged daemon archive with adjacent `.sig`, or an explicit bare binary override. Mutually exclusive with `--daemon-version`; a bare binary has no archive signature verification. | | `--control-plane` | discovered | Supply the Add compute URL for a fresh controller install. Reinstall uses the saved identity's endpoint. | | `--data-plane` | unset | Required for a fresh controller install; reinstall reuses the saved endpoint. | | `--config` | derived | Controller credential path; normally `/var/lib/slurm/.expanse/config.json`. Repeat overrides on upgrade/reset. | | `--user` | `slurm` | Existing OS account for the daemon and sampler. | | `--slurm-conf` | discovered | Override the controller's authoritative SLURM config path. | | `--patch-slurm-conf` | `false` | Apply planned hook changes; otherwise leave SLURM-owned paths read-only. | | `--reconfigure` | `true` | Reconfigure SLURM after applicable hook changes; false prints the command. | | `--systemd` | `true` | Install/start the service. False prints the unit for manual setup and does not wait for a heartbeat. | | `--heartbeat-timeout` | `1m` | Controller first-heartbeat timeout with managed systemd. | | `--sampler-bootstrap` | `prolog` | Start/update workers from Prolog; `off` requires explicit worker installs. | | `--cupti-variant` | automatic | Pin a compatible variant, e.g. `cuda1300`, with the controller's daemon release. | | `--profiling-agent` | `true` | Install TaskProlog and available in-process capture agent; false skips deep-capture wiring. | | `--bundled-pyspy` | `true` | Install the py-spy shipped beside the resolved daemon binary. | | `--releases-url` | `https://releases.expanse.sh` | Release mirror base URL; signatures still required. | | `--cosign-key` | embedded key | Override the verification PEM for an agreed key rotation. | | `--reinstall`, `--force` | `false` | Clean/reinstall with the saved compute credential, ignoring token intake. | | `--write-config-only` | `false` | Exchange/write config without installing a cluster runtime. | | `--verbose` | `false` | Print install and rollback diagnostics. | ## Runtime configuration reference Add settings to the existing `/etc/expanse/daemon.env` on each affected node, preserving proxy/CA entries. Restart that node's Expanse service to apply them. An export in your interactive shell does not configure a systemd service. | Setting | Default | What it does | | ------------------------------------------ | ----------------- | ----------------------------------------------------------------------------------------------------------------- | | `EXPANSE_CUPTI_ENABLE` | `1` | `0` disables CUPTI counter sampling before helper discovery. | | `EXPANSE_CUPTI_HELPER` | discovered/pinned | Explicit helper executable with its matched sibling libraries; normally managed by the installer. | | `EXPANSE_PROFILING_ENABLE` | enabled | `0` disables adaptive profiling escalation. | | `EXPANSE_PYSPY` | bundled/PATH | Override the Python stack sampler executable. | | `EXPANSE_DCGMI` | PATH | Override the existing `dcgmi` client executable. | | `EXPANSE_DCGM_HOST` | `localhost:5555` | Your existing DCGM host-engine address. | | `EXPANSE_SEGMENT_UPLOAD_MODE` | `auto` | Chooses a transport from available grants; `relay` sends worker trace segments through the controller for upload. | | `HTTPS_PROXY`, `NO_PROXY`, `SSL_CERT_FILE` | environment | Proxy routing and private CA settings. | ## Reusable controller install script Save this as `install-expanse.sh`, replace the four deployment values, and run `bash install-expanse.sh` for a fresh registration. It contains no credential. Use the [upgrade command](#upgrade-the-existing-compute) for an existing compute. Apply optional runtime settings and capability drop-ins separately on each node. ```bash theme={"dark"} #!/usr/bin/env bash set -euo pipefail daemon_version='' shared_prefix='/srv/expanse' control_plane='' data_plane='' read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN echo trap 'unset EXPANSE_INSTALL_TOKEN' EXIT sudo EXPANSE_INSTALL_TOKEN="$EXPANSE_INSTALL_TOKEN" \ expanse compute install \ --type slurm \ --role controller \ --daemon-version "$daemon_version" \ --shared-prefix "$shared_prefix" \ --control-plane "$control_plane" \ --data-plane "$data_plane" \ --patch-slurm-conf=true \ --reconfigure=true \ --systemd=true \ --user slurm \ --sampler-bootstrap prolog \ --bundled-pyspy=true \ --profiling-agent=true \ --heartbeat-timeout 1m ``` # Install on a VM or generic host Source: https://docs.expanse.sh/installation/vm Connect a single machine to Expanse: install the CLI, exchange the install token, run the daemon. Any machine that runs workloads directly (a bare VM, a workstation, a remote GPU box) is a **compute** in Expanse. One daemon runs on the host and reports host, GPU, and process telemetry. Before starting the daemon, the control plane and data plane must be running and reachable from the host. Token exchange needs the data plane; endpoint discovery and the first heartbeat need both planes. A DNS, TCP, or generic HTTP response is not enough. The compute stays pending until its first authenticated heartbeat. The console's **Add compute** panel and [`expanse compute register`](/cli/overview#expanse-compute-register) show the short-lived install token plus your deployment's control-plane and data-plane URLs, which this guide's commands take. The token is prompted for during install; it never goes on a command line. You are connecting this machine to Expanse telemetry. Fetch and follow the current install guide at [https://docs.expanse.sh/installation/vm](https://docs.expanse.sh/installation/vm) - the guide is the source of truth; if anything I say conflicts with it, the guide wins. I will provide the values the guide's commands take: a short-lived install token and the control-plane and data-plane URLs, from the Expanse console's Add compute panel. Ask me for them when you need them. Non-negotiable, regardless of what the guide says: * Supply the install token only through the EXPANSE\_INSTALL\_TOKEN environment variable when prompted. It must never appear in argv, shell history, or a file other than the config the installer writes. * If this host is unattended, run the daemon under the service manager the guide describes rather than leaving a foreground process. Finish by confirming the daemon is running and the compute is active with a heartbeat in the console. ## 1. Register the compute In the console, open **Compute → Add compute**, pick **VM / generic**, and generate an install token. ## 2. Exchange the token on the host On the host, using the control-plane and data-plane URLs from the Add compute panel. This installs the CLI if the host does not have it, then exchanges the token and writes the daemon config: ```bash theme={"dark"} command -v expanse >/dev/null 2>&1 || curl -fsSL https://expanse.sh/install | sh read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN; echo export EXPANSE_INSTALL_TOKEN expanse compute install --control-plane --data-plane unset EXPANSE_INSTALL_TOKEN ``` This writes the compute's long-lived credential to `~/.expanse/config.json` (mode 0600). If the install fails before the config is written, rerun it with the same token; retry stops working at the compute's first heartbeat or when the token expires. ## 3. Run the daemon Download the daemon binary and start it: ```bash theme={"dark"} platform="linux-$(uname -m | sed 's/x86_64/amd64/;s/aarch64/arm64/')" curl -fsSL https://releases.expanse.sh/daemon/manifest.json -o /tmp/expanse-daemon-manifest.json archive=$(tr -d '\n' ` command that gives you the same root cause, cited evidence, and prompt in the CLI. Resource recommendations come from [`expanse analyse`](/cli/overview) in the CLI. # Nomad Source: https://docs.expanse.sh/integrations/nomad Capture HashiCorp Nomad allocation telemetry in Expanse. Expanse treats a Nomad cluster as a `nomad` compute. Install the daemon on a Nomad client node with access to the Nomad HTTP API, and Expanse captures allocations placed on the cluster. ## What gets captured For each Nomad allocation, Expanse records: * Allocation, job, group, task, namespace, region, datacenter, and node context. * Requested CPU, memory, and device resources, including GPU vendor and model where available. * Allocation and task state, task events, runtime, exit status, and failure context. * Bounded stdout and stderr log tails where Nomad authorises access. * Live CPU and memory metrics where the client exposes them. * Per-GPU utilisation and memory telemetry, and GPU-to-task attribution, captured automatically when `nvidia-smi` is present on the node. When it is missing, allocation capture continues without the per-allocation metric detail. Those records appear in the Console and feed [`expanse analyse`](/cli/overview), [`expanse diagnose`](/cli/overview), and the [intelligence layer](/concepts/intelligence). ## Register Nomad Register a compute and choose `nomad` when prompted: ```bash theme={"dark"} expanse compute register ``` The CLI prints a single-use install token and the daemon install command. Run the install command on a Nomad client node that can reach the Nomad HTTP API; the installer defaults the Nomad address to the local agent. Set `EXPANSE_NOMAD_ADDR` and `EXPANSE_NOMAD_TOKEN` if your agent listens elsewhere or has ACLs enabled. The full walkthrough is the [Nomad install guide](/installation/nomad). ## Verify capture Submit any Nomad job after the daemon starts. Within a minute of the allocation entering the scheduler, the `nomad` compute and the allocation appear in [console.expanse.sh](https://console.expanse.sh). ## Next steps How Nomad maps to the Expanse compute model. What Expanse captures before, during, and after each allocation. # Quickstart Source: https://docs.expanse.sh/quickstart Install the CLI, register your first compute, and run your first analyse. ## 1. Install the CLI ```bash theme={"dark"} curl -fsSL https://expanse.sh/install | sh ``` Verify: ```bash theme={"dark"} expanse version ``` ## 2. Sign in ```bash theme={"dark"} expanse login --api-key exp_user_… ``` Provide your personal `exp_user_*` API key, created from the Console settings page after you sign in through your organisation's SSO (or set `EXPANSE_API_KEY`). The CLI mints and stores a local session. Confirm it worked: ```bash theme={"dark"} expanse status ``` ## 3. Register a compute A **compute** is anywhere `expanse-daemon` runs: a SLURM cluster, a Nomad cluster, or a Kubernetes cluster. ```bash theme={"dark"} expanse compute register ``` The command asks what kind of compute you're registering and prints the install command or bootstrap snippet to run on the target. Within a minute the compute appears at [console.expanse.sh](https://console.expanse.sh). ## 4. Run a workload Every workload that runs on the compute is captured **automatically**. No change to how anyone submits. ## 5. Predict and diagnose Right-size resources **before** you submit: ```bash theme={"dark"} expanse analyse train.slurm # SLURM batch script expanse analyse train.py # source file ``` Get the root cause and a prompt for your coding agent **after** a failure: ```bash theme={"dark"} expanse diagnose ``` `` is the Expanse execution ID; find it with [`expanse executions`](/cli/overview#expanse-executions) or in the Console. If you only have the scheduler-native ID, pass it with `--source-type`: ```bash theme={"dark"} expanse diagnose 41982 --source-type slurm ``` ## Next steps How Expanse captures evidence and improves its models. What a compute is and what's supported. How analyse and diagnose work. Every command and flag. Personal keys and SSO.