Install on Kubernetes

A Kubernetes cluster is one compute in Expanse. One Helm chart installs a daemon that watches pods and jobs (and the batch systems built on them: Volcano, Kueue, Argo Workflows, Flux, Ray, Flyte) and reports each run to the console: lifecycle, requested against used resources, GPU activity and, when enabled, where in the code a run spends its time. Workloads are untouched. The console’s Add compute panel shows the short-lived install token and your control-plane and data-plane URLs. The token is prompted for during install and never goes on a command line or into Helm values.

Prefer to delegate? Copy this into an AI assistant with kubectl and Helm access; it fetches this guide itself, so it always follows the current steps.

Prerequisites

Console access in your Expanse organisation (any member can register).
A workstation with kubectl access to the cluster and Helm 3.
Outbound HTTPS from the daemon pod to your Expanse control and data planes. Both planes must be up before the pod starts: token exchange needs the data plane, discovery and the first heartbeat need both, and the compute stays pending until that heartbeat.
For Python stack attribution or the local run hop, a daemon namespace that admits privileged pods: the node sampler then runs with hostPID and as root, and the install notes print the label to set. Pods opt in to stack attribution with the expanse.sh/code-attribution: enabled annotation or a namespace in nodeSampler.codeAttribution.namespaces; the sampler reads at most nodeSampler.codeAttribution.readsPerSecond stacks a second per node, 10 by default, shared fairly across the node’s opted-in pods.

GPU nodes, especially HGX/NVSwitch systems: disable unattended package upgrades and apply updates during planned maintenance with the node drained. Never restart nvidia-fabricmanager while GPUs are in use. Reboot after kernel or NVIDIA driver updates before admitting GPU workloads again.A fabric-manager restart under live GPUs can leave NVLink fabric stale: NCCL collectives may fail even while single-GPU metrics look healthy. If this happens, drain and reboot the affected node. Kubernetes preflight cannot check the host’s unattended-upgrade configuration; the node administrator must verify it.

1. Register the compute

In the console, open Compute → Add compute, pick Kubernetes, and generate an install token.

2. Exchange the token

On your workstation, with the URLs from the Add compute panel:

command -v expanse >/dev/null 2>&1 || curl -fsSL https://expanse.sh/install | sh
read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN; echo
export EXPANSE_INSTALL_TOKEN
expanse compute install --type kubernetes --namespace expanse-system \
  --control-plane <your-control-plane-url> --data-plane <your-data-plane-url>
unset EXPANSE_INSTALL_TOKEN

This writes the compute’s long-lived credential to ~/.expanse/config.json and a first pull credential for the Expanse image registry to ~/.expanse/registry-dockerconfig.json; neither is printed. If the install fails before the config file is written, rerun it with the same token; retry stops working at the compute’s first heartbeat or when the token expires. --namespace selects the namespace used by every command the CLI prints. Use the same --kubeconfig and --context flags on preflight and install when selecting a file or context explicitly. Install carries these selectors into its printed kubectl and Helm commands. Without --namespace, both use the selected kubeconfig context’s namespace, or default when the context has none. If no usable context exists, install requires --namespace before exchanging the token. An explicit namespace works without kubeconfig.

3. Create the namespace and the Secrets

The chart uses existing Secrets and never creates one. Keep the Secrets and the release in the same namespace; this guide uses expanse-system.

kubectl create namespace expanse-system --dry-run=client -o yaml | kubectl apply --server-side -f -
kubectl -n expanse-system create secret generic expanse-daemon-config \
  --from-file=config.json="$HOME/.expanse/config.json" \
  --dry-run=client -o yaml | kubectl apply --server-side --field-manager=expanse-compute-install -f - && \
  kubectl -n expanse-system annotate secret expanse-daemon-config kubectl.kubernetes.io/last-applied-configuration-

The Expanse image registry is private. Create its pull Secret from the file the install step wrote:

kubectl -n expanse-system create secret docker-registry expanse-registry \
  --from-file=.dockerconfigjson=$HOME/.expanse/registry-dockerconfig.json \
  --dry-run=client -o yaml | kubectl apply --server-side --field-manager=expanse-compute-install --force-conflicts -f - && \
  kubectl -n expanse-system annotate secret expanse-registry kubectl.kubernetes.io/last-applied-configuration-

The dedicated field manager prevents kubectl from copying credentials into last-applied-configuration, including on Secrets created by older CLI versions. The chained annotation removal clears previously copied credentials and is safe to rerun when the annotation is absent. If the install step printed that it could not issue a pull credential, generate one in the console under Platform → Registry and select Copy token. Run the fallback block printed by the CLI in Bash with base64 and jq installed, then paste the registry host and token at its prompts. That block creates the Secret in the selected namespace and context using server-side apply, then removes its temporary credential file.

4. Install the chart

Set the Secret name, a config revision and the registry pull refresher. The chart version you install names the daemon release, so the values carry no image tag. Change the revision whenever you replace the config Secret so every daemon pod restarts with that config.

# values.yaml
config:
  existingSecret: expanse-daemon-config
  rolloutRevision: initial-config
registryPull:
  enabled: true

For certificate injection, DNS registration or admission policies, add annotations and labels to the values for each affected component before installing. With registryPull.enabled, the chart runs a small refresher that writes a fresh token into expanse-registry every 4 hours (registryPull.refreshInterval) and adds the Secret to every chart pod’s imagePullSecrets, so nobody handles a pull token after install. The refresher may read and update that one Secret and nothing else; it never creates it, so step 3 comes first. Keep the setting in values.yaml rather than on the command line so every upgrade keeps it. Without the refresher, the token in the Secret expires 12 hours after it was issued: list expanse-registry under imagePullSecrets and rerun the console’s command before a helm upgrade, a node replacement or any other fresh pull. The same 12 hours apply to a console credential used because the install step could not issue one. Clusters that pull only from their own mirror set image.repository; a mirror needs no expiring Secret.

From charts.expanse.sh
From a local chart

helm repo add expanse https://charts.expanse.sh --force-update && helm repo update
helm upgrade --install expanse-daemon expanse/expanse-daemon \
  -n expanse-system -f values.yaml

Air-gapped clusters install from a chart directory or .tgz, pulled with helm pull expanse/expanse-daemon --version <chart-version> on a connected machine or taken from your release bundle mirror:

helm upgrade --install expanse-daemon ./expanse-daemon \
  -n expanse-system -f values.yaml

Images then come from your registry mirror: set image.repository and, if the mirror needs credentials, list its pull Secret under imagePullSecrets. A cluster that cannot reach the Expanse control plane leaves registryPull off, since the refresher has nowhere to fetch a token from.

Everything else has a default. helm show values expanse/expanse-daemon prints every value with its comment, and the values reference at the end of this page lists them all. In production set state.persistence.enabled=true so queued source uploads and the daemon’s cached endpoints and identity survive pod replacement.

5. Verify

The compute becomes active after its first authenticated heartbeat, normally within a minute of the pod starting. If nothing arrives, read kubectl -n expanse-system logs deploy/expanse-daemon.

Upgrade

Upgrade the release in place to the chart version you are moving to:

helm upgrade expanse-daemon expanse/expanse-daemon \
  -n expanse-system -f values.yaml --version <chart-version>

Each chart version pins one daemon release, its appVersion, and that release’s image digest, so all pods pull one pinned version and the chart version is the only thing an upgrade changes. A values file that still sets image.tag stays on that release until you remove it. With registryPull.enabled the pull Secret is already fresh; without it, re-issue the Secret from the Registry page first if the cluster pulls from the Expanse registry, since its credential lasts 12 hours. The observer restarts once. With state.persistence.enabled=false its queued source uploads sit on an emptyDir and do not survive that restart, so enable persistence before the first production upgrade. Turning nodeSampler.cupti, nodeSampler.dcgm or nodeSampler.codeAttribution on or off is a rolling restart of the node sampler DaemonSet, one GPU node at a time; the workloads on those nodes are not touched. The liveness probe (probes.enabled) is on by default and needs daemon v0.5.0 or newer, which carries the healthcheck subcommand it runs. On an older image every probe fails and the kubelet restarts the pods, so set probes.enabled=false there.

Replace a lost compute credential

If the config Secret is deleted, or the cluster is rebuilt, the compute keeps its identity. On the compute’s console page choose Re-issue install token, then exchange it from a workstation with the CLI (step 2 installs it):

read -rsp 'Expanse install token: ' EXPANSE_INSTALL_TOKEN; echo
EXPANSE_INSTALL_TOKEN="$EXPANSE_INSTALL_TOKEN" expanse compute install --type kubernetes --reinstall --namespace expanse-system \
  --control-plane <your-control-plane-url> --data-plane <your-data-plane-url>
unset EXPANSE_INSTALL_TOKEN

Cluster intact, Secret lost: recreate the Secret and restart the daemon workloads so they load the new credential. The daemon reads its config once at startup; until then the old pods loop on rejected heartbeats. If the Secret still exists, --force-conflicts replaces its configuration even when another field manager owns it.
kubectl -n expanse-system create secret generic expanse-daemon-config \ --from-file=config.json="$HOME/.expanse/config.json" \ --dry-run=client -o yaml | kubectl apply --server-side --field-manager=expanse-compute-install --force-conflicts -f - && \ kubectl -n expanse-system annotate secret expanse-daemon-config kubectl.kubernetes.io/last-applied-configuration- kubectl -n expanse-system rollout restart deployment/expanse-daemon
Add kubectl -n expanse-system rollout restart daemonset/expanse-daemon-node-sampler when nodeSampler.enabled=true, and kubectl -n expanse-system rollout restart deployment/expanse-daemon-registry-pull when registryPull.enabled=true.
Cluster rebuilt: nothing is there to restart; repeat steps 3 and 4 with the exchanged config.

The previous credential stops working at the exchange. The token can be exchanged once.

Optional: GPU metrics

The default install observes workload lifecycle only. On GPU clusters, enable the node sampler DaemonSet: it ships node and per-GPU metrics and attributes each GPU to the pod holding it, non-root by default.

nodeSampler:
  enabled: true
  # Only where the NVIDIA runtime is a RuntimeClass (k3s, RKE2 with the toolkit).
  # A node without that runtime cannot start the pod, so pair it with a selector.
  runtimeClassName: nvidia
  nodeSelector:
    nvidia.com/gpu.present: "true"

The sampler runs on every node by default. CPU-only nodes still get host and per-pod usage and Python stacks; they report no GPU chips, so the GPU health summary counts GPU nodes only. Set nodeSelector to save the pods on large CPU pools or when a RuntimeClass is required. The sampler reads the host /proc and /sys, so its namespace must allow privileged pods:

kubectl label ns expanse-system pod-security.kubernetes.io/enforce=privileged --overwrite

GPU-to-pod attribution runs non-root: the sampler joins the processes nvidia-smi lists on each card to their pod through the read-only host /proc. nodeSampler.podResources.source=podresources reads the kubelet pod-resources socket instead, which also names the owner of a card whose pod has not started a GPU process yet, at the cost of running the sampler as root. nodeSampler.podResources.enabled=false drops per-pod attribution and the per-pod CPU and memory series altogether.

Optional: GPU profiling (CUPTI and DCGM)

Both live in the node sampler and are off by default. Nothing is installed on the node, and Expanse never runs a DCGM host engine of its own.

nodeSampler:
  # GPU performance counters. Adds the compute driver capability and CAP_PERFMON;
  # needs an NVIDIA driver of the r580 branch or newer.
  cupti:
    enabled: true
    mode: perfmon
  # DCGM from the host engine you already run: the GPU Operator's nvidia-dcgm, or a
  # host nvidia-dcgm.service bound to the node IP.
  dcgm:
    hostEngine: "$(NODE_IP):5555"

Without a reachable engine the DCGM chip on the compute page reads disabled with the reason and the standard GPU metrics keep flowing. To keep counters restricted, retain RmProfilingAdminOnly=1 on the node. The default nodeSampler.cupti.mode=perfmon needs Linux 5.8+ and grants CAP_PERFMON without forcing root for counters. The pod-resources attribution source and Python attribution require root independently. sysadmin explicitly grants CAP_SYS_ADMIN and forces root for older hosts; none grants neither. Replace the retired cupti.sysAdmin key with cupti.mode when upgrading.

Optional: Python stack attribution

nodeSampler.codeAttribution.enabled=true lets the node sampler read the Python stacks of pods that opted in: a pod annotated expanse.sh/code-attribution: enabled, or one in a namespace listed in nodeSampler.codeAttribution.namespaces; nothing else is sampled. The sampler reads at most nodeSampler.codeAttribution.readsPerSecond stacks a second per node, 10 by default, shared fairly across the node’s opted-in pods. An execution page then shows where in the code a run spends its time. Python only; compiled workloads get no stacks. The sampler then needs the host PID namespace, CAP_SYS_PTRACE, root and read-only pods access, which the chart applies. System namespaces and the release’s own pods are never sampled, and watch.namespaces scopes it like the observer. GPU profiling and stack attribution need daemon v0.5.0 or newer: a chart whose appVersion is v0.5.0 or later, with no older image.tag set.

Optional: in-process GPU profiling agent

Deep captures widen the out-of-process counters by default. To also record kernel activity, NVTX ranges and, where the driver allows, PC samples inside the workload, enable agent injection and label the GPU pods that may receive it:

agentInjection:
  enabled: true
  tls:
    certManager:
      issuerRef:
        kind: ClusterIssuer
        name: your-issuer

kubectl label ns <workload-namespace> expanse.sh/profiling-agent=true

metadata:
  labels:
    expanse.sh/profiling-agent: "true"

The observer’s admission webhook adds the agent library, a control mount and four environment variables to a labelled GPU pod in a labelled namespace when it is created; the node sampler stages the library and arms captures for those pods only. A mutated pod gains a hostPath volume, which the Pod Security baseline and restricted profiles refuse, so an opted-in namespace must run at the privileged level and the webhook never touches a namespace labelled as enforcing either. Without cert-manager, set agentInjection.tls.certManager.enabled=false and point agentInjection.tls.existingSecret and agentInjection.tls.caBundle at your own serving pair. Pods created before the webhook existed keep running unprofiled until restarted.

Optional: namespace-scoped install

Set watch.namespaces and rbac.clusterWide=false: the chart grants namespace permissions only where it watches, plus the refresher’s get and update on the pull Secret, and renders nothing cluster-scoped. The queue view then reports degraded, because queue and capacity snapshots need one cluster-scoped read, list nodes; grant it with rbac.listNodes=true when an administrator allows. Per-pod GPU attribution and stack attribution also read pods cluster-wide; leave both off for a strictly namespace-scoped install.

Optional: network policy

The daemon dials only your control and data planes, the API server and, when configured, the DCGM host engine. Turn on the chart’s NetworkPolicy and narrow the plane addresses:

networkPolicy:
  enabled: true
  egress:
    planes:
      - cidr: <control-plane-cidr>
        port: 443
      - cidr: <data-plane-cidr>
        port: 443

On EKS with Pod Identity add networkPolicy.extraEgress for 169.254.170.23/32 on TCP 80.

Optional: pin the image digest

A chart from charts.expanse.sh already pins its daemon release by digest. If you set image.tag yourself, set image.digest to the sha256:... from that release’s manifest beside it; a re-pushed tag can then never change what the cluster runs.

Optional: OpenShift

oc adm policy add-scc-to-user restricted-v2 -z expanse-daemon -n expanse-system
oc adm policy add-scc-to-user privileged -z expanse-daemon-node-sampler -n expanse-system

Set podSecurityContext: null so the SCC’s uid range applies to the observer. The second line is needed only with the node sampler.

Getting Started

Installation

Core Concepts

Deployment

Integrations

Install on Kubernetes

Prerequisites

1. Register the compute

2. Exchange the token

3. Create the namespace and the Secrets

4. Install the chart

5. Verify

Upgrade

Replace a lost compute credential

Optional: GPU metrics

Optional: GPU profiling (CUPTI and DCGM)

Optional: Python stack attribution

Optional: in-process GPU profiling agent

Optional: namespace-scoped install

Optional: network policy

Optional: pin the image digest

Optional: OpenShift

Getting Started

Installation

Core Concepts

Deployment

Integrations

​Prerequisites

​1. Register the compute

​2. Exchange the token

​3. Create the namespace and the Secrets

​4. Install the chart

​5. Verify

​Upgrade

​Replace a lost compute credential

​Optional: GPU metrics

​Optional: GPU profiling (CUPTI and DCGM)

​Optional: Python stack attribution

​Optional: in-process GPU profiling agent

​Optional: namespace-scoped install

​Optional: network policy

​Optional: pin the image digest

​Optional: OpenShift

Prerequisites

1. Register the compute

2. Exchange the token

3. Create the namespace and the Secrets

4. Install the chart

5. Verify

Upgrade

Replace a lost compute credential

Optional: GPU metrics

Optional: GPU profiling (CUPTI and DCGM)

Optional: Python stack attribution

Optional: in-process GPU profiling agent

Optional: namespace-scoped install

Optional: network policy

Optional: pin the image digest

Optional: OpenShift