Performance

Expanse’s collectors run on every node of your cluster, so we measure what they cost the jobs running there. On each scheduler we ran the same ResNet-50 training job many times with Expanse on and off, and compared how long it took. Each figure is how much longer the whole job took with Expanse on, with its interval in brackets. Where the interval includes zero, we found no measurable difference. Every result applies to the machine it was measured on; other GPUs and node types need their own runs.

​

SLURM

​

How we measured

• One cloud machine with one NVIDIA GH200 and 64 Arm vCPUs, running SLURM.
• ResNet-50 training on the Imagenette dataset: 30 epochs on the GH200, about 164 seconds, and one epoch on 32 CPU cores, about 399 seconds.
• Each setup ran beside a run without Expanse, in random order. We added pairs until the last three agreed within half a percentage point, so the intervals are wider than plain 95% intervals. 228 runs in all.
• Metrics went from each node through Expanse’s controller daemon. Sending them straight from the node measured the same.

​

Training time

Standard collection made no measurable difference to the GPU job and added about 3.8% to the CPU job. Most of that comes from CPU hardware counters: on their own they added about 3.9% to the same CPU job, and you can switch them off. The two profilers are optional, and you can leave them off.

​

Deep GPU traces

When a rule detects a likely problem, such as a GPU sitting idle while the CPU is busy, an agent inside the job traces its GPU work for about 25 seconds and then stops until the next rule fires. Every run below had collection and the GPU profiler on. In both traced setups, about 31% of the run was being traced.

​

Resource use on the node

The GPU job on one GH200, as the median across runs. Network and disk include the controller daemon, which itself used 0.01 cores and 87 to 145 MiB.

​

What the collectors recorded

• Every collection and trace check passed.
• The longest gap between samples was 5.5 seconds.
• When checked, the newest sample in Expanse’s data plane, inside the network, was a median 11.5 seconds old.

​

Kubernetes

​

How we measured

• One cloud machine with two NVIDIA H100 80GB GPUs, 52 vCPUs and 442 GiB of memory, running Kubernetes (k3s).
• ResNet-50 training on the Imagenette dataset: 30 epochs on one H100, about 200 seconds, and one epoch on 32 CPU cores, about 176 seconds.
• Each setup ran back to back with a run without Expanse, in random order, and the intervals are 95% confidence intervals. All 76 runs finished.

​

Training time

Standard collection made no measurable difference to the GPU job and added about 1.3% to the CPU job. CPU hardware counters were off in these runs. They are now on by default, and we have not yet measured them on Kubernetes.

​

Deep GPU traces

The same agent as on SLURM, traced for 25 to 30 seconds at a time. Every run below had collection and the GPU profiler on. In the runs traced twice, 27% of the run was being traced. Tracing on each request opened two or three windows a run, about one every 95 seconds.

​

Resource use on the node

The GPU job on one H100, as the median across runs.

​

What the collectors recorded

• Every 5-second sample arrived, and none was dropped.
• When checked, the newest sample in Expanse’s data plane, inside the network, was a median 4.6 seconds old.

​

Run it on your own nodes

Our benchmark kit runs the same test on one of your Kubernetes GPU nodes, with our training job or your own. Its images and manifests are pinned, so you get the setup we had. Use a cluster you can spare: the test uninstalls Expanse’s collectors, which stops collection on every node until the kit reinstalls them with your saved settings. For a SLURM cluster, talk to us about a run on your nodes. When the kit finishes, it sends the results to us. It shows you exactly what it will send and waits for your yes first. You can also keep the results and send them later, which is how an air-gapped node sends them.

​

Talk to us