> ## Documentation Index > Fetch the complete documentation index at: https://docs.expanse.sh/llms.txt > Use this file to discover all available pages before exploring further. # Performance > What Expanse's collectors cost a training job on SLURM and Kubernetes, how we measured it, and how to measure it on your own nodes. Expanse's collectors run on every node of your cluster, so we measure what they cost the jobs running there. On each scheduler we ran the same ResNet-50 training job many times with Expanse on and off, and compared how long it took. Each figure is how much longer the whole job took with Expanse on, with its interval in brackets. Where the interval includes zero, we found no measurable difference. Every result applies to the machine it was measured on; other GPUs and node types need their own runs. ## SLURM ### How we measured * One cloud machine with one NVIDIA GH200 and 64 Arm vCPUs, running SLURM. * ResNet-50 training on the Imagenette dataset: 30 epochs on the GH200, about 164 seconds, and one epoch on 32 CPU cores, about 399 seconds. * Each setup ran beside a run without Expanse, in random order. We added pairs until the last three agreed within half a percentage point, so the intervals are wider than plain 95% intervals. 228 runs in all. * Metrics went from each node through Expanse's controller daemon. Sending them straight from the node measured the same. ### Training time | What was on | GPU job, one GH200 | CPU job, 32 cores | | - | - | - | | Collection | +0.37% (-0.04 to +0.78) | +3.81% (+2.27 to +5.37) | | Collection and the GPU profiler | +0.25% (-0.65 to +1.15) | Not reported | | Collection and the Python stack profiler | +0.45% (0.00 to +0.91) | +3.85% (+2.60 to +5.12) | | Collection and both profilers | +0.32% (-0.40 to +1.05) | +3.72% (+1.47 to +6.02) | Standard collection made no measurable difference to the GPU job and added about 3.8% to the CPU job. Most of that comes from CPU hardware counters: on their own they added about 3.9% to the same CPU job, and you can switch them off. The two profilers are optional, and you can leave them off. ### Deep GPU traces When a rule detects a likely problem, such as a GPU sitting idle while the CPU is busy, an agent inside the job traces its GPU work for about 25 seconds and then stops until the next rule fires. Every run below had collection and the GPU profiler on. | Deep GPU traces | GPU job, one GH200 | | - | - | | Agent loaded, not tracing | +0.34% (-0.34 to +1.02) | | Traced twice, when a rule fired | +1.13% (+0.20 to +2.07) | | Traced again on each request | +1.15% (+0.17 to +2.14) | In both traced setups, about 31% of the run was being traced. ### Resource use on the node The GPU job on one GH200, as the median across runs. Network and disk include the controller daemon, which itself used 0.01 cores and 87 to 145 MiB. | What was on | Collector CPU | Collector memory, peak | Memory added to the job | Network sent | Disk written | | - | - | - | - | - | - | | Collection | 0.05 cores | 182 MiB | None | 5.7 KiB/s | 85 KiB/s | | Collection and the GPU profiler | 0.06 cores | 304 MiB | None | 6.1 KiB/s | 90 KiB/s | | Collection and the Python stack profiler | 0.21 cores | 279 MiB | None | 6.8 KiB/s | 137 KiB/s | | Collection and both profilers | 0.21 cores | 461 MiB | 0.1 GiB | 7.3 KiB/s | 141 KiB/s | | Deep GPU traces, on each request | 0.44 cores | 694 MiB | 0.7 GiB | 366 KiB/s | 881 KiB/s | ### What the collectors recorded * Every collection and trace check passed. * The longest gap between samples was 5.5 seconds. * When checked, the newest sample in Expanse's data plane, inside the network, was a median 11.5 seconds old. ## Kubernetes ### How we measured * One cloud machine with two NVIDIA H100 80GB GPUs, 52 vCPUs and 442 GiB of memory, running Kubernetes (k3s). * ResNet-50 training on the Imagenette dataset: 30 epochs on one H100, about 200 seconds, and one epoch on 32 CPU cores, about 176 seconds. * Each setup ran back to back with a run without Expanse, in random order, and the intervals are 95% confidence intervals. All 76 runs finished. ### Training time | What was on | GPU job, one H100 | CPU job, 32 cores | | - | - | - | | Collection | +0.12% (-0.43 to +0.67) | +1.32% (+0.38 to +2.26) | | Collection and the GPU profiler | +0.06% (-0.59 to +0.72) | +0.81% (-0.66 to +2.31) | | Collection and the Python stack profiler | +0.71% (+0.52 to +0.90) | +3.08% (+1.40 to +4.79) | | Collection and both profilers | +0.50% (-0.03 to +1.03) | +2.50% (+1.37 to +3.64) | Standard collection made no measurable difference to the GPU job and added about 1.3% to the CPU job. CPU hardware counters were off in these runs. They are now on by default, and we have not yet measured them on Kubernetes. ### Deep GPU traces The same agent as on SLURM, traced for 25 to 30 seconds at a time. Every run below had collection and the GPU profiler on. | Deep GPU traces | GPU job, one H100 | | - | - | | Agent loaded, not tracing | +0.09% (-0.92 to +1.12) | | Traced twice, when a rule fired | +0.92% (+0.71 to +1.12) | | Traced again on each request | +1.02% (+0.51 to +1.54) | In the runs traced twice, 27% of the run was being traced. Tracing on each request opened two or three windows a run, about one every 95 seconds. ### Resource use on the node The GPU job on one H100, as the median across runs. | What was on | Collector CPU | Collector memory, peak | Memory added to the job | Network sent | Disk written | | - | - | - | - | - | - | | Collection | 0.05 cores | 52 MiB | None | 19 KB/s | 1.7 KB/s | | Collection and the GPU profiler | 0.06 cores | 236 MiB | None | 21 KB/s | 2.5 KB/s | | Collection and the Python stack profiler | 0.43 cores | 97 MiB | None | 48 KB/s | 1.6 KB/s | | Collection and both profilers | 0.43 cores | 308 MiB | 0.08 GiB | 50 KB/s | 2.4 KB/s | | Deep GPU traces, on each request | 0.30 cores | 580 MiB | 0.7 to 0.9 GiB | 156 KB/s | 260 KB/s | ### What the collectors recorded * Every 5-second sample arrived, and none was dropped. * When checked, the newest sample in Expanse's data plane, inside the network, was a median 4.6 seconds old. ## Run it on your own nodes Our benchmark kit runs the same test on one of your Kubernetes GPU nodes, with our training job or your own. Its images and manifests are pinned, so you get the setup we had. Use a cluster you can spare: the test uninstalls Expanse's collectors, which stops collection on every node until the kit reinstalls them with your saved settings. For a SLURM cluster, [talk to us](https://expanse.sh/contact) about a run on your nodes. When the kit finishes, it sends the results to us. It shows you exactly what it will send and waits for your yes first. You can also keep the results and send them later, which is how an air-gapped node sends them. ## Talk to us > **[Book a call](https://expanse.sh/contact)** > > We'll go through the results with you and plan a run on your own nodes. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.