The GPU Utilisation Problem Nobody Wants to Talk About
OneDot61 is a technology distributor. OneDot61 has a commercial interest in products discussed here.
Explore CoreSpan Systems and compute the solution→
AI infrastructure investment is accelerating, but the economics of how GPUs are actually used tells a different story. Across data centres and enterprise AI environments, GPU utilisation rates routinely sit well below 50%. Some workloads burst to full capacity for hours, then idle. Others need two GPUs for inference but are locked to a server provisioned with eight. The hardware is expensive. The waste is significant.
This isn't a purchasing problem — most organisations have invested heavily in GPU capacity. It's an architecture problem. The traditional server model binds GPUs directly and permanently to a specific host. Once assigned, those resources can't move. When a workload finishes or scales down, the GPUs sit stranded until the next job is manually provisioned.
Why the Scale-Up Model Doesn't Work for AI
Enterprise AI workloads are dynamic by nature. Training runs are intensive and time-bounded. Inference loads vary throughout the day. Fine-tuning jobs spike unpredictably. The fixed server-to-GPU relationship that worked well for traditional compute is fundamentally mismatched to these patterns.
Cloud environments offer some flexibility, but at a cost — both financial and in terms of control. Organisations running large benchmark workloads have reported exhausting significant cloud credits in short periods, simply because the provisioning model doesn't allow for right-sized allocation. You pay for what you reserved, not what you used.
CoreSpan Systems Delivers Composable Infrastructure Built for AI
Corespan Systems takes a different approach entirely. Rather than binding GPUs to servers, their PRU 2500 chassis pools 8–12 NVIDIA GPUs as shared infrastructure, connected via PCIe Gen5 photonic interconnects using the FIC 2500 fabric interface. The Corespan Composer software then virtualises and dynamically allocates those pooled resources — GPUs, storage, and PCIe — across multiple hosts in real time, matched to actual workload demand.
The result is a scale-across architecture that replaces the traditional scale-up model. A workload that needs four GPUs gets four. When it finishes, those GPUs are immediately available to the next job. Existing Docker and Kubernetes workflows integrate without changes. Ageing GPUs that would otherwise be retired can be redeployed into the shared pool, extending their useful life.
For neocloud providers, enterprise AI teams, and HPC environments where GPU utilisation and operational cost are constant pressures, Corespan's composable infrastructure offers a path to doing more with what you already have and spending less on what you don't need.