The AI industry has become fixated on counting GPUs, announcing ever larger clusters and measuring investment in gigawatts rather than outcomes. Yet as AI training scales into tens of thousands of accelerators, the organisations creating the greatest competitive advantage may not be those buying the most infrastructure, but those extracting the greatest amount of productive work from every GPU they already own.
The AI industry has developed an unhealthy obsession with size. Every few weeks another hyperscaler announces a larger cluster, another operator unveils another gigawatt-scale campus, and another funding round promises unprecedented GPU capacity. Yet beneath the headlines sits a much less glamorous reality. Some of the biggest gains in AI performance may come not from buying more infrastructure, but from preventing existing infrastructure from wasting the compute it already has.
What if the greatest source of competitive advantage is no longer buying more compute, but recovering the enormous amount of compute that is already being lost?
That question is becoming increasingly relevant as enterprises move beyond AI experimentation and begin operating production-scale infrastructure. Purchasing thousands of GPUs is no longer the difficult part. Ensuring those processors spend every possible second performing useful work has become the much greater operational challenge. AI infrastructure is quietly exposing an uncomfortable truth that traditional enterprise computing largely avoided. Installing expensive hardware does not necessarily mean organisations receive equivalent business value from it.
Dan Zheng, Chief Business Officer at Clockwork, believes the industry has reached precisely this inflection point. Rather than continuing to judge infrastructure by the number of GPUs installed or the theoretical performance figures printed on specification sheets, he argues that enterprises need to start measuring something much closer to commercial reality.
“There is more demand than supply and there will continue to be bottlenecks for at least the next couple of years,” he says. “The real question is not simply how to buy more GPUs. It is how to squeeze more work from the GPU clusters you already have. In many ways efficiency becomes the new capacity. If you are not getting the most efficient use from existing infrastructure, you are simply going to be slower to market, whether that means training new models, developing new drugs or building better financial models.”
The observation is deceptively simple, yet it fundamentally changes how AI infrastructure should be evaluated. Until now, the industry has largely viewed efficiency as an optimisation exercise to be addressed after infrastructure has been deployed. Zheng suggests it should instead become one of the primary design objectives because every percentage point of lost utilisation represents compute that has already been purchased but is no longer generating competitive advantage.
AI has changed the rules of infrastructure
Part of the problem is that AI training behaves nothing like the enterprise workloads data centres were originally designed to support. Traditional business applications consist largely of independent transactions. If an individual server fails, another server usually takes over. The workload continues, users rarely notice, and infrastructure has done precisely what it was designed to do.
Large AI training clusters operate according to a completely different contract. Thousands of GPUs work together as a single distributed system, synchronising continuously while exchanging enormous volumes of information across high-speed networks. Progress depends upon every participant completing each stage of the calculation before the next can begin. The cluster therefore moves at the speed of its slowest component, and the failure of a single participant can halt the progress of every other processor involved in the training run.
“The workloads are very different,” Zheng explains. “AI training involves continuous cycles of compute and communication running in lockstep. If one GPU slows down compared with its peers, or one network path develops a problem, it slows the entire job. When you are operating across thousands or even hundreds of thousands of GPUs, you have effectively built one enormous, distributed computer. You need resilience at the system level, not simply at the hardware level.”
That distinction has profound implications for infrastructure architecture. Reliability is no longer determined solely by the quality of an individual server or accelerator. Instead, it emerges from the behaviour of the entire system, including networking, storage, orchestration software and workload scheduling. The larger AI clusters become, the more difficult that systems engineering challenge becomes because every additional GPU also introduces additional optical links, switches, network interfaces and potential points of failure.
The mathematics quickly becomes uncomfortable. A training cluster containing around one thousand GPUs may also contain many thousands of networking components, each statistically reliable when viewed individually but collectively guaranteeing that failures become routine rather than exceptional. Published research already suggests that mean time between failures falls dramatically as cluster sizes increase, meaning interruptions become an operational certainty rather than a possibility.
The hidden cost nobody has been measuring
The remarkable aspect of this problem is not that failures occur. Engineers have always expected hardware to fail eventually. The more revealing question is how the industry has chosen to respond.
Today, most large AI training jobs rely upon periodic checkpoints. During training, the model state is written to persistent storage at predetermined intervals so that if something fails the workload can restart from the most recent checkpoint rather than beginning again from scratch. It is a practical solution that has become accepted throughout the industry, yet it also hides an enormous economic penalty.
Every failure occurring between two checkpoints forces organisations to repeat work they have already completed. GPUs that were fully occupied generating valuable computation are instead asked to perform exactly the same calculations for a second time. The larger the model, the larger the cluster and the longer the training run, the more expensive that recomputation becomes.
“What people often overlook is that it is not simply a restart,” Zheng says. “You lose all the work completed since the previous checkpoint. There is always a balance because if you checkpoint too frequently you spend more time writing data to storage than progressing the model. If you checkpoint less frequently, every failure becomes more expensive. The industry has largely accepted that compromise because it has not had many alternatives.”
That acceptance may have been reasonable when only a handful of frontier AI laboratories possessed infrastructure of this scale. It becomes far less convincing as enterprises across manufacturing, financial services, pharmaceuticals and energy begin operating increasingly sophisticated AI environments where every additional day of model training delays commercial outcomes.
One large enterprise customer, Zheng explains, recently calculated that improvements in infrastructure resilience were saving tens of thousands of GPU hours every month. Viewed purely as utilisation statistics, those numbers appear impressive. Viewed as business outcomes, they represent additional research projects, faster product development and shorter routes to market without purchasing a single additional accelerator.
Measuring outcomes rather than availability
Perhaps the most significant change now taking place concerns how infrastructure reliability itself is measured. For decades, enterprise IT has relied upon familiar metrics such as uptime and availability because they reflected the nature of traditional applications. If servers remained operational, businesses generally remained productive. AI training exposes the limitations of that assumption because hardware may remain available while contributing surprisingly little useful work.
“It is not really about raw server uptime,” Zheng argues. “It is about job completion time. The objective is making sure you are not wasting GPU hours and that you accelerate completion of the workload. That allows more jobs to run and creates far more value from the infrastructure.”
This represents an important philosophical shift. Organisations are beginning to recognise that infrastructure should ultimately be judged by business outcomes rather than engineering statistics. A cluster that completes models predictably and efficiently creates greater commercial value than one achieving impressive availability numbers while repeatedly forcing expensive recomputation after failures.
The same logic increasingly extends beyond infrastructure. Zheng believes AI itself will gradually move towards outcome-based commercial models rather than charging simply for consumption. “Everything should move towards outcomes,” he says. “Even with AI tools today we pay for tokens, but not all tokens create the same value. Infrastructure should work in exactly the same way. What matters is whether the outcome is achieved.”
That thinking also explains why contractual commitments based upon protected training progress rather than traditional uptime guarantees are beginning to appear. They reflect a market that is becoming more commercially mature and less willing to accept engineering metrics as proxies for business performance.
Software becomes the differentiator
The AI industry understandably devotes enormous attention to hardware innovation, but the interview repeatedly returns to a different conclusion. Hardware alone cannot solve a systems problem.
Zheng draws an interesting parallel with the early evolution of hyperscale computing. During his time at Google, distributed storage systems were built from commodity hard drives that failed with remarkable regularity. The breakthrough did not come from eliminating hardware failures altogether. It came from building software capable of making those failures largely invisible to the applications using the infrastructure.
“I remember building distributed file systems where individual disks failed all the time,” he recalls. “Developers did not have to think about those failures because the software handled them. The concept is not new. What is different is that we now need to apply exactly the same thinking to large-scale AI infrastructure. Hardware resilience remains important, but software resilience becomes equally important because you are operating one very large distributed system.”
It is a comparison worth reflecting upon because it suggests AI infrastructure may be following a familiar pattern. The first generation focused on building faster hardware. The next generation will focus on ensuring increasingly complex collections of hardware behave as a coherent, dependable platform.
That evolution is likely to become even more important as networking itself emerges as one of the defining constraints on future AI performance. Zheng believes communication, rather than raw compute, will become the principal bottleneck as clusters continue growing across multiple racks and, increasingly, multiple data centres. Keeping every GPU continuously supplied with data becomes an infrastructure discipline, demanding software capable of orchestrating an environment that grows more interconnected and more complex with every generation.
The AI infrastructure industry has spent several years competing to build the biggest clusters. That race is far from over, but it is no longer the only contest that matters. As enterprises begin scrutinising the return on investments measured in hundreds of millions of pounds, attention is shifting towards a more demanding question. The winners will not necessarily be those who own the largest GPU estates. They are more likely to be the organisations capable of converting the greatest proportion of that infrastructure into productive work, delivering models faster, shortening innovation cycles and extracting commercial value from hardware that their competitors still allow to sit idle.
