The tech industry is currently comforting itself with a massive delusion. The narrative goes something like this: training massive AI models requires monolithic, gigawatt-scale data centres, but executing those models—the inference phase—can be downsized, decentralized, and pushed to the edge. Industry pundits suggest that smaller, nimbler facilities will soon inherit the earth because inference takes less computational heavy lifting than training.
They are dead wrong.
This downsizing myth is based on a fundamental misunderstanding of how AI workloads scale, how network topology operates, and how enterprise economics actually work. I have spent two decades designing infrastructure and watching enterprises throw millions of dollars down the drain chasing the latest hardware fad. The belief that inference will allow us to shrink our data centre footprint is the next multi-billion-dollar mistake.
Inference is not a lightweight afterthought. It is a data-hungry, latency-sensitive monster that will require larger, more centralized, and more dense infrastructure than anything we have built to date.
The Flawed Logic of the Light Inference Workload
The core argument for downsizing relies on a simplistic equation: training requires thousands of GPUs running for months, while inference only requires a fraction of a second of compute per request. Therefore, buy smaller servers, stick them in regional hubs, and watch the savings roll in.
This ignores the brutal reality of concurrency.
Training a model is a predictable, bounded workload. You know exactly how much data you have, how many parameters you are adjusting, and how long the cluster will run. Inference is the exact opposite. It is chaotic, unpredictable, and subject to massive spikes in user demand.
When millions of users simultaneously query an enterprise AI application, the aggregate compute requirement doesn't just equal training levels—it blows past them. A single user asking a model to analyze a legal document is cheap. Ten thousand enterprise clients asking that same model to analyze compliance documents across their entire history simultaneously creates an infrastructure bottleneck that crushes small-scale nodes.
Furthermore, we are moving rapidly away from single-turn text generation. The current state of the art relies on agentic architectures. These systems do not just spit out the next word; they reason, use tools, call external APIs, and run internal verification loops before returning an answer. A single user prompt can trigger dozens of internal inference cycles.
Imagine a scenario where an automated financial auditor agent is tasked with reviewing an acquisition. It doesn't run one inference query. It runs thousands of autonomous sub-queries, cross-referencing ledger entries, regulatory codes, and market data. The compute footprint of that single task looks less like a simple search query and more like a localized training run. If you try to execute that on a scaled-down, regional data centre, your application will crawl to a halt.
The Memory Bandwidth Trap
To understand why downsizing fails, you have to look at the silicon level. The primary bottleneck in AI inference is not raw compute power; it is memory bandwidth.
Large Language Models (LLMs) have billions of weights. During inference, every single one of these weights must be loaded from memory into the processor cache to generate a single token. If your model has 70 billion parameters, you need to move roughly 140 gigabytes of data through the processor just to generate one word.
This means inference hardware requires massive memory architectures like High Bandwidth Memory (HBM). This hardware is expensive, runs hot, and requires immense power delivery systems. You cannot easily distribute these systems into small, uncooled server closets or low-spec regional hubs without sacrificing performance or driving up maintenance costs exponentially.
The physics of hardware deployment dictate that high-density configurations are always more efficient. Consolidating high-bandwidth hardware into massive facilities allows operators to optimize power usage effectiveness (PUE) and deploy advanced liquid cooling at scale. Splitting that infrastructure into hundreds of smaller sites multiplies your overhead, creates massive cooling inefficiencies, and increases your surface area for hardware failure.
The Edge Inference Lie
Proponents of downsizing frequently point to the edge. They claim that local devices—smartphones, laptops, and branch-office servers—will handle the bulk of inference, removing the burden from central data centres.
This completely misunderstands enterprise security, data gravity, and the reality of model fragmentation.
- Data Gravity: AI models are useless without context. To give a precise answer, an enterprise model needs access to internal databases, real-time telemetry, and customer histories. Moving the model to the edge means you either have to constantly sync terabytes of corporate data to every local node, or you have to stream that data back and forth over the network. The moment you start streaming data back to a central repository to inform an edge inference step, you have destroyed any latency advantages the edge was supposed to provide.
- Security and IP Protection: Proprietary models are corporate crown jewels. Shipping a fine-tuned, highly specialized model out to edge devices or distributed regional facilities is an information security nightmare. Reverse-engineering a model from physical hardware access is a known vector. Enterprises will keep their most valuable models locked inside highly secure, centralized fortresses.
- The Power Penalty: A localized micro data centre cannot match the efficiency of a hyperscale facility. Deploying a dozen small sites across a country means paying localized utility rates, dealing with fragmented grid connections, and maintaining twelve separate cooling loops instead of one optimized system.
Dismantling the Latency Argument
The most common question raised by advocates of downsizing is: "How can you deliver real-time AI responses if everything has to travel back to a massive central data centre?"
The premise of the question is fundamentally flawed. It assumes that network transit time is the primary component of AI latency. It isn't.
Right now, the time to first token (TTFT) and the overall generation speed are overwhelmingly dominated by compute latency—the time it takes the silicon to process the request and move data through memory. Shaving 15 milliseconds off your network ping by placing a server in a regional hub means nothing if the server itself takes 800 milliseconds to process the agentic reasoning loops because it lacks the high-density clustering found in a massive data centre.
Users do not care if their data travels 50 miles or 500 miles. They care about how fast the system returns a complete, intelligent response. Centralized hubs allow for massive clusters where models can be sharded across multiple top-tier chips using ultra-fast interconnects like NVLink. This internal clustering speeds up processing times far more than geographic proximity ever could.
The Cost of the Contrarian Reality
Let us be completely transparent about the downsides of keeping things big. Double downing on massive, centralized data centres creates immense challenges that the industry is currently struggling to solve.
+----------------------------+----------------------------+
| Centralized Super-Hubs | Distributed Micro-Nodes |
+----------------------------+----------------------------+
| High upfront capital expense| Excessive operational ruin |
| Grid capacity bottlenecks | Nightmare logistics |
| Extreme cooling density | High security vulnerabilities|
| Maximum compute speed | Severe processing bottlenecks|
+----------------------------+----------------------------+
| Winner | Loser |
+----------------------------+----------------------------+
The power grid is the biggest hurdle. Securing 100 megawatts or a gigawatt of power for a single site is becoming nearly impossible in major technological hubs. Companies are being forced to look at alternative energy integration, such as pairing data centres directly with small modular nuclear reactors (SMRs) or geothermal plants.
This is incredibly difficult, capital-intensive work. But the alternative—spreading those megawatts across a hundred smaller, inefficient facilities—is an operational nightmare that compounds your costs through logistical friction, fragmented maintenance teams, and sub-optimal hardware utilization.
Stop Planning for a Smaller Footprint
If you are an enterprise technology leader planning your budget for the next five years, ignore the sirens singing about the downsizing of the data centre.
Do not invest millions in building out a fragmented network of regional micro-nodes under the assumption that AI inference will become lightweight enough to run on scraps. The models will get larger, the agentic loops will get deeper, and the data dependencies will become tighter.
The companies that win the AI race will not be those that tried to shrink their infrastructure to fit the legacy architectures of yesterday. The winners will be those that accept the reality of massive, high-density scale and build the massive power and cooling infrastructure required to feed the monster.
Stop trying to downsize your infrastructure. Build bigger, build denser, and prepare for a world where compute demands are absolute.
