Memory Is the New Oil: Why the AI Industry May Be About to Split in Two
Notes by SDG ·
It flipped how I look at the AI race. We obsess over who has the smartest model - ”GPT-5, Claude, Gemini, Llama” but this paper argues we are looking at the wrong bottleneck. The next few years won't be won by the cleverest algorithms, but by whoever controls memory bandwidth.
Here is my breakdown of how memory scarcity and open-source models are quietly restructuring the entire AI landscape.
1. Forget "Compute" - ”It's a Highway Problem
For years, we have treated AI progress as a pure compute problem: buy more GPUs, crank out more FLOPs, and build bigger clusters.
But during inference (running the models), GPUs actually spend most of their time waiting around. They are bottlenecked by how fast they can pull model weights and KV cache from High Bandwidth Memory (HBM). Because of this:
- The new metric that matters: We need to stop looking at cost per FLOP and start looking at the cost per petabyte of memory bandwidth delivered.
- The analogy: Think of a trucking company. Your bottleneck isn’t how powerful your truck engines are; it’s how many highway lanes are open to let them move. HBM is the highway.
2. The HBM Moat and the "Depreciation Conveyor"
Hardware usually gets cheaper over time, but HBM is bucking the trend. Because everyone is screaming for it at once, prices have skyrocketed, making memory a massive chunk of accelerator costs. This creates an unfair advantage for early players:
- The Cheap Fleet Advantage: If Company A bought 100,000 GPUs two years ago, and Company B buys the exact same setup today at inflated HBM prices, Company A can serve the same models for vastly cheaper.
- The Depreciation Conveyor: Incumbents constantly inherit depreciated, "paid-off" hardware fleets. New entrants are forced to finance today’s hyper-expensive hardware, making it almost impossible to compete on price. It's like trying to launch a new airline when your competitor already owns their entire fleet outright.
3. AI Becomes the Airline Industry
We used to assume closed frontier models would always stay miles ahead of open-source. That gap is closing fast. Open-source models have gotten incredibly good and radically cheaper to run.
Since businesses buy outcomes rather than raw, generalized intelligence, a model that solves 95% of their problems at 10% of the cost will win almost every time. This splits the market:
- Luxury/Frontier: Think First Class. Highly capable, expensive, closed models (like OpenAI or Anthropic) used for incredibly complex tasks.
- Mass-Market/Open: Think Economy. "Good enough" open models running commodity workloads at scale. Both survive, but the bulk of the volume goes to economy.
4. Training is Priceless, Copying is Cheap
The paper highlights a wild divergence: the cost to train a brand-new, cutting-edge frontier model is exploding because frontier ambitions are outrunning hardware cost declines.
At the same time, replicating yesterday's frontier performance using techniques like reinforcement learning, distillation, and open-source foundation models is getting incredibly cheap. The cost gap between pioneering and cloning is projected to widen from a factor of tens today to potentially thousands of times by 2030.
5. The Infrastructure Paradox: Is Efficiency a Threat?
Right now, tech giants are pouring trillions into data centers. But what happens if AI software gets too efficient?
If next-generation models require half the memory bandwidth, or if routing systems compress and shrink workloads, we might actually end up with too much supply. Matsuoka’s paper estimates that current infrastructure plans require roughly 2x annual token-demand growth just to stay comfortably utilized. If demand slows down or efficiency spikes too fast, those shiny new data centers could become incredibly expensive ghost towns.
6. The Real Winners Aren't Model Makers
If models are commoditizing, where does the money go? The paper suggests value is shifting upward to two places:
The Orchestration Layer
Instead of picking one model, enterprises will use smart routing systems. The winner will be the "Google Flights" of AI”the system that automatically decides:
- When to use a cheap open model.
- When to run a task locally on your device.
- When to escalate to a premium frontier model.
Local and Edge AI
Not everything belongs in the cloud. As open models shrink, running them locally on enterprise servers or personal devices becomes the default for privacy, latency, and compliance. This doesn't reduce AI usage; it just changes who gets paid (think device makers and local IT instead of cloud giants).
7. Five Scenarios for the Future of AI
The paper does not make a single bet. Instead, it outlines five potential economic futures for the industry:
Scenario
What It Means
1. Rotating Landlord Oligopoly
The players who own the physical infrastructure and HBM reap almost all the economic rewards.
2. Jevons Absorption
Demand for AI tokens grows so fast that it completely overwhelms any efficiency gains, keeping all data centers maxed out.
3. System-Layer Re-differentiation
Value shifts entirely away from models and onto the orchestration, workflows, and routing systems.
4. Commoditization Crash
Open-source efficiency and routing reduce the profitability of infrastructure, leading to a massive market correction.
5. Geopolitical Bifurcation
The global AI supply chain splits into heavily protected, isolated regional ecosystems.
My Biggest Takeaway
This paper made me realize that leadership in the next era of AI isn't about who can build the most brilliant digital mind. It's about who can orchestrate infrastructure, run at the lowest cost, and seamlessly blend AI into everyday workflows.
It also explains why a giant like Microsoft is playing a genius long game. Their ultimate value doesn't just hinge on their partnership with OpenAI; it's anchored in Windows, Office, Azure, and security. They already own the workflows and the distribution. Even if models become a dirt-cheap commodity, the players who control the pipes and the integration will still run the show.