Nvidia will remain the gold standard for AI training chips, CEO Jensen Huang told investors, even as rivals seek to chip away at its market share and one of Nvidia’s key suppliers gave a moderate forecast for AI chip sales.
Everyone from OpenAI to Elon Musk’s Tesla relies on Nvidia semiconductors to run their big language or computer vision models. The rollout of Nvidia’s “Blackwell” system later this year will only solidify that lead, Huang said at the company’s annual shareholder meeting on Wednesday.
Unveiled in March, Blackwell is the next generation of AI training processors that will succeed its flagship H100 “Hopper” line of chips, one of the tech industry’s most valuable commodities, fetching prices in the tens of thousands of dollars each.
“The Blackwell architecture platform is probably the most successful product in our history and indeed in the entire history of computing,” Huang said.
Nvidia briefly eclipsed Microsoft and Apple this month to become the world’s most valuable company in a remarkable rally that has fueled much of this year’s gains in the S&P 500 index. At more than $3 trillion, Huang’s company was at one point worth more than entire economies and stock markets, only to suffer a record loss in its market value as investors locked in profits.
Yet as long as Nvidia chips remain the benchmark for AI education, there is little reason to believe that the long-term outlook is murky and that the fundamentals continue to look strong.
One of Nvidia’s key advantages is a sticky AI ecosystem known as CUDA, short for Compute Unified Device Architecture. Just as ordinary consumers are reluctant to upgrade from their Apple iOS device to a Samsung phone running Google Android, a whole cohort of developers have been working with CUDA for years and feel so comfortable that there’s no reason to consider using another software platform. Just like hardware, CUDA has effectively become a standard in its own right.
“The Nvidia platform is widely available from all major cloud providers and computer manufacturers, creating a broad and attractive base for developers and customers, making our platform more valuable to our customers,” Huang added on Wednesday.
Micron’s Forecast for Next Quarter Revenue Not Enough for Bulls
The AI business recently suffered a blow after memory chip supplier Micron Technology, a supplier of high-bandwidth memory (HBM) chips to companies like Nvidia, forecast that AI revenue fourth quarter would only match market expectations of around $7.6 billion.
Micron shares fell 7%, far underperforming a slight gain in the broader tech-heavy Nasdaq Composite index.
In the past, Micron and its Korean rivals Samsung and SK Hynix have experienced cyclical boom-and-bust cycles common to the memory chip market, long considered a core business compared to logic chips such as graphics processors.
But enthusiasm has grown because of demand for its chips needed to train AI. Shares have more than doubled over the past 12 months, meaning investors have already priced in much of the growth management predicted.
“The forecasts were broadly in line with expectations, and in the AI hardware world, if you follow them, that’s considered a slight disappointment,” said Gene Munster, a technology investor at Deepwater Asset Management. “Momentum investors just didn’t see that additional reason to be more positive about the story.”
Analysts are closely tracking high-bandwidth memory demand as a leading indicator for the AI sector because it is crucial to solving the biggest economic constraint facing AI education today: the question of scaling.
HBM chips solve scaling problem in AI training
Costs do not increase with the complexity of a model (the number of parameters it has, which can reach billions), but rather exponentially. This results in diminishing efficiency returns over time.
Even if revenues grow at a steady pace, losses are likely to run into the billions or even tens of billions per year as the model evolves. This threatens to overwhelm any company that doesn’t have a deep-pocketed investor like Microsoft who can ensure OpenAI can still “pay the bills,” as CEO Sam Altman recently put it.
One of the main reasons for this decline in performance is the growing gap between the two factors that determine AI training performance. The first is the raw computing power of a logic chip, measured in FLOPS, a type of calculation per second, and the second is the memory bandwidth needed to quickly feed it data, often expressed in millions of transfers per second, or MT/s.
Since they work in tandem, scaling one without the other simply leads to waste and cost inefficiency. That’s why FLOPS, or the amount of compute that can actually be used, is a key metric for judging the cost-effectiveness of AI models.
Sold out until the end of next year
As Micron points out, data transfer rates have failed to keep up with the increasing pace of computing power. The resulting bottleneck, often referred to as a “memory wall,” is a major cause of today’s inherent inefficiency when scaling AI training models.
This explains why the US government focused heavily on memory bandwidth when deciding which specific Nvidia chips to ban from export to China in order to weaken Beijing’s AI development agenda.
On Wednesday, Micron said its HBM business was “burned out” through the end of the next calendar year, which is a quarter behind its fiscal year, echoing similar comments from Korean rival SK Hynix.
“We expect to generate several hundred million dollars of revenue from HBM in FY24 and several billion dollars of revenue from HBM in FY25,” Micron said Wednesday.