As the cost of powerful graphics processing needed for large language models (LLMs) have ballooned, OpenAI CEO Sam Altman recently announced at an MIT event that the era of ever-growing AI models is hit its limit. Altman indicated that further progress would come from improving model architectures, data efficiency, and developing innovative algorithmic techniques, instead of relying on merely copying and scaling existing models.
The primary driver of the reversal in the “scaling is all you need” approach is related to the skyrocketing expenses that come with training and operating massive language models. For instance, ChatGPT – a popular LLM – requires over ten thousand GPUs to run, and correspondingly more resources to maintain. Bought from the US’s main graphics provider, Nvidia, a single H100 GPU costs up to $30,603 – and that is up for negotiations on eBay. Consequently, the price of running a cutting-edge LLM is estimated to be hundreds of millions of dollars.
Elon Musk is also making a statement by preparing to invest in an AI company called X.ai. During his recent Twitter Spaces interview, Musk noted that both his companies – Tesla and Twitter – are planning on buying thousands of GPUs to accelerate development. However, Ronen Dar of Run AI explained that GPUs of such size won’t be readily available, and it can take up to months to source example the amount that Musk and his team need.
Gomez of Cohere countered Altman and suggested the limitation in AI model scaling is rather due to the fact that it is possible that projects have been making models excessively large, referencing Altman’s comment at the MIT event that “we may have made models larger than necessary” and “parameters count will trend up for sure”, as well as his analogy with the “Gigahertz race in chips in the 1990s and 2000s.”
OpenAI is one of the largest organizations in the space of AI research, headed by Sam Altman who, since his appointment in 2014, has had an enormous influence over the space. OpenAI has grown rapidly since its creation, but, as Altman has revealed, the company is finally reaching the point where “scaling is all you need” no longer applies. With the dramatic influx of GPUs in the market and their challenging costs of acquisition, progress in AI may be redirected away from large-scale models towards refining the resources involved with AI and ML.
Elon Musk is no stranger to incredible investments, and his latest move to purchase tens of thousands of GPUs is yet another testament to his ambition. However, due to the current constraints of the market, the timeline of Musk and his team’s AI project may be dependent on the availability of their choice graphics processors. Musk himself seemed to be aware of this challenge, emphasizing the fact that many companies are currently struggling for GPUs and he may have to wait for his.
All in all, OpenAI’s CEO has drawn attention to the difficulties related to working with large language models, from obtaining the necessary GPUs to the financial obstacles and diminishing returns that come with scaling models. The AI space is rapidly shifting away from overwhelmingly large models, and the competition for GPU resources may reveal the next creative steps taken to push the field forward.
