At a high-profile AI event in London, Meta executives on Tuesday provided the first official confirmation and details about the upcoming release of Llama 3, the next version of the company’s open source large language model.
Nick Clegg, Meta’s president of global operations, said in a statement that Meta “hopes to start rolling out a new family of Llama 3 models, the next-generation base model, within the next month, actually less time, hopefully in a very short period of time.” . AI Day London, reported Tech Crunch.
Clegg said the Llama 3 will be comprised of “a variety of models with a variety of features and versatility” and will be released within the year.
Once released, Llama 3 is expected to be the most advanced open source model yet, with Meta investing heavily in development. According to Meta, the model was trained with 140 billion parameters, twice the capacity of Llama 2. Meta CEO Mark Zuckerburg revealed some technical details in January.
“We are building a large compute infrastructure to support our future roadmap, including 350,000 H100s by the end of this year, or approximately 600,000 H100 equivalents overall, including other GPUs,” Zuckerberg said at the time. . This computing power is significantly more than what OpenAI uses to train GPT-4 (estimated to require approximately 25,000 GPUs for 90 to 100 days).
Zuckerberg also revealed that the AI assistant, Meta AI, will be powered by Llama 3.
Chief Product Officer Chris Cox said Llama 3 will be integrated throughout Meta.
“Our plan is to ensure that Llama 3 offers a variety of products and experiences across our app suite,” he said.
open source strategy
The impact of the launch of Llama 3 extends far beyond Meta, given the company’s philosophical commitment to developing under an open source model, in clear contrast to the closed and proprietary approach taken by competitors such as OpenAI and ChatGPT.
By open-sourcing its language model, Meta aims to foster an open AI development ecosystem and position the Llama suite as the foundation for a wide range of tools and applications created by third-party developers and researchers.
“It’s very important to realize that innovation is always based on others’ prior contributions, sometimes very similar ones,” Yann LeCun, head of AI research at Meta, wrote on Twitter last month. “This is why open research is important – it allows everyone to advance the field faster.”
From afar, innovation appears to emerge spontaneously in a vacuum.
However, it is very important to recognize that innovation always builds on the prior contributions of others, sometimes very similar contributions.
This is why open research is important. The field… https://t.co/JMvQD2h5OZ— Yann LeCun (@ylecu) March 20, 2024
This open spirit has already created a vibrant community that gathers around Llama. Some of today’s most advanced open source language models, including Mistral, Falcon, and Beluga, were built by fine-tuning older Llama 2-based models. Some of these community models have matched or outperformed GPT-3.5 on certain benchmarks.
The release of Llama-3, another open-source baseline model, will likely pave the way for the next generation of LLM, which will set the bar even higher in terms of quality and efficiency in AI.
Challenge OpenAI dominance
Llama 3’s open source premise poses a powerful, multi-layered challenge not only to OpenAI’s current market dominance, but also to other proprietary models such as Claude and Gemini.
The open source community will soon be able to build on Llama 3 and quickly iterate on variations to potentially match or exceed the capabilities of GPT-4, just as they did with GPT-3.5. Because of the low cost of training shared across contributors, the open ecosystem can go beyond OpenAI’s proprietary model development, which requires enormous computing resources and costs.
If open source products regularly achieve parity with commercial counterparts, companies may gravitate toward more accessible and cost-effective ecosystems like Llama rather than relying on and paying for OpenAI. Currently, GPT-4 is the most expensive model on the market in terms of cost per token.
Additionally, open source communities become stronger as more people participate. Meta benefits by building a huge community on top of the model, fine-tuning it, developing new technologies, and improving it for free. This makes it easier for Meta to develop better versions of the model while also monetizing the model through alternative schemes, such as licensing it for commercial use by large-scale industries.
That means continued inertia and network effects could make it harder for OpenAI’s proprietary model to attract users and customers in the future.
Clearly, OpenAI currently holds a strong lead in terms of profitability. Anthropic can boast of having a top-performing LLM in AI. But Llama 3 will likely represent another strategic attack by Meta to upend the generative AI landscape.
Of course, a lot will depend on the actual performance and adoption of Llama 3 next year. However, the open source AI community is very active and already likes Llama-2. Things are going to get very interesting in the coming months. In particular, OpenAI’s GPT-5 is scheduled to be released soon.
Edited by Ryan Ozawa.