The massive Llama 3.1 405B model challenges GPT-4 and Claude 3.5 while remaining open source and freely accessible
Meta has unveiled Llama 3.1 405B, a giant 405 billion parameter model that stands as the largest open source AI system in history.
Meta only launched the Llama 3 line of models in April but has since been working on a mammoth version of the model that’s a whopping 400 billion parameters in size.
The company unveiled that giant model, which stands at 405 billion parameters, an enormous AI system that Meta described as “in a class of its own.”
Meta’s latest release offers a powerful underlying model for use cases including multilingual conversational agents or long-form text summarization, with a revised Stack API enabling users to implement it easily.
Meta said the new giant model will let the AI research community “unlock new workflows, such as synthetic data generation and model distillation.
The massive Llama model is designed to compete with foundation-level models, including OpenAI’s GPT-4 and Anthropic’s new Claude 3.5 while remaining open source.
“I believe the Llama 3.1 release will be an inflection point in the industry where most developers begin to primarily use open source, and I expect that approach to only grow from here,” Meta CEO Mark Zuckerberg wrote on Facebook.
Before this release, the largest published Llama model boasted 70 billion parameters.
The new Llama 3.1 405B towers over OpenAI's GPT-3, which contains 175 billion parameters.
While OpenAI has never revealed GPT-4's size, industry speculation suggests it could reach into the trillions of parameters. However, this remains unconfirmed due to the company's secretive approach.
Despite potentially being smaller than OpenAI’s model, Meta says its new Llama model rivals closed systems like GPT-4 across a range of tasks including general knowledge, steerability, math, tool use and multilingual translation.
Benchmark results show Llama 3.1 405B beats models like Claude 3.5 and GPT-4o at tests including GSM8K and Nexus while remaining competitive on industry-standard tests including HumanEval and MMLU.
Credit: Meta
The model was trained on more than 15 trillion tokens, a process that took several months using 16,000 H100 GPUs from Nvidia.
It has a context length of 128,000 tokens, the equivalent of 96,241 words. While not as large as Gemini 1.5 Pro’s lauded 2 million context length, Meta's new Llama model has improved reasoning capabilities meaning it can process and understand longer sequences of text more effectively.
Instead of designing a complex system underneath the giant new model, Meta’s AI engineers instead employed a standard decoder-only transformer model architecture adapted with minor changes.
The team that built the new model also undertook several rounds of post-training, creating synthetic data that was used to tweak and progressively improve the model’s performance.
Upon unveiling Llama 3.1 405B, Meta emphasized the model was designed safely.
The larger a model, the more difficult it can be to manage how safe it is likely to be given the sheer mass of data that’s fed into it.
Before releasing the new model, the company said it conducted several risk assessments and safety evaluations as well as extensive red-teaming with both internal and external experts to stress test it.
Meta’s researchers also scaled its evaluations across the model’s multilingual capabilities to ensure outputs were safe and sensible beyond English.
Zuckerberg said open source models like the new Llama 3.1 405B should be safer than closed model systems as they’re “more transparent and can be widely scrutinized.”
Meta said it also added a series of safety measures including the new prompt injection filter, though the company claims this does not impact the quality of responses.
Like all Llama models before it, the giant 3.1 system is open source and available to anyone.
It can be downloaded from Hugging Face, GitHub or directly from Meta.
The model is also accessible from several cloud providers, including AWS, Nvidia, Microsoft Azure and Google Cloud, among others.
Given its size, users will need sizable hardware to run it, an issue that could limit accessibility, according to Victor Botev, Iris.ai’s co-founder and CTO.
“Many researchers and organizations lack the infrastructure to effectively utilize such massive models,” Botev said. The environmental impact of training and running these behemoths is a growing concern that cannot be ignored.
“Innovations in model efficiency might benefit the AI community more than simply scaling up to larger sizes. Achieving similar or superior results with smaller, more manageable models would not only reduce costs and environmental impact but also make advanced AI more accessible to a broader range of users and applications.”
Read more about:
ChatGPT / Generative AIYou May Also Like
