Meta has just released Llama 3, the next generation of its state-of-the-art open source large language model (LLM). Llama 3 represents a major leap forward from the previous Llama 2 model in terms of capabilities and performance.
| Key Takeaway |
|---|
| Llama 3 is Meta’s new state-of-the-art open source large language model, available in 8B and 70B parameter versions. |
| It demonstrates breakthrough performance on benchmarks for reasoning, coding, question answering and other key capabilities compared to previous models. |
| Meta claims the 70B instruction-tuned Llama 3 outperforms GPT-3.5, Claude and others based on human evaluations. |
| Llama 3 incorporates innovations like efficient tokenization, grouped query attention, data filtering, scaling laws optimization, and advanced instruction fine-tuning methods. |
| The initial release focuses on English text generation, but future versions will add multilinguality, multimodality, longer context windows, and enhanced overall performance. |
| Meta is open sourcing Llama 3 and releasing tools like Llama Guard 2 to enable responsible development of apps using the model across cloud platforms, model hubs, and notebooks. |
| The release represents Meta’s commitment to an open source, community-driven approach for shaping the ethical trajectory of advanced AI capabilities like large language models. |
You can access the model here: https://www.meta.ai/ if it is available in your country.
Meta’s Transition to an AI Company
The release of Llama 3 is part of Meta’s broader strategic shift to position itself as an AI powerhouse. After facing challenges in its core social media business, the company formerly known as Facebook has been going all-in on artificial intelligence under the leadership of Mark Zuckerberg and Chief AI Scientist Yann LeCun.
Meta has been investing billions into AI research, hiring top talent from across the field, and building specialised AI supercomputers and data centers to train cutting-edge large language models like Llama 3. The company sees generative AI as a key part of its future roadmap for reimagining computing experiences.
The Open Source Approach to AI
Meta has been at the forefront of advocating for an open development approach to AI systems as an alternative to the previous industry trend of closed-source, proprietary models. By releasing models like Llama 2 and Llama 3 openly, Meta hopes to accelerate innovation, establish safety best practices, and shape the responsible development of advanced AI.
| Advantage | Description |
|---|---|
| Accelerates Innovation | By releasing models openly, Meta hopes to spur faster innovation across the AI ecosystem. |
| Establishes Safety Best Practices | An open approach allows collective learnings to shape responsible development practices. |
| Shapes AI Development | Meta aims to influence the trajectory of advanced AI systems through open sourcing. |
The company believes that openness will lead to better, safer, and more beneficial AI systems compared to closed development by companies going it alone. It’s a bold stance that differentiates Meta from the closed AI strategies of Big Tech competitors like Google, Apple, and Amazon.
Quest for AI That Augments Humans
While Meta is pushing the frontiers of large language model capabilities with Llama 3, the company’s ultimate vision is to create AI systems that can augment and empower humans rather than replicate or replace us. Meta sees generative AI assistants powered by models like Llama 3 as a breakthrough that can boost human intelligence and productivity.
The ambition is to build AI co-pilots that can understand natural conversations, retrieve relevant knowledge, analyze information, generate content, code software, and even learn new skills – all in service of helping humans get more done. Llama 3 takes a big step toward realizing that powerful human-AI collaboration.
What is Llama 3?
Llama 3 is a transformer-based language model that has been pre-trained on a massive 15 trillion token dataset. It is available in 8 billion and 70 billion parameter versions, which Meta claims establish new state-of-the-art performance for open source LLMs at those model sizes.
Some of the key innovations and improvements in Llama 3 include:
- A new 128K token vocabulary that encodes language more efficiently
- Grouped query attention (GQA) to improve inference efficiency
- Training on sequences up to 8,192 tokens with a mask to prevent crossing document boundaries
- Pretraining data filtered for quality and diversity across use cases
- Scaling laws used to optimize performance across capabilities like coding and reasoning
- A combination of supervised fine-tuning, rejection sampling, and reinforcement learning for instruction-tuning
Llama 3’s Impressive Performance Gains
Llama 3 demonstrates impressive performance gains over previous models on a wide range of benchmarks evaluating reasoning, coding, question answering, and other key capabilities. According to Meta’s human evaluation studies, the 70B instruction-tuned Llama 3 model outperforms models like GPT-3.5, Claude, and others in real-world scenarios.
Llama 3 is Fully Open Source and Broadly Available
Open Source and Broad Availability True to its open source roots, Meta is releasing Llama 3 models to the public to spur further innovation. The 8B and 70B versions are available now on all major cloud platforms, model hosting services like Hugging Face, coding notebooks like Kaggle, and more.
To support responsible development, Meta has also released new tools like Llama Guard 2 for content filtering, CyberSec Eval 2 to assess safety risks, and Code Shield to secure code generated by the model. Developers can leverage the torchtune library for training LLMs as well.
Llama 3 Capabilities that you can Access
Longer Context, Multilinguality, Multimodality Coming While the initially released versions are English-only text models, Meta says much more is coming for Llama 3 in the near future. Planned capabilities include multilinguality to converse in multiple languages, multimodality to handle images/video, a dramatically longer context window, and overall enhanced performance.
| Capability | Description | Status |
|---|---|---|
| Text Generation | State-of-the-art open source English language model | Available Now (8B, 70B) |
| Multilinguality | Converse fluently across multiple languages | Forthcoming |
| Multimodality | Perceive and generate images, video, etc. | Forthcoming |
| Expanded Context | Much longer context window for inputs | Forthcoming |
| Overall Enhanced Performance | Next-level general capabilities across the board | Forthcoming |
While the initially released versions are focused on English text, Meta says much more is coming for Llama 3 in the near future, including:
- Multilinguality to converse fluently in multiple languages
- Multimodal capabilities to perceive and generate images/video
- A dramatically expanded context window for much longer inputs
- Overall enhanced performance across the board
Meta’s largest Llama 3 models over 400B parameters are still training, but the company shared some preliminary results hinting at the strong multimodal, multilingual, and analytical capabilities these future versions may possess.
The release of Llama 3 represents a major milestone for open source AI. With its cutting-edge performance and Meta’s commitment to responsible, open development, Llama 3 could kick off a new wave of innovation in applications, tools, and AI system building using large language models.
If you want to learn more about the latest developments in AI, particularly around large language models like Llama 3, be sure to visit our AI blog.