Google Unveils Open Source Models to Rival Meta, Mistral

Google released a new family of open source language models today, in a salvo aimed at Meta, Mistral and other open source rivals.

The search giant took the wraps off Gemma, its line of open source language models that is based on Gemini, Google’s most powerful large multimodal model to date. However, Gemma is text and code only.

Google released two sizes of what it calls “state of the art” Gemma models: the 2-billion parameter model and 7-billion parameter model.

The most popular rival open source models in these weights are those from Meta’s Llama 2 and Mistral.

But Google said Gemma outperforms or holds its own against Llama 2 (both 7 billion and 13 billion parameters) and Mistral (7 billion parameters) in dialogue, reasoning, math and coding.

Credit: Google

Google said the 2-billion parameter Gemma model runs on CPUs for on-device or edge applications in hardware such as laptops. The 7-billion parameter model runs on GPUs and TPUs. The models can also be accessed through Google Cloud’s Vertex AI platform and Google Kubernetes Engine.

Google allows “responsible” commercial use of Gemma and distribution for organizations of all sizes. In contrast, Meta restricts free use of its Llama 2 family of language models to organizations with fewer than 700 million monthly active users. Otherwise, they must get a license.

Google is giving researchers free access in Kaggle, a free tier for Colab notebooks and $300 in credits for first-time Google Cloud users. Researchers can also apply for Google Cloud credits of up to $500,000 for their projects.

To get started in Gemma, head here.

“Google's announcement of Gemma 2B and 7B is a sign of the fast-growing capabilities of smaller language models. A model being able to run directly on a laptop, with equal capabilities to Llama2, is an impressive feat and removes a huge adoption barrier for AI that many organizations possess," said Iris.ai CTO Victor Botev, in emailed comments to AI Business.

"Bigger isn’t always better. Practical application is more important than massive parameter counts, especially when considering the huge costs involved with many large language models (LLMs)," he added.

“To truly unlock the potential of smaller models, the user experience and interfaces must also be tailored to specific domains. Purpose-built interfaces and workflows allow for more successful use rather than expecting a monolithic model to excel at all tasks. With applications across the sciences, business and more in mind, front-end specialization is key to realizing the power of these nimble yet advanced models."

Inside the Gemma model

Gemma 2B was trained on two trillion tokens of text while 7B was trained on six trillion of primarily English data, “using similar architectures, data and training recipes” as Gemini, Google DeepMind researchers said in their paper, ‘Gemma: Open Models Based on Gemini Research and Technology.’ Its architecture is based on the transformer decoder.

Google is also releasing the model weights, with pre-trained and instruction-tuned variants. Also offered are toolchains for inference and supervised fine-tuning across JAX, PyTorch, TensorFlow through native Keras 3.0 and other major frameworks. Here is Gemma’s model card.

Gemma is integrated with Hugging Face, MaxText, NVIDIA NeMo, TensorRT-LLM, among others.

Google also is releasing a Responsible Generative AI Toolkit for users to create safer applications. “We believe the responsible release of LLMs is critical for improving the safety of frontier models, for ensuring equitable access to this breakthrough technology, for enabling rigorous evaluation and analysis of current techniques, and for enabling the development of the next wave of innovations,” the researchers said.

As for privacy, Google said it filters out sensitive data and other personal information before training, using automated techniques.

Google said it embedded safety measures in Gemma to mitigate potential risks that might come up, recognizing that releasing the open source models is “irreversible.” It also urged the open source community to “move beyond simplistic ‘open vs. closed’ debates” and stay away from “exaggerating or minimizing potential harms.” Instead, Google recommends a “nuanced, collaborative approach to risks and benefits.”

But Melissa Ruzzi, director of AI at AppOmni, told AI Business in an email that "it is nearly impossible to implement enough controls to keep (bad actors from using AI for attacks) and still have a functional model. This is the biggest problem to solve with open source AI models."