Meet Humpback: Meta’s New AI Model That’s a Whale of an Upgrade of Llama

Meta's whale-themed AI model makes a splash, self-training to follow instructions better than LLaMA

Ben Wodecki, Jr. Editor

August 16, 2023

3 Min Read

At a Glance

  • Meta unveils a new Humpback model that self-curates unlabeled data, reducing reliance on human annotations.
  • Built on top of Llama, Humpback outperforms the likes of Claude, Vicuna and Claude at following instructions.

AI researchers at Meta have showcased Humpback, a new AI model that can self-curate vast amounts of unlabeled data to generate outputs.

Humpback was unveiled in the paper ‘Self-alignment with instruction back translation’. The language model was built using a dataset comprising large amounts of unlabeled data. Humpback was built using an instruction-following dataset comprised of large amounts of unlabeled data.

Using a new method – dubbed instruction back translation – Meta’s AI researchers were able to build a Llama-based model capable of self-curating data by predicting its quality and then being self-trained on only the highest quality instructions.

The results saw Humpback outperform Anthropic’s Claude, Meta’s own LIMA model and all other Llama-based models.

Humpback is part of Meta’s continuing efforts to build auto-regressive large language models that can learn without the need for human intervention. Meta’s chief AI scientist and AI luminary Yann LeCun has continuously proposed the idea. He tweeted that Humpback was “making a splash with self-alignment.”

View post on X

How does Humpback AI work?

Meta's researchers took a base LLaMA model and gave it a small number of English language responses annotated by a human.

The resulting model was then used to generate instruction prompts for unlabeled web documents – showing self-augmentation.

Related:Meta Lays Off Team Behind Revolutionary Protein-folding Model

The model’s self-augmentation ability was used to create a sizable set of instruction pairs.

The model was then finetuned on a large unlabeled dataset of instruction examples from the Open Assistant dataset. Humpback was then tasked with picking high-quality examples effectively showing the model self-curating the data.

The researchers repeated the process to gradually improve the model's ability to follow instructions.

Meta’s approach in effect enables Humpback to progressively improve based on its own predictions, reducing reliance on external annotations.

Performance: How does Humpback compare?

Meta’s researchers compared Humpback with various other instruction-following models, including Alpaca, Vicuna and Falcon Instruct 40B.

In terms of generation quality, Humpback was among the top-performing models on the AlpacaEval test for instruction following models.

Humpback was the highest-scoring non-distilled AI model – meaning it was a model trained without relying on any external model like ChatGPT for any form of supervision. The only model to beat Humpback on the AlpacaEval test was OpenAI's GPT-4 which has limited resources available to determine its actual makeup.

Related:Meta Open Sources Text-to-Audio and Music Models

On the popular MMLU benchmark that covers natural language processing tasks, the 65 billion parameter version of Humpback achieved an average score of 59.0. By comparison, the standard LLaMA 65B achieved 54.8.

Move over camelids, it’s cetacean time

Jason Weston, a Meta AI researcher and one of the Humpback authors, said the method that made the model was “exciting because it could be scaled up further” using a stronger base model.

However, since the data used to build Humpback was sourced from a web corpus, it could perpetuate biases from web data.

Compared to the base LlaMA model, Humpback boasts improved accuracy in detecting biases – but that doesn’t mean it’s less likely to generate responses that contain biases.

Meta’s researchers tested its model on what they described as “potentially sensitive” prompts and found the model “model tends to produce a cautious response, or even refuses to provide information to fulfill the instruction.”

Humpback will surely make a splash for Meta and its efforts to make AI systems that learn in similar ways to how men do. The company's researchers suggest that future work utilizing the Humpback method of instruction back translation should scale to consider considering larger unlabeled corpora.

Read more about:

ChatGPT / Generative AI

About the Author(s)

Ben Wodecki

Jr. Editor

Ben Wodecki is the Jr. Editor of AI Business, covering a wide range of AI content. Ben joined the team in March 2021 as assistant editor and was promoted to Jr. Editor. He has written for The New Statesman, Intellectual Property Magazine, and The Telegraph India, among others. He holds an MSc in Digital Journalism from Middlesex University.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like