Meet Humpback: Meta’s New AI Model That’s a Whale of an Upgrade of Llama

AI researchers at Meta have showcased Humpback, a new AI model that can self-curate vast amounts of unlabeled data to generate outputs.

Humpback was unveiled in the paper ‘Self-alignment with instruction back translation’. The language model was built using a dataset comprising large amounts of unlabeled data. Humpback was built using an instruction-following dataset comprised of large amounts of unlabeled data.

Using a new method – dubbed instruction back translation – Meta’s AI researchers were able to build a Llama-based model capable of self-curating data by predicting its quality and then being self-trained on only the highest quality instructions.

The results saw Humpback outperform Anthropic’s Claude, Meta’s own LIMA model and all other Llama-based models.

Humpback is part of Meta’s continuing efforts to build auto-regressive large language models that can learn without the need for human intervention. Meta’s chief AI scientist and AI luminary Yann LeCun has continuously proposed the idea. He tweeted that Humpback was “making a splash with self-alignment.”

How does Humpback AI work?

Meta's researchers took a base LLaMA model and gave it a small number of English language responses annotated by a human.

The resulting model was then used to generate instruction prompts for unlabeled web documents – showing self-augmentation.

Related:Meta Lays Off Team Behind Revolutionary Protein-folding Model

The model’s self-augmentation ability was used to create a sizable set of instruction pairs.

The model was then finetuned on a large unlabeled dataset of instruction examples from the Open Assistant dataset. Humpback was then tasked with picking high-quality examples effectively showing the model self-curating the data.

The researchers repeated the process to gradually improve the model's ability to follow instructions.

Meta’s approach in effect enables Humpback to progressively improve based on its own predictions, reducing reliance on external annotations.

Performance: How does Humpback compare?

Meta’s researchers compared Humpback with various other instruction-following models, including Alpaca, Vicuna and Falcon Instruct 40B.

In terms of generation quality, Humpback was among the top-performing models on the AlpacaEval test for instruction following models.

Humpback was the highest-scoring non-distilled AI model – meaning it was a model trained without relying on any external model like ChatGPT for any form of supervision. The only model to beat Humpback on the AlpacaEval test was OpenAI's GPT-4 which has limited resources available to determine its actual makeup.

Related:Meta Open Sources Text-to-Audio and Music Models

On the popular MMLU benchmark that covers natural language processing tasks, the 65 billion parameter version of Humpback achieved an average score of 59.0. By comparison, the standard LLaMA 65B achieved 54.8.

Move over camelids, it’s cetacean time

Jason Weston, a Meta AI researcher and one of the Humpback authors, said the method that made the model was “exciting because it could be scaled up further” using a stronger base model.

However, since the data used to build Humpback was sourced from a web corpus, it could perpetuate biases from web data.

Compared to the base LlaMA model, Humpback boasts improved accuracy in detecting biases – but that doesn’t mean it’s less likely to generate responses that contain biases.

Meta’s researchers tested its model on what they described as “potentially sensitive” prompts and found the model “model tends to produce a cautious response, or even refuses to provide information to fulfill the instruction.”

Humpback will surely make a splash for Meta and its efforts to make AI systems that learn in similar ways to how men do. The company's researchers suggest that future work utilizing the Humpback method of instruction back translation should scale to consider considering larger unlabeled corpora.