Google unveils language model that solves math problems

Minerva solved about a third of more than 200 STEM problems, at times through ‘incorrect’ reasoning.

Researchers from Google have unveiled Minerva: a language model that can solve math and science questions.

As showcased in a research paper, the model solves problems by generating solutions that include numerical calculations and symbolic manipulation.

Minerva was able to correctly answer one third of more than 200 undergraduate-level problems in physics, biology, chemistry and economics that required quantitative reasoning.

The model was evaluated across several STEM benchmarks – including MATH, MMLU-STEM and GSM8K. In all cases, the model obtained “state-of-the-art” results, the research team suggested.

Figure 1: Minerva generates multiple solutions to each question and picks the most common answer.

Google's researchers contend that, prior to Minerva, solving equations required natural language and mathematical notation, as well as the ability to recall relevant formulas and constants.

By focusing on collecting training data that is relevant for quantitative reasoning problems, training models at scale and employing certain inference techniques, developers can achieve performance gains for models attempting to undertake reasoning tasks.

In a blog post detailing the model, Ethan Dyer and Guy Gur-Ari, research scientists in Google Research’s Blueshift Team, wrote that at times, the Minerva can arrive at a correct final answer using incorrect reasoning steps – something that cannot be automatically detected.

Figure 2:

While this limitation is not present in formal methods for theorem proving, they argue that an informal approach can be “applied to a highly diverse set of problems [that] may not lend themselves to formalization.”

“While machine learning models have become impressive tools in many scientific disciplines, they are often narrowly scoped to solve specific tasks,” the blog post reads. “We hope that general models capable of solving quantitative reasoning problems will help push the frontiers of science and education.

“Models capable of quantitative reasoning have many potential applications, including serving as useful aids for researchers and enabling new learning opportunities for students. We present Minerva as a small step in this direction.”