Using AI to explain how AI came to a decision about AI

Ben Wodecki, Jr. Editor

May 10, 2023

2 Min Read

At a Glance

  • ChatGPT-maker OpenAI proposes using AI models to analyze other AI model’s effectiveness based on natural language outputs.

As AI systems become bigger and more powerful, research teams have found it increasingly difficult to analyze how a model comes to a decision.

That is why OpenAI decided to speak power to power. The research lab behind ChatGPT revealed that it is using GPT-4, its largest and most powerful language model to date, to automate analysis of an older version, the GPT-2 model.

Its team of researchers proposed using AI to produce and score natural language explanations of the neurons behavior of a model and apply it to neurons in another language model.

This process would typically be done by humans, which take considerable time and does not scale well especially if neural networks have tens or hundreds of billions of parameters.

OpenAI proposes using GPT-4 to speed up the process, according to its paper, “Language models can explain neurons in language models.” The company said this approach could be used to “comprehensively audit the safety of models before deployment.”

GPT-4 to GPT-2: I am watching you

GPT-4, which was released in March, was used to measure GPT-2’s ability to compress and reconstruct neuron activations using natural language. GPT-2, released back in 2019, is far simpler compared to its successor models, housing just 1.5 billion parameters.

However, while this concept works on smaller models like GPT-2, OpenAI said the technique does “poorly for larger models, possibly because later layers are harder to explain.”

Nevertheless, “although the vast majority of our explanations score poorly, we believe we can now use ML techniques to further improve our ability to produce explanations,” the researchers claimed.

OpenAI said it is open sourcing the datasets and visualization tools for GPT-4-written explanations of all 307,200 neurons in GPT-2. Also being published was the code for explanation and scoring using publicly available models on the OpenAI API.

“We hope the research community will develop new techniques for generating higher-scoring explanations and better tools for exploring GPT-2 using explanations,” according to a company blog post.

Read more about:

ChatGPT / Generative AI

About the Author(s)

Ben Wodecki

Jr. Editor

Ben Wodecki is the Jr. Editor of AI Business, covering a wide range of AI content. Ben joined the team in March 2021 as assistant editor and was promoted to Jr. Editor. He has written for The New Statesman, Intellectual Property Magazine, and The Telegraph India, among others. He holds an MSc in Digital Journalism from Middlesex University.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like