OpenAI Proposes Explaining AI – With AI

As AI systems become bigger and more powerful, research teams have found it increasingly difficult to analyze how a model comes to a decision.

That is why OpenAI decided to speak power to power. The research lab behind ChatGPT revealed that it is using GPT-4, its largest and most powerful language model to date, to automate analysis of an older version, the GPT-2 model.

Its team of researchers proposed using AI to produce and score natural language explanations of the neurons behavior of a model and apply it to neurons in another language model.

This process would typically be done by humans, which take considerable time and does not scale well especially if neural networks have tens or hundreds of billions of parameters.

OpenAI proposes using GPT-4 to speed up the process, according to its paper, “Language models can explain neurons in language models.” The company said this approach could be used to “comprehensively audit the safety of models before deployment.”

GPT-4 to GPT-2: I am watching you

GPT-4, which was released in March, was used to measure GPT-2’s ability to compress and reconstruct neuron activations using natural language. GPT-2, released back in 2019, is far simpler compared to its successor models, housing just 1.5 billion parameters.

However, while this concept works on smaller models like GPT-2, OpenAI said the technique does “poorly for larger models, possibly because later layers are harder to explain.”

Nevertheless, “although the vast majority of our explanations score poorly, we believe we can now use ML techniques to further improve our ability to produce explanations,” the researchers claimed.

OpenAI said it is open sourcing the datasets and visualization tools for GPT-4-written explanations of all 307,200 neurons in GPT-2. Also being published was the code for explanation and scoring using publicly available models on the OpenAI API.

“We hope the research community will develop new techniques for generating higher-scoring explanations and better tools for exploring GPT-2 using explanations,” according to a company blog post.