AI Pioneer Bengio Joins Project Developing AI Safety Standards

Turing Award winner and deep learning pioneer Yoshua Bengio has joined Safeguarded AI as scientific director. 

The U.K. government-backed initiative aims to develop an AI system capable of monitoring and mitigating risks posed by other AI agents, effectively creating a "gatekeeper" for AI safety.

The gatekeeper system would be able to, hypothetically, monitor and understand autonomous AI agent interactions, ensuring they only operate within set safety guardrails.

A thesis penned by the group suggests the gatekeeper system would “not only reduce the risks of frontier AI and enable its use in safety-critical applications, they would also unlock the upside of frontier AI in business-critical applications and commercial activities where reliability is key.

Safeguarded AI has received a substantial $75 million investment from the U.K. government’s Advanced Research and Invention Agency, providing financial support for the next four years.

Now it has one of the biggest names in AI helping it.

The group said Bengio will provide the project with “scientific and strategic advice” to program director David “davidad” Dalrymple and the wider Safeguarded team.

“Every time Yoshua and I get the opportunity to exchange technical ideas, our R&D plans increasingly converge—and to the extent they still differ, we now both want to try pursuing both,” said Dalrymple.

Related:A Bold Call from OpenAI, Safety Crusaders: Limit Compute

Bengio won the 2018 Turing Award, commonly known as the “Nobel Prize of Computing,” alongside other AI godfathers Geoffrey Hinton and Yann LeCun, for groundbreaking work on deep neural networks.

Bengio would become one of the more cautious researchers of the trio when it comes to AI safety.

While serving as an advisor to former U.K. Prime Minister Rishi Sunak on AI safety, Bengio recently joined calls for global AI regulations to focus on limiting compute rather than placing restrictions on software.

He has since co-authored the paper with davidad and others, proposing the creation of a family of AI safety approaches called “Guaranteed Safe AI.” 

Guaranteed Safe AI aims to provide rigorous, high-confidence safety guarantees for AI systems especially those with high autonomy or what some might describe as artificial general intelligence (AGI), as well as systems used in safety-critical contexts.

It requires three essential components:

A world model: A mathematical description of how the AI system affects the outside world, accounting for both Bayesian and Knightian uncertainty.

Safety specifications: A mathematical description of what effects are considered acceptable.

Related:Turing Winner Bengio Advising UK PM Sunak on Next-gen AI Safety

A verifier: To provide an auditable proof certificate that the AI satisfies the safety specification relative to the world model.

Bengio, davidad and the other Safeguarded researchers suggest this approach could potentially produce feasible safety guarantees while remaining competitive with AI systems that lack such guarantees. They view different approaches within the GS AI framework as complementary research and development efforts forming a portfolio of strategies for ensuring AI safety.

In addition to adding Bengio, the Safeguarded project has also launched its funding call for potential organizations interested in using the gatekeeper AI system to safeguard their products.

Safeguarded wants to apply the tool to domain-specific applications, namely for optimizing energy networks, clinical trials or telecommunications networks. 

Businesses have until Oct. 2 to apply.