Hoping to make AI less susceptible to confusion caused by anomalous data

The University of Pennsylvania’s School of Engineering and Applied Science and the Children’s Hospital of Pennsylvania (CHOP) have been awarded a $6 million grant to develop techniques that can defend AI systems against emerging types of cyber attacks.

The Multidisciplinary University Research Initiative grant, a Department of Defense (DoD) program, was based on the research team’s proposal to leverage “insights from human cognitive development to make artificial intelligence systems better at protecting themselves from malicious disruptions.”

The intent is to build the same robust, adaptive learning capabilities exhibited by biological intelligence into AI-based systems, so that they can work with broader categories of information and are not easy to confuse.

Operating in the real world

For example, one potential danger is that instead of directly accessing the software controlling a self-driving car’s accelerator, someone could alter a speed-limit sign, so the car’s AI does not recognize it.

The study aims to benefit researchers and designers of autonomous systems by increasing awareness of the dangers such systems could present when operating in the real world. The research team also plans to create new tools and technologies to reduce risks emerging from the adoption of AI.

While deep learning approaches require a significant amount of labeled data to be effective, their knowledge is often limited to a narrowly focused domain on which they were trained. Their ability to make correct decisions falls apart when applied in novel settings, while humans naturally apply lessons learned in one context to others with no explicit training.

“Robust, concept-learning techniques will assure that trained models operate effectively in the presence of malicious attacks, offering a substantial improvement over the vulnerability of today’s systems that can be easily compromised by even small anomalies,” explained Insup Lee, professor in Penn Engineering’s departments of Computer and Information Science and Electrical and Systems Engineering.

Lee is also the director of the Precise Center, which is researching the technical strategies for advanced safety and security of cyber-physical systems.