New York City AI-Powered Shot Detection System Accuracy Questioned

The New York City Comptroller has recommended in a report that the New York Police Department (NYPD) not renew its current $22 million contract with a company that provides an AI-powered shot detection system without fully evaluating its performance.

The audit, conducted by Comptroller Brad Lander’s office, found that ShotSpotter only confirmed shootings 13% of the time. The auditory gunfire detection and location technology from security technology company SoundThinking also reported a high degree of false positives and overestimated the reduction in police response times.

The report found that in sample months in 2022 and 2023 ShotSpotter alerts only resulted in confirmed shootings between 8% and 20% of the time.

“During the month of June 2023, for example, out of the 940 ShotSpotter alerts that NYPD responded to 771 could not be confirmed as shootings upon arrival at the scene (82%), 47 were determined to be unfounded (5%) and 122 were confirmed as shootings (13%),” it said.

It added that NYPD officers spent 426.9 hours investigating alerts that were not confirmed as shootings.

Responding to the audit’s recommendations, the NYPD wrote that “non-renewal of ShotSpotter services may endanger the public and not renewing the ShotSpotter contract until the Department conducts further analysis would be a premature measure.”

Related:Boston Dynamics’ Brendan Schulman on AI-enabled Robot Ethics

SoundThinking issued a statement saying it was “mystified” by the report.

“A central flaw in the comptroller’s report is the faulty assertion that the lack of immediate physical confirmation of a shooting by officers dispatched to a ShotSpotter alert means that the alert was triggered by a loud sound other than gunfire,” the company said.

“This is simply not how the technology and NYPD protocols are designed and is a perilous, dangerous assumption. ShotSpotter enables NYPD to become a digital witness to criminal gunfire, allowing NYPD to review a gunfire incident more fully and accurately than if officers heard it outdoors while on patrol, not only in close to real-time but also afterward.”

ShotSpotter uses acoustic sensors attached to street infrastructure such as lampposts to detect the sound of gunfire which AI algorithms then identify and locate. There are more than 2,000 sensors across the five boroughs of New York City. The system compares the signals received by several sensors to triangulate the location of the gunfire.

Any recorded alert is reported to the ShotSpotter incident review center for evaluation by SoundThinking staff members. If a gunfire incident is confirmed, the reviewer alerts the NYPD with details of the type, number and location of shots heard. The appropriate NYPD precinct may then view the incident location using CCTV cameras and dispatch officers to investigate.

Related:Microsoft Blocks Police Use of OpenAI for Facial Recognition Cameras

The NYPD has been using ShotSpotter since 2015.

An Associated Press investigation in 2022 identified serious flaws in using ShotSpotter as evidentiary support for prosecutors and that the algorithms it uses are proprietary so cannot be properly investigated by the public, jurors or legal bodies.