Contact center team appear unable to control the rogue AI system in a viral video

Louis Stone, Reporter

May 19, 2021

5 Min Read

Contact center team appear unable to control the rogue AI system in a viral video

A Waymo self-driving car got confused on a public road, blocking traffic and evading a support team sent to contain the disoriented hunk of metal.

A remote operator also appeared uncertain as to what was happening, and could not control the vehicle as it sporadically lurched forward.

The entire ordeal was captured on camera by passenger Joel Johnson, who records entire unedited Waymo trips for his YouTube channel. The incident happened in Chandler, Arizona, where Google’s subsidiary is trialing its autonomous service.

Why are you running?

The video was of Johnson’s 54th trip, and third incident – and the most serious failure of the system he has experienced to date.

Ten minutes into an otherwise uneventful drive, the autonomous car took an unusual shortcut from a main road, turning off to drive a longer distance through a residential area. It then tried to rejoin the main road.

But, it was unable to turn right as intended because part of the road was blocked off by a few incredibly common traffic cones. With another car honking behind, the vehicle stopped moving. The multibillion-dollar artificial intelligence system and the culmination of decades of R&D had met its match.

"Sorry," Johnson said as he gestured to the person behind. "Go around." The other driver opted for reversing and driving back the way they came.

After a few seconds, the Waymo car's display said that a roadside assistance team was just six minutes away. The team trails Waymo cars, watching for issues, and jumping in when needed.

The vehicle's internal cameras also kicked in, and there was a call from a rider support assistant. Assistants can watch video feeds from the car, but cannot remotely control it. Terrifyingly, it also appears that they cannot shut the car down, even if it starts acting irrationally.

"Please remain seated with your seatbelt fastened in case the car starts moving again," the assistant said.

After several minutes of awkward conversation, with the roadside assistance team perpetually four minutes away, the car suddenly began to move. The vehicle pulled out onto the main road, and then parked again – half of it still on the road. Honking followed.

"Let me see if I can get my team to hold that vehicle," the remote assistant said – again, suggesting that they had no control over the car, even if it suddenly decided to drive into traffic.

The short journey from cone to cone also appeared to have canceled the call for assistance. "I don't even have roadside assistance assigned as of right now, because the car was no longer stranded, but it does look like it is stranded again,” the assistant said. “We're going to try to see if we can get roadside assistance assigned.

"Hang tight there."

After several more minutes, the car started reversing, blocking more of the road.

As this was happening, construction workers came along to remove the cones, accompanied by more honking. Eventually, there was movement again. A sudden jerk forward, eliciting a groan from Johnson.

"Are we moving?" the assistant asked hopefully, as the vehicle picked up speed.

Just a minute later, disaster struck yet again. The car had found more cones, and was blocking the road once more. But not to worry, roadside assistance was on its way. Support vehicle pulled up and its driver started walking towards the errant car. But the Waymo was having none of it.

As he approached, the car darted forward – with neither the remote or roadside assistants able to stop it.

It did not make it far, once again being foiled in its escape by pesky cones.

Finally, success – the roadside assistant managed to get into the car. “We've been tracking you for a while," he said. "It tried to unstick itself a couple of times there. We figured it would probably lose that fight."

The human managed to take Johnson home without further incident. Sadly, he revealed that his shift had ended 20 minutes prior.

Blame it on humans

Humans have had to fill many of the gaps within the Waymo service as it expands in Arizona. Last year, despite a job that has involved cleaning cars of used needles and dealing with hostile locals, Waymo slashed benefits and pay for the employees, who are all contractors.

In a statement, Waymo said that "the Waymo [AI] Driver detected an unusual situation and requested the attention of a Fleet Response specialist to provide additional information. During that interaction, the Fleet Response team provided incorrect guidance, which made it challenging for the Waymo Driver to resume its intended route, and required Waymo's Roadside Assistance team to complete the trip."

The claim does not appear to be supported by unedited video, in which the roadside assistance team is called immediately (before being accidentally canceled and recalled).

It is also not clear what the Fleet Response team guidance was – as neither roadside or remote teams had control over the vehicle, or appeared to be able to predict when it would begin moving.

"While the situation was not ideal, the Waymo Driver operated the vehicle safely until Roadside Assistance arrived," Waymo added. Again, this could be disputed, as the vehicle repeatedly blocked traffic, made sudden moves, and parked illegally in an area blocked off by highway maintenance.

The failure comes after Waymo's CEO stepped down earlier this year, soon followed by the head of investor relations, head of manufacturing and global supply, and general manager of Waymo’s Laser Bear LiDAR business. A few months ago, it emerged that Waymo's chief safety officer had also left the business.

About the Author(s)

Louis Stone

Reporter

Louis Stone is a freelance reporter covering artificial intelligence, surveillance tech, and international trade issues.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like