4Chan Users Clone Celebrities’ Voices to Make Vile Audio

At the start of the year, ElevenLabs, a startup developing AI-powered translation tools, opened its generative synthetic voice model to beta testers. But just a few weeks later, it detected voice-cloning abuses -- and has now been forced to rethink its work.

ElevenLabs was founded by former Google software engineer Piotr Dabkowski and ex-Palantir deployment strategist Mati Staniszewski. The pair went on to create a research lab focusing on deploying AI tools for long-form synthetic speech, voice cloning and automatic dubbing. ElevenLabs recently secured $2 million in funding.

Its tech has been showcased in narrating F. Scott Fitzgerald’s “The Great Gatsby.” Its AI models can generate speech based on a voice’s characteristics, with users able to set the speaker’s core identity, including gender, age, accent, pitch and speaking style.

But upon releasing it to the public during beta testing, ElevenLabs’s work was showcased for the wrong reasons. Users on 4Chan, an online community known for shocking commentary, trolls and memes, began using ElevenLabs’s model to generate inappropriate audio.

Among the examples, 4Chan users posted included a voice that bore resemblance to Harry Potter actor Emma Watson reading Mein Kampf, the autobiographical manifesto of Nazi leader Adolf Hitler.

Related:AI community condemns YouTuber’s language model prank

Another saw the titular Rick from the animated show Rick & Morty making references to spousal abuse. Justin Roiland, who voices Rick, was recently fired from the show after being charged with felony domestic abuse in Orange County, part of the Greater Los Angeles area.

And without signing up for the beta, users can go onto ElevenLabs’s homepage and try out the tool using several example voices – with no restrictions on what you type in.

Responding to the uses on 4Chan, ElevenLabs said it had detected “an increasing number of voice cloning misuse cases.”

In a statement posted on Twitter, ElevenLabs said it can “trace back” any generated audio to the user and that it plans to implement a series of safeguards.

The proposed safeguards include additional account verifications to enable voice cloning functionality, including providing credit card information or full ID verification and verifying copyright to the voice by submitting a sample with prompted text.

The startup even touted dropping its Voice Lab option altogether and manually verifying each Cloning Request.

However, in its tweets, the startup listed the options as "ideas" with no solid changes referenced. AI Business has contacted ElevenLabs for comment.

This is not the first time AI and 4Chan have come together to create vulgarities. Last August, ML engineer and YouTuber Yannic Kilcher trained a language model on three years’ worth of 4Chan posts. The result, unsurprisingly, saw it generate thousands of offensive posts.