ElevenLabs’ AI voice tools used to create inappropriate audio

Ben Wodecki, Jr. Editor

January 31, 2023

2 Min Read
Cloud bubble of punctuation marks signifying offensive language

At a Glance

  • 4Chan users cloned the voice of Harry Potter star Emma Watson and made it read Hitler's Mein Kampf
  • Ex-Rick & Morty voice actor Justin Roiland cloned to spout vile domestic abuse content
  • ElevenLabs says it can trace users who generate vile content and proposes safeguards

At the start of the year, ElevenLabs, a startup developing AI-powered translation tools, opened its generative synthetic voice model to beta testers. But just a few weeks later, it detected voice-cloning abuses -- and has now been forced to rethink its work.

ElevenLabs was founded by former Google software engineer Piotr Dabkowski and ex-Palantir deployment strategist Mati Staniszewski. The pair went on to create a research lab focusing on deploying AI tools for long-form synthetic speech, voice cloning and automatic dubbing. ElevenLabs recently secured $2 million in funding.

Its tech has been showcased in narrating F. Scott Fitzgerald’s “The Great Gatsby.” Its AI models can generate speech based on a voice’s characteristics, with users able to set the speaker’s core identity, including gender, age, accent, pitch and speaking style.

But upon releasing it to the public during beta testing, ElevenLabs’s work was showcased for the wrong reasons. Users on 4Chan, an online community known for shocking commentary, trolls and memes, began using ElevenLabs’s model to generate inappropriate audio.

Among the examples, 4Chan users posted included a voice that bore resemblance to Harry Potter actor Emma Watson reading Mein Kampf, the autobiographical manifesto of Nazi leader Adolf Hitler.

Related:AI community condemns YouTuber’s language model prank

Another saw the titular Rick from the animated show Rick & Morty making references to spousal abuse. Justin Roiland, who voices Rick, was recently fired from the show after being charged with felony domestic abuse in Orange County, part of the Greater Los Angeles area.

And without signing up for the beta, users can go onto ElevenLabs’s homepage and try out the tool using several example voices – with no restrictions on what you type in.

Responding to the uses on 4Chan, ElevenLabs said it had detected “an increasing number of voice cloning misuse cases.”

In a statement posted on Twitter, ElevenLabs said it can “trace back” any generated audio to the user and that it plans to implement a series of safeguards.

The proposed safeguards include additional account verifications to enable voice cloning functionality, including providing credit card information or full ID verification and verifying copyright to the voice by submitting a sample with prompted text.

The startup even touted dropping its Voice Lab option altogether and manually verifying each Cloning Request.

However, in its tweets, the startup listed the options as "ideas" with no solid changes referenced. AI Business has contacted ElevenLabs for comment.

This is not the first time AI and 4Chan have come together to create vulgarities. Last August, ML engineer and YouTuber Yannic Kilcher trained a language model on three years’ worth of 4Chan posts. The result, unsurprisingly, saw it generate thousands of offensive posts.

About the Author(s)

Ben Wodecki

Jr. Editor

Ben Wodecki is the Jr. Editor of AI Business, covering a wide range of AI content. Ben joined the team in March 2021 as assistant editor and was promoted to Jr. Editor. He has written for The New Statesman, Intellectual Property Magazine, and The Telegraph India, among others. He holds an MSc in Digital Journalism from Middlesex University.

Keep up with the ever-evolving AI landscape
Unlock exclusive AI content by subscribing to our newsletter!!

You May Also Like