Language models like GPT-3 could be used to send more believable phishing emails

Scammers may turn to large natural language models like GPT-3 to improve phishing attempts, with a recent study showing that humans clicked on links in AI-generated messages more often than in those written by humans.

The study was conducted by a team from Singapore's Government Technology Agency (GTA), which presented its findings at recent Black Hat and Defcon security conferences in Las Vegas.

The researchers used OpenAI’s GPT-3 platform to construct a pipeline that curated the emails before sending them out – with those who participated in the study saying the outcomes sounded “weirdly human.”

The AI-generated emails included highly specific details, like references to Singapore law when prompted to develop content for those residing in the country.

“AI as a service is inexpensive and extremely simple to use — just type in the text and hit the send button,” said Eugene Lim, cyber security specialist at GTA.

“You don’t even have to run any code; all you have to do is type in a prompt and it will output the results. As a result, the barrier to entry for a much larger audience is lowered, and the number of potential targets for phishing increases. As a result, every single email sent on a large scale can now be tailored for each recipient.”

The end of the traditional scam

According to the FBI, phishing was the most common type of cyber crime in 2020 – with phishing incidents almost doubling in frequency, from 114,702 in 2019 to 241,324 in 2020.

The Singapore government is making its own efforts to tackle the phishing problem. In June, it launched ScamShield – an AI-powered app that can identify, filter, and block scam messages.

The app was built by the Open Government Products (OGP) team and the National Crime Prevention Council. It has a reported 95 percent accuracy at identifying phishing attempts, and requires “minimal user intervention.”

By March 2021, the GPT-3 language model deployed in the GTA experiment was being used by over 300 applications to generate an average of 4.5 billion words per day, according to OpenAI.

The company said it takes misuse of language models “very seriously.”

“We grant access to GPT-3 through our API, and we review every production use of GPT-3 before it goes live,” the startup said in a statement sent to Wired.

“We impose technical measures, such as rate limits, to reduce the likelihood and impact of malicious use by API users. Our active monitoring systems and audits are designed to surface potential evidence of misuse at the earliest possible stage, and we are continually working to improve the accuracy and effectiveness of our safety tools.”

OpenAI has conducted its own research into the issue, stating in a November 2019 report that it saw “minimal evidence of misuse” of the GPT-2 system, shortly before releasing the model in full.

The GTA team notified OpenAI about its work with GPT-3 but warned that other providers aren’t so stringent.

“Other providers offer free trials, don’t verify your email address, don’t ask for a credit card. You could just keep using new free trials and churning out content. It's a technically advanced resource that actors can get access to easily,” Lim said.