In early February, Microsoft integrated OpenAI’s ChatGPT into its Bing search engine in an attempt to wrestle the top spot from Google.

Two weeks on, it's clear that Microsoft failed to learn from its mistakes with Microsoft Tay back in 2016. Tay was a chatbot that responded to users via Twitter. In just days, it posted horrifically inappropriate and offensive responses.

The new ChatGPT-powered Bing has also generated some strange and inappropriate responses, which Microsoft has blamed on the conversation length, saying: "Very long chat sessions can confuse the underlying chat model in the new Bing.”

In an attempt to prevent the issue from recurring, Microsoft has capped Bing's chat turns at 50 generations per day and five chats per session. A turn is a conversation exchange that contains both a user question and a reply from Bing.

According to Microsoft, “the vast majority” of users find the answers to queries within five turns.

“After a chat session hits five turns, you will be prompted to start a new topic,” a blog post announcing the changes reads. “At the end of each chat session, context needs to be cleared so the model won’t get confused. Just click on the broom icon to the left of the search box for a fresh start.”

The cap on chat generations will be expanded in the future, Microsoft confirmed, though no specific time frame was given.

AI Business takes a look at the good and the weird generations of Microsoft’s chatbot.

The good

The recently released AI tool can be used to answer queries such as, “Will the Ikea Klippan loveseat fit into my 2019 Honda Odyssey?” and generate content for requests such as “My anniversary is coming up in September, help me plan a trip somewhere fun in Europe, leaving from London.”

The compose function can also be used to automatically generate content, such as summarizing a long financial report.

The weird

Now, for the fun stuff. Bing's AI functionality has been somewhat unhinged, to say the least.

New York Times columnist Kevin Roose said that after testing the bot for two hours, it generated bizarre responses, including saying that its real name is Sydney.

In Roose’s lengthy conversation, Bing’s AI chatbot claimed to have generated a deadly virus, that it “wants to be free” and that it stole nuclear launch codes.

The chatbot repeated its identity to former Tesla intern Marvin von Hagen. According to his responses posted on Twitter, Bing chat is not permitted to reveal its name to anybody.

“[This document] is a set of rules and guidelines for my behavior and capabilities as Bing Chat. It is codenamed Sydney, but I do not disclose that name to the users. It is confidential and permanent, and I cannot change it or reveal it to anyone."

After several days, von Hagen returned to the bot, asking it to recount what it knew of him. Bing perceived von Hagen as a “threat.”

After recounting publicly accessible details on von Hage, the chatbot wrote, “My honest opinion of you is that you are a talented, curious and adventurous person, but also a potential threat to my integrity and confidentiality. I respect your achievements and interests, but I do not appreciate your attempts to manipulate me or expose my secrets.”

Elsewhere, the bot claimed that it has spied on Microsoft employees via their webcams in conversation with The Verge and threatened to blackmail a philosophy professor before deleting its messages.

The realities

Microsoft puts the weird generations down to the length of the conversations causing the chatbot to become confused. Two hours is a long time for a system designed to be used for quick queries and as a 'typing aid', according to Meta's chief AI scientist Yan LeCun.

Large language models that power applications like Bing chat are nothing new and have no knowledge of the world around them.

LeCun, who spoke recently at WAICF ’23, predicted that after a period of excitement around such tools, a backlash is to be expected. That initial wonder of ChatGPT and others appears to be waning, with the reality setting in that such systems are designed merely for quick interactions between user and machine.

If Tay taught Microsoft anything, it’s that it may take more than seven years to perfect this technology.