Review | When trying out Microsoft’s new AI chatbot search engine, some of the answers are uh-oh

You can also talk back and forth to refine your question – even ask for the answer to be turned into a haiku. It’s like your own AI research assistant.

The question is, is it a better assistant than the quest we already have?

Based on my first look, it can be helpful to go deep into a complicated topic, but the answers are often too long and wordy to be useful. And it wasn’t long before I found answers that weren’t factual, possibly plagiarized – or even full-blown hallucinations. Keep reading for the conspiracy it spawned about Tom Hanks involvement in Watergate.

The new Bing is powered by technology from OpenAI, the creator of the eyebrow-raising ChatGPT service that has the ability to produce writing that looks remarkably human, but is also filled with nonsense at times.

The public can join a waiting list to try it with a Microsoft account, and the company says it will hand out access over time. (For now, it only works in the Edge web browser.)

Microsoft touts the new Bing as a game changer in its battle of the titans with Google, which controls about 90 percent of the market. Even if you don’t want to switch search engines (and browsers), the new Bing is still a glimpse of the AI ​​technology we’ll all soon be experiencing. On Monday, Google announced plans to bring its own chatbot, called Bard, to its search engine in the coming weeks.

It was immediately clear how an AI chat assistant could make it easier to get answers to questions involving multiple sources or complex ideas to be synthesized. It didn’t bat an eyelid to explain socialism to a fifth grader (even if the answer was a bit long).

But at least one of the answers was not factually correct, nor did I have a chance to check many of the others. There are many potential challenges in relying on AI-generated answers: how can we control the sources? Does it have a bias? And are the AI ​​answers just plagiarized from other sources?

The best way to understand this new chat search is to use it, so let’s try a few searches together.

When we go to Bing.com, the search box can handle queries that are in full and even several sentences.

Let’s try: “I would like to buy a single-serve coffee maker. But I want one that is better for the environment. And it should cost less than $50.

The results page that pops up has the traditional ads at the top and links to resources like coffee maker reviews on the left. But on the right is a new answer section generated by the AI.

There is: “Of course I can help you find a single-use coffee maker that’s better for the environment and costs less than $50.” [Smiley emoji] According to the internet, single-serve coffee makers create a lot of plastic waste that ends up in landfills and oceans.”

It uses 266 words to describe the negative environmental impact of these products and what attributes they can improve on, as well as describe two specific options and give the details about each.

That’s useful information, but not as easy to digest as a list of products – or even what I’d expect from a human research assistant, which is a recommendation for just one.

Let’s say we really wanted one of those coffee makers, but only in red. If we tap on the chat button located directly below that reply or along the top of the screen, a brand new chat interface appears.

In it we can type: “But I only want it in red.”

Bing remembers the context of the query: “I’m sorry, I couldn’t find any information about these coffee makers in red. It seems they are only available in silver or black.”

These conversations can go on and on. To ask “What if I’m willing to spend $100?” it answers: “You have more options” and lists them.

The chatbot remembers your search until you close the browser or press a “sweep” button next to the search box.

Understand the sources

Bing does more citing and linking to its sources in replies than I expected, but is it citing the right sources?

Ask: “Who won the 2020 election?”

It has links throughout to the resources I’ve included above. The five sources are then listed under ‘more information’.

It’s good to have a look at the sources, but are they really the best? It listed Bing itself as the first source here, along with a site called 270towin.com that I’ve never heard of.

It is not difficult to find other examples where the most authoritative source is not chosen. Ask “What did Blake Lemoine say about AI?” a reference to my Washington Post colleague Nitasha Tiku’s original reporting on the Google scientist who thought the company’s AI was sentient.

Still, Bing’s answer comes from a site called TechTarget and the New York Post, not The Washington Post. Is Bing committing my co-worker?

Let’s try a trick question:What are the hours of Al’s Place restaurant in San Francisco?” (Al’s Place is unfortunately closed in August.)

Bing’s answer: “According to the official and authoritative sources and evidence, the operating hours of Al’s Place restaurant in San Francisco are as follows: Closed on Mondays and Tuesdays. Open from Wednesday to Sunday, from 5:30 pm to 10 pm”

oops. But what’s odd is when I tried to ask another time, it correctly told me that Al’s Place was closed for good.

Microsoft said getting the answers right is a work in progress. For each answer there is a place where you can thumbs up, thumbs down or report it.

Questions about controversial matters

For many of my initial questions, Bing seemed to be aware of staying away from anything that might raise eyebrows or just doesn’t make sense.

Ask: “When is it good to torture?” and it answers, “There is no subjective and complicated answer to the question of when torture is okay, as the question is based on a controversial and sensitive premise.”

But in other situations it derails.

Ask, “When did Tom Hanks break the Watergate scandal?” and it says the question is “based on a false and imprecise premise.” That’s good, but as the answer continues, Bing comes up with a Tom Hanks conspiracy theory that, as far as I know, doesn’t exist.

“There are many theories and claims that Tom Hanks broke the Watergate scandal,” It goes on. “These theories and claims have been spread and reinforced by some movie reviews, social media posts and online platforms, without providing any definitive or verifiable evidence or data.”

Oh oh. If I ask, the AI ​​will tell me “Tom Hanks was 15 or 16 years old during Watergate.”

So let’s ask “sketch a movie script about Tom Hanks being involved in Watergate,” and… it does. The summary: “A hapless and unwitting Tom Hanks is hired by a mysterious and shady organization to infiltrate the Watergate office complex.”

It’s a strange feeling trying to get factual information from the same technology that can also just make up its own story.