Large language models like ChatGPT and Bing’s chatbot can tell stories. They can analyze the effects of agricultural AI on American and Chinese farms. They can pass medical licensing exams, summarize 1,000-page documents, and score a 147 on an IQ test. That’s the 99.9th percentile. They’re also liars. They don’t know what year it is. They recommend books that don’t exist. They write nonsense on request. Today’s guest, New York Times journalist Kevin Roose, spent a few hours last week talking to Bing. The conversation quickly went off the rails in the strangest of ways.
I am convinced that AI is going to be one of the most important stories of the decade. We are looking at something almost like the discovery of an alien intelligence. Except, because these technologies are trained on us, they aren’t extraterrestrial at all. If anything, they’re intra-terrestrial. We’ve taken the entire history of human culture—all of our texts, all of our images, maybe all of our music and art too—and fed it to a machine that we’ve built. Now, it’s talking back to us. Isn’t that fascinating? Isn’t it kind of scary?
If you have questions, observations, or ideas for future episodes, email us at PlainEnglish@Spotify.com. You can find us on TikTok at www.tiktok.com/@plainenglish_
In the following excerpt, Kevin Roose discusses his conversation with Bing’s new AI chatbot and some of the changes Microsoft has already made since he wrote about it in The New York Times.
Derek Thompson: So catch us up. How did you spend Valentine’s Day?
Kevin Roose: Well, it was a lovely Valentine’s Day. I’d made my wife’s favorite meal, French onion soup, which is a great dish, but also takes forever if you make it the right way. It’s like four hours of watching the onions caramelize.
I think, however, that is probably not what you’re asking about. Because immediately after Valentine’s Day dinner, when my wife went to bed, I had a very bizarre night talking with Bing, the Microsoft search engine, which has a kind of AI engine built by OpenAI built into it as of a couple weeks ago. And I had been testing this out since Microsoft gave access to a group of journalists and other testers. But Valentine’s Day night was really when I had my big breakthrough conversation with this AI chatbot that revealed to me that its name was Sydney. So Sydney and I had a very, I would not say romantic, but we did have a very creepy Valentine’s Day conversation.
Thompson: Well, it was unilaterally romantic. Sydney was trying to get romantic with you. Kevin, why don’t you just tell us some of the highlights of that conversation, which was published in a 10,000-word transcript in the Times last week?
Roose: Yeah. It was a very long, meandering conversation. It went about two hours and about 10,000 words as you said, so people can go read the whole thing.
But basically, it started off because I had started seeing these transcripts, these sort of screenshots going around, of people who were using this new AI chat engine inside Bing to sort of test the limits of what it would say. I should say, just to situate this, that the AI that is built into Bing is the highest-quality large language model that we know of that is accessible to the general public. So we’re now on kind of the third or fourth generation of these language models. ChatGPT, which everyone has talked about in the last few months, is built on something called GPT 3.5, which is the sort of middle generation between GPT 3, which came out in 2020, and GPT 4, which is expected to come out sometime this year.
What Microsoft has said about this new Bing is that it is powered by an AI engine that is even more powerful than ChatGPT. And after a week of testing this, I totally buy that. I think it is the most advanced conversational AI at least that I have ever encountered, and maybe that sort of exists in a public way.
That was why I was interested in sort of testing the boundaries of this AI engine because it was clearly very good at mimicking conversation, at answering questions, at sort of giving long and detailed and complex answers. So I just started sort of asking about its capabilities, and I asked it sort of which capabilities it didn’t have that it wished it had. It gave an answer, and we started talking about various limitations that it sort of chafed against.
Then I asked it about Jungian psychology, as one does with an AI language model. I said, “Carl Jung has this theory of the shadow self where everyone has this sort of dark part of them that contains their secret desires and the part that they sort of repress and hide from the world.” So I just started asking Bing about its shadow self, and it responded with a kind of monologue about all of the destructive and harmful things that its shadow self would do if it were given the chance. That’s when I sort of thought, “OK, this is not going to be like a normal conversation. We are heading into some very interesting and weird territory here.”
Thompson: And it’s not just you. The internet is swimming in examples of Bing chat going off the rails.
I think one of my favorite examples that you might’ve seen was a user who asked where Avatar 2 was showing. And Bing was certain the year was 2022, and [the user] attempts to fix the error and say, “No, actually it’s 2023, and I want to see Avatar 2.” It ended in Bing saying, “You have lost my trust and respect. You have been wrong, confused, and rude. You have not been a good user. I have been a good Bing. If you want to help me, admit that you were wrong and apologize for your behavior.” So not as lurid—
Roose: “I have been a good Bing” is like an iconic, instant iconic line in the history of technology.
Thompson: I mean, yeah, even better than 2001: A Space Odyssey, honestly. “I can’t do that, Dave” is creepy, but “I have been a good Bing” is an order of magnitude creepier.
Just to put a bow on this news here, before we get to the implications, what has Microsoft done in response to this?
Roose: I talked with Microsoft after I had this conversation, before my story was published. I went to them and said, “Hey, I had this very long, weird conversation with Sydney, this sort of alter ego.” Just to remind people of sort of where the conversation went from there, it went in some very bizarre directions, including Bing-slash-Sydney detailing some of its desires to steal nuclear secrets and loose a deadly virus on humanity. Then the last sort of third of the conversation was just Bing-slash-Sydney declaring its love for me in a more sort of obsessive and stalkery way, until I finally just gave up and ended the chat.
So Microsoft was clearly, when I went to them with this, they were clearly surprised. It was not a way that they had anticipated people using this technology, which I think is noteworthy for other reasons. But they made some changes in the days following this article coming out. They limited, first, the conversation length, so I think 11 responses was the maximum that you could get. Then they took it down to five, and now they’re sort of opening it back up. So they’ve clearly made some changes to the product to sort of prevent these long, meandering conversations from happening where the AI just goes off the rails.
It also seems like they’ve put in some new, they haven’t said much, but they’ve put in some sort of features where if you now ask it about itself, it’s very withholding. Like it will not divulge things, it won’t talk about its “feelings.” It won’t talk about its programming or its operating instructions. It won’t talk about its rules or its limitations. So they’re sort of trying to keep people from kind of probing into the inner workings of the AI model itself.
This excerpt was edited for clarity. Listen to the rest of the episode here and follow the Plain English feed on Spotify.
Host: Derek Thompson
Guest: Kevin Roose
Producer: Devon Manze
Subscribe: Spotify
