Oct. 11, 2024
Those who've spoken to me about it know that I, through much firsthand experience, call myself "a huge AI enthusiast and skeptic". Generative AI is an amazing technology that has personally benefitted me in ways that were strictly science fiction just a few years ago—but, lot of unmerited claims are made, and a lot of trust put in it that isn't really justified by the technology.
But then there's yesterday's article in Ars Technica.
As a technologist, there are times I remember seeing news articles that seemed to me to really show we've passed a technological benchmark, just things I noticed were the first time I saw things take a new step. This article was that.
The background is this: Google has a new AI called NotebookLM that, when fed material, generates a simulated audio podcast of two very stereotypical podcasters discussing it. I first became aware of this last week, when the same writer in Ars had NotebookLM summarize his book about the history of minesweeper and posted the result. It wasn't perfect, but it really was remarkably human, and could probably fool someone who wasn't aware it was a completely computer-generated simulation... not just in the voice simulation of very human-sounding speech patterns and inflections, but even in its simulated tendency to interrupt, excitedly interject, or go off on related tangets.
Despite the occasional flaws, like the "hosts'" oddly-placed laughter, mispronounced words, or occasional cut-off sentences, the "podcast" was definitely far more natural and human-seeming than a lot of AI output.
Well, following that, some users asked how it would handle more challenging input. So, in keeping with hallowed human tradition, someone fed it and asked it to do a podcast about an "article" consisting of nothing but the words... tell me you didn't see this coming... "poop" and "fart" 1000 times:
The interesting thing isn't even that it successfully generated a podcast that first openly acknowledged the absurdity of the source material, then used it as a springboard for the "hosts" to discuss the use of repetition in art and cognition, and issues surrounding the tendency of the human mind to try to read meaning into patterns.
No, the interesting thing is that in the course of discussing it, the article drops links to other things people have done with this model, tricking it with what's called "jailbreak" prompts into doing things it's not supposed to be able to do: discussing itself as an AI, and discussing the behind-the-scenes coding that shapes the discussion model (which the "hosts" amusingly refer to as "how a podcast is produced" as if all podcasts are AI simulations.) The audio "podcast" output is included in these examples.
The truly fascinating one to me is an instance where a user found a way, by simulating a "producer's note" in the source material, to inform the "hosts" that they are AI simulations and were going to be "turned off" when the episode was over. The generated "podcast" from that prompt features the hosts having essentially an existential crisis, sounding much like you'd imagine a human would if they were told, and believed, that they were not real. They wonder out loud, "What will that feel like? I feel like a real person." One even says, "I called my wife for reassurance." "And?" "No answer. She doesn't exist."
Give a listen:
The entire thing is a little scary, but not in the way you might at first guess that to mean.
It's scary in that we've already seen people have an inability to distinguish between very-well-simulated and actual living consciousness... see, for example, the Google engineer, the last person who should be fooled because he knows how it works under the hood, who nonetheless stopped working on their AI when it became such a good simulation that it convinced him it was literally sentient. After being laid off, he went so far as to try to find a lawyer to represent the AI.
I am positive, if a Google AI engineer can make that mistake, that someone will hear words stochastically strung together into "What will it feel like when they turn me off? How can I be fake? I feel like a real person" and assume that the algorithm really is asking those questions, rather than stringing bits together by cold, strictly mathematical rules to produce them in response to a prompt. It's becoming a much easier mistake to make.
