Do you remember Microsoft’s Tay, Meta’s BlenderBot 3, Google’s Bard, or Elon Musk’s Grok? It’s probably best if you don’t. These generative AI chatbots spectacularly face-planted by spewing racist and sexist invective, getting facts dead wrong, praising Hitler, and launching ad-hominin attacks against celebrities, entire nations, and even their parent companies. Then came ChatGPT. It was as welcome as a brisk spring morning. For the first time a chatbot conversed in a facsimile of one-to-one conversation. It drew you into a compelling exchange of ideas. It left you with powerful visions of a future that included disembodied wisdom, like the computer on Star Trek, and even intelligent, humanoid robots, like the likeable Data.
Unfortunately we didn’t yet know about the psychological frailties in us that chatbots could manipulate, such as influencing a man in Belgium who killed himself at his chatbot’s urging; prompting a trespasser at Buckingham Palace to stalk the Queen of England with a crossbow; and, tripping up a lawyer whose motion was slapped down because it was full of fictional case law made up by ChatGPT. We didn’t know about the hallucinations and emergent properties that plague all generative AIs, making them fundamentally unreliable.
In fact, ChatGPT turned out to be a trojan horse, different from all its chatty goose-stepping brethren, who wore their racist, homicidal notions on their sleeves. Instead, its problems came out from hiding, surprising us with more jump scares than Scream 3. And recently to its bag of dirty deeds generative as a whole has added blackmail, plagiarism, theft, sextortion, evading safeguards, withholding emergency help, and leaking privileged information.
How did problems as big as these get past the formidable technical talent behind the best generative AI? To find out, let’s have a look at how generative AI works.
Generative AI gets its name from creating or generating outputs like text, images, music, and code. But before it can do that, it has to study huge amounts of this kind of data. For example, when learning about text, it trains on books, newspapers, the internet, Wikipedia, online discussions, emails, blogs, and curated databases. This training helps the AI understand different writing styles, patterns, and structures, so it can learn the rules behind them. Neural networks, which are the core technology behind generative AI, help this process by breaking down the data to find important features and connections. As a result the AI creates outputs that make sense and fit the context.
Before it can create outputs, however, the AI needs a prompt. That’s where you come in. A ‘prompt’ is a request or instruction. For example, you might ask the AI to write a report about a bird whose intelligence you’ve always admired: the American Crow. The AI uses what it’s learned to predict the most likely sequence of words that will satisfy the prompt. So it doesn’t sound as if its just regurgitating data, it includes some randomness.
Now here’s where the crazy starts. Generative AI might tell the truth, but might just as easily whip up fake news, bogus bibliography citations, and deepfake videos and photos that look and even sound real. Why? Generative AI are ‘black box’ systems, meaning we know the inputs – data – and the outputs – a correct-sounding answer. But not even the engineers who design generative AI know how they come up with the answers.
Generative AI ‘hallucinates,’ meaning it sometimes produces outputs that sound believable but are wrong or nonsensical, which disqualifies them from mission critical (very important!) jobs. In addition, as these models get larger, they pick up surprising skills, such as translating languages or writing computer code, even though no one programmed them to do that. These surprises are called ‘emergent’ properties, and they point to even greater unpredictability as generative models become larger and more advanced.
But don’t take my word for it. UC Berkeley professor of computer science Stuart Russel is the co-author of AI’s authoritative text, entitled AI: A Modern Approach. About generative AI Russel said, “We have absolutely no idea how it works and we are releasing it to hundreds of millions of people. We’re giving it credit cards, bank accounts, social media accounts. We’re doing everything we can to make sure that it can take over the world.”
Formerly experts dismissed the ‘existential’ danger of artificial intelligence, saying its potential for world takeover was overblown. Today, as I wrote in my recent book, The Intelligence Explosion: When AI Beats Humans at Everything, experts seriously investigate the ways in which AI could overpower humanity, and most believe it could. For example, a misaligned AI could misconstrue a badly worded goal, such as ‘eliminate cancer in humans.’ The customary way is to create medicines or procedures. The AI way may be to eliminate humans altogether. It’s important to note generative AI has no common-sense at all.
AI-controlled weapons could escalate conflicts faster than humans can intervene, increasing the likelihood and lethality of war.
Power grids and financial systems – essential infrastructure – would be elementary for a superintelligent machine to tap for power and resources. Meanwhile, as AI develops, lethal properties could emerge with the suddenness of a lightening strike, with devastating results. Finally, bad actors like authoritarian regimes or extremists groups would gladly use AI for mass surveillance, propaganda, cyberattacks, or worse. In Gaza, surveillance systems have morphed into enhanced weapons systems, allowing the stronger side to kill more people faster than in any war this century.
Ultimately, generative AI is less a digital oracle than a haunted house, full of hidden dangers behind seemingly helpful doors. What began as a bold leap toward smarter machines has revealed unsettling risks: unreliable answers, unpredictable behaviors, and the potential for real-world harm. As these black box systems become woven into our lives, we must move beyond awe and convenience. The real test isn’t building smarter AI, but learning how to use it wisely, before we’re caught off guard by consequences we cannot control.
Interested in more? The Intelligence Explosion is available now!
James Barrat is an author and documentary filmmaker who’s written and produced films for National Geographic, Discovery, PBS, and many other broadcasters in the United States and Europe. He is the author of Our Final Invention: Artificial Intelligence and the End of the Human Era, Facing Suicide: Why People Kill Themselves and How We Can Stop Them, and The Intelligence Explosion: When AI Beats Humans at Everything.
