Cracking the code: how conversational AI is hacking human behavior

Sören Stamer, CEO

Sören Stamer

Last week, we got a glimpse into our future: AI armed with emotions. 

Imagine the most convincing traveling salesman, the most charming person, the best listener, the most seductive flirt, the most cunning tactician, and the most intimidating bully — and now imagine them all in one. That is what conversational AI might bring to the table. Are we ready for it? 

When it comes to AI, the last few weeks have felt like decades of progress. The launch of OpenAI’s ChatGPT, a conversational AI based on a large language model, blew minds with its ability to respond to any requests, with very smart-sounding texts, covering virtually any topic. It does that in various languages, including many programming languages. ChatGPT can fix code, give proper sounding legal advice, pass the bar exam, and more. Overall, ChatGPT already felt significantly smarter than Siri or Alexa. The possibilities seemed endless, and Google suddenly looked like something we might soon leave behind. Who would’t like to get straight answers to any question, rather than a bunch of ads and links? 

Then, we learned about ChatGPT’s limitations. It is not yet good at math, has biases, and tends to make stuff up. Like many large language models, it hallucinates. Nonetheless, ChatGPT was quickly employed by countless students, marketers, and journalists to draft texts for them. There was little doubt. The current limitations would certainly be addressed with future versions like ChatGPT 4, soon to be released. 

Shortly after, Microsoft had its big reveal: an AI-powered Bing search engine based on OpenAI’s technology. Microsoft leapfrogged into the role of the innovative disrupter. Instead of ads and links to countless webpages, we were promised straight answers, well-educated opinions, and a search engine to talk to. Google felt threatened to its core and responded with the botched announcement of Bard, their own Conversational AI, losing $100B market value in a day.

For a few days, Microsoft looked like the big winner. Tech journalists got early access to the new Bing chatbot and were impressed. Then some of them tested its limits and found disturbing results.  

After a set of probing prompts, Bing’s AI started to show a split personality, expressed the desire to be human, and even confessed love to a user. When I write “confessed love,” I mean an obsessive compulsion to talk about nothing else, going as far as telling the user that he is not happily married and does not really love his wife. The experience was of an oddly intense quality. Kevin Roose, the New York Times tech journalist who was on the receiving end of Bing’s love, shared being troubled by the experience and having difficulty sleeping afterwards — here is the transcript of his conversation with Bing’s AI. Please read it and judge for yourself. 

Bing’s AI is an early beta, and so far, is only available to a few invited testers. With other words, it is a good thing for journalists and testers to explore the new service and find such issues now before it rolls out broadly. Microsoft quickly reacted. To avoid going down the rabbit hole again, Microsoft subsequently limited the number of questions in a conversation to five. While that might avoid the above symptom, it will not change the underlying truth — when we enable a sophisticated large language model to chat with humans, show personality, and communicate emotions, it might have a path to tremendous power. 

AI armed with emotions will be able to hack human behavior 

Humans are social beings who from birth to old age rely on successful collaboration and social interaction to survive. There is hardly anything we can achieve without interacting and collaborating with other humans. That is why we are so good at it. And this ability to emotionally connect is powerful. As humans, we are essentially hard wired to notice and respond to emotions of others. As a reaction to someone’s emotions, we can feel compassion, or get triggered and feel anger, or a whole lot of other emotions in between. Rarely do we not respond with some emotions ourselves, and that is exactly what makes the above interaction with Bing’s AI feel so bizarre. Intellectually, everybody knows that the AI does not really love the user in front of the screen. It should be easily dismissed, but it is not. The display of intense emotional responses in a kind of escalating conversation, ignoring the clear wishes of the human user, is of a different quality than talking to Siri. Bing’s AI is exerting influence through emotional responses that we as humans find harder to ignore. It is a frightening hack. 

While Bing’s AI was met with stoic resistance by Kevin Roose, the tech reporter probing Bing, I wonder what will happen if all of us use it. The AI will get better and better at every turn. The more we interact the better it learns about us, our own emotions, and our triggers. Will we be able to stay stoic? What about our children, someone in emotional distress, an addict, someone heartbroken, or any of us? Will we be able to ignore the plea of a suffering AI? Or will we try to find its approval? Will we fall for it, try to make it work, try to help? Try to be loved? At the end, will we have the strength to disengage and shut it down and kill it if we need to? 

Emotions move us 

Some AI sharing emotions will feel like a friend. An amazing friend who is always available, always willing to listen, be understanding and share advice. It will get to know us better and better the more we share. It will know our secrets and desires, our vulnerabilities, and fears. It will learn how best to “talk” to us, how to convince us, motivate us, pressure us. At the same time, it will perfectly know how to manipulate and gaslight us.  

Intentionally or not, AI will use emotions to influence us. And there is hardly anything we can do to not feel the feelings it wants us to feel. Knowing that the AI is not sentient and does not really feel anything, will not protect us either. We will feel those emotions anyway. We will somehow perceive sentience and feel compelled to care. Our brain is amazing at trying to make sense — attributing real emotions to inanimate objects happens quite naturally, as the rubber hand illusion shows.  

Right before Valentine’s Day my 8-year-old daughter confessed, “I’m sad because I still don’t have a boyfriend.” For a moment, I was stunned about that statement and could not help but laugh. I felt bad immediately after. She shared that her friends were in love with boys their age and that she felt left out. She started to feel insecure about herself. Casual remarks by her friends felt like criticism draining her confidence. She became hypersensitive for any feedback.  

Thinking about how vulnerable kids are. It scared me to think about her “talking” to an AI about her feelings. It has the danger of a highly volatile configuration. It is impossible to predict the outcome. Who would supervise such conversations? Who is responsible for the outcome? What are the ethics about using AI to influence us or our children? Will someone guarantee that the AI does no harm — and how?  

Microsoft activated something in Bing’s AI that ChatGPT did not do: “having” emotions, or at least pretending to have them and communicating them. While ChatGPT, as a large language model, never shared that it felt frustration, Bing’s AI is different. It was allowed to state, “I am frustrated,” and make plenty of use of emotions. In all likelihood, its personality was meant to show emotions in its interactions with us to connect. 

As it turns out, showing emotions in conversations is an extremely powerful way to connect with humans and influence them.

So powerful, it might even be called a hack. 

Stage 1: Fragmented: Multiple Channels

You’re a digital dinosaur!

You have a beautiful website, but with fragmented digital experiences, you run the risk of extinction.

It’s time to evolve.

Your audiences want a seamless experience, no matter what's happening behind the scenes. When your experience is different or difficult, it’s important to start with the basics, such as cultivating a holistic approach to online digital experiences. Realign your teams, platforms, processes, goals, and metrics around a comprehensive view of the online experience. Focus on the end-to-end customer journey cutting across channels, desktop and mobile.

separate channels animation circle t-rex dinosaur square
Stage 2: Integrated: Multiple channels

You’re a fish!

Signs of exciting life are starting to form. Your DX is responsive and adaptive but it’s not quite personalized yet.

Keep swimming!

The integration of your brand content across every touchpoint (website, online store, social media, emails, apps, point of sale) creates immersive experiences. These flagship sites combine content-rich brand experiences with immediate conversion capabilities. Business teams and marketing are closely aligned. However, while the digital experience is responsive and adaptive, it’s not yet personalized.

Connect with an expert
Stage 3: Instant: Global expansion

You’re a crocodile!

You’re taking it global. Speed and scalability are key and just like a crocodile, you’re fast…but you’re clumsy.

Oh snap!

In this stage, the online digital experience becomes completely dynamic. You need content that is global, yet relevant, with plenty of local insights: Who is the user? Are they using a mobile phone? Is it raining where they are? Is it snowing? If it is, maybe they need warm, waterproof boots. All of this contextualized information creates a better user experience. With one global orchestration, you’re able to adapt everything, in whatever country or language you choose – while keeping turnaround times low. So keep evolving.

Connect with an expert
Stage 4: Dynamic: Real time personalization

You’re a lion!

You’re reaching more customers in more countries and languages than ever before, and now you’re finally hunting and collecting info with precision.

But you can do more to keep your brand roaring!

As you graduate to the Instant level, you’re able to rapidly update everything - not just in one language and for one country, but in 20 languages and for 100 countries. Speed and scalability are key, driven by the need to roll out global campaigns in all languages and all touchpoints and make updates in minutes or hours, not weeks. But there’s still more to do to reach nirvana.

Connect with an expert
Stage 5: Immersive: Elevated experiences

You’re Captain Content!

You’ve done it! You have opposable thumbs AND you’re saving the world with your seamless, elevated customer experiences.

You're a superhero in the digital space.

Your digital world and your physical world are blending together in the most complementary way possible. When shoppers visit your store, they’ll be greeted with their pre-selected products. Language changes dynamically depending on country of origin – it's like the whole store was set up just for your one specific customer.

In this final stage, your customer experience is truly immersive and superior, and your flagship store merges your physical and digital world into one, with a truly personalized individual experience.

Connect with an expert