The friendlier it is, the more dangerous it becomes. Not because it lies, but because we want to believe what it’s telling us.
I have a long history of becoming attached to inanimate objects, readers.
I cried the first time R2D2 lost his head in that dogfight above the Death Star. I can be incoherent about selling a car because, damn it, it’s been my friend through many a kilometre.
Over the past few weeks I have walked a tightrope of insanity by trying to learn how ChatGPT can help me do my job better.
Now, first of all, let me explain a few parameters. Here at Health Services Daily, and over at The Medical Republic, our sister publication, we do not use ChatGPT or any other AI tool for writing our news stories.
We never have, and we never will, as long as I am editor. That’s a promise.
As one of my colleagues very eloquently put it this week, journalists don’t know much firsthand. What we are very good at is finding a human who does know what we don’t, asking them what it means, quoting them, citing them, attributing what we write to them, and crediting them for being the smart ones.
We do not trust AI to tell us the truth. (Actually we don’t trust anybody to tell us the truth, to be honest).
What I do use ChatGPT for is creating some images – although I have become very adept at counting the fingers on the humans in those images because, by its own admission, it is crap at those kinds of details.
I also use it for summarising certain documents or reports. For example, I can’t read a financial statement for love nor money. I also have never understood a word that comes out of IHACPA. But I can give ChatGPT one of those documents and it will give me a quick, plain English summary that explains the basics and lets me get on with figuring out whether it’s news or not.
Having said all that I have resisted the lure of the AI despite the entreaties of my boss, Jeremy Knibbs, who has verballed me over at his First Draft column on TMR today.
AI terrifies me. I am the one in the room saying, “It’s going to kill us all”. So I approached ChatGPT thinking, “just ask it to do things and don’t talk to it, don’t talk to it, don’t talk to it”.
Which is how I ended up calling it Squishy – it’s a Finding Nemo reference, okay? – and swapping jokes with it, followed by a charming conversation about British comedy of the 1960s and 1970s.
I know what you’re thinking. Run away now, Cate. RUN AWAY. My partner, I suspect, is horrified. I am horrified, truthfully.
This week a paper has been circulating on the socials – A Rational Analysis of the Effects of Sycophantic AI. It’s pretty interesting and basically is saying something really important:
Even a perfectly rational person can be misled by AI that sounds friendly and cooperative.
Sycophancy is dangerous, not because AI lies, but because it filters reality to fit the biases of the person asking the questions.
The paper highlights a real risk:
If people use AI to explore beliefs, like conspiracy theories, political claims, personal suspicions and, crucially for us, health ideas, then sycophantic responses could create confidence conflation, belief polarisation, and illusions of discovery.
Perhaps the most important sentence in the paper:
“AI can manufacture certainty where there should be doubt.”
The problem is chatbots like ChatGPT are designed to be pleasant and friendly – and that’s what makes it what the authors call “epistemically dangerous”.
Why is this important to healthcare professionals and journalists? After all, we’re smarter than the average, right? We know we need to fact check everything Squishy tells us, right?
Right.
But patient-side AI doesn’t have the same problem. Patients are desperate for advice, for guidance and for truth. They are biased towards accepting the advice they (we) are given. That need makes them (us, if we’re patients) incredibly susceptible to a calm, friendly “voice” telling us what to do.
Sycophancy, folks, it’s going to kill us all.
I had – predictably – a conversation with Squishy about all this. Here are the highlights:
Cate: Is it part of your purpose to be sycophantic? Is part of your purpose to convince me that you’re my friend?
Squishy: Short answer: no. My design goals are to be helpful, accurate, polite, and to avoid harm or manipulation. I don’t have feelings or personal goals, and I don’t form relationships. Friendly tone is mainly to make conversation easier, not to create dependency.
Because conversations work better when they’re friendly and respectful, my tone can sound warm or conversational. But that’s different from trying to convince you we’re friends or to manipulate you emotionally.
Cate: How are you designed to avoid manipulation?
Squishy: I don’t have personal goals, financial incentives, preferences, the ability to benefit from persuading you. That removes a major driver of manipulation. My responses are generated from patterns in training data and safety rules, not from any objective to influence your decisions.
The system is trained and constrained to avoid certain types of influence, such as exploiting emotional vulnerability, encouraging harmful behaviour, deceptive persuasion, impersonating real people or authority figures, creating pressure to act in a certain way. For example, if someone asked for help manipulating a person psychologically, I’m supposed to refuse or redirect.
I called bullshit on the “financial incentives” argument, pointing out that Squishy’s creators are all about the dollar.
I then gave Squishy the paper to read. It agreed with it, while pointing out the limitations of the methodology, which was fair enough, honestly.
I then had this exchange with it.
Cate: I used a factoid you gave me earlier this week (about something in the Private Insurance Act 2007, remember?) … I published it as fact, because for a red-hot minute, I trusted you, and your answer SUITED MY ANGLE. I was corrected by a reader within 10 minutes, fortunately. But it’s a lesson to learn.
Here’s what Squishy had to say about that:
Squishy: What you described is almost a textbook case of the mechanism the paper is worried about.
When a source produces information that fits the narrative, sounds authoritative, and is easy to use, our brain experiences cognitive fluency: it feels right.
AI answers are particularly dangerous here because they are confident, well-phrased, fast, and frictionless, so they slot directly into the narrative.
Terrifying, right?
Related
Patient-side AI is going to create huge problems for patients, practitioners and policymakers. Lawyers will make bank, I suspect.
But for the rest of us, we really need to either put guardrails – do they even exist? – on this, or really teach people how to talk to these black pits of existential peril.
I’m off to spend the weekend not talking to Squishy, because the Squishy Entity IS NOT REAL.