No one is positioned to respond to how fast AI is iterating and creating new, potentially dangerous dynamics.
In conversation this week with Grahame Grieve, the software architect who created FHIR, the interoperability standard underpinning a lot of the world’s healthcare data sharing, he identified a subtle and so far largely undiscussed way that AI could damage patient trust in doctors and our whole current medical model.
The problem is not because AI gets things wrong, but because it’s too literal, and arrives faster at an information destination than anyone can adjust to.
The mechanism sits inside normal empathetic and competent clinical judgment of a doctor.
A doctor examining a patient with a set of general symptoms will often be working through a mental differential diagnosis – some of it common, some of it potentially rare and frightening or maybe almost certainly not relevant.
Renal cancer and a simple infection share several of the same early and major symptoms.
A doctor will usually use a test result and clinical experience to quietly rule out the rare, alarming possibilities before ever mentioning them to the patient.
That’s not evasion. It’s being a good doctor. There’s no reason to frighten someone with a diagnosis you can quickly exclude or already have excluded.
The problem starts when that same patient goes home and asks ChatGPT or Claude about their symptoms and what the doctor said. It might even be that the patient now takes back from the consult the actual transcript of their discussion with their doctor.
The AI has no access to the doctor’s private reasoning, the blood test that can quietly rule out the scary outcome, or the years of clinical pattern-matching that told the doctor not to worry.
The AI usually has no information on the doctor’s track record and trust relationship already established with that patient.
So AI raises all the possibilities regardless. It’s not because it’s smarter than the doctor, it’s because it doesn’t know what’s already been excluded.
The patient is left looking at a plausible, AI-generated alternative their own doctor never mentioned, and a real possibility of drawing an obvious, but wrong, conclusion: that something is either being withheld from them or that their doctor isn’t that competent.
Also in today’s edition:
- EXCLUSIVE: SDPRIA review results due Friday as leak reveals 3500 unresolved rollout incidents
- Australia risks becoming ‘hostage to geopolitics’ over healthcare AI
- Physical assault rates among healthcare workers double in 10 years
- Northern Territory to come full circle on voluntary assisted dying
- Coatsworth slams public health and clinicians for ‘failed’ smoking policies
- With AI in the consultation room, governance matters more than ever
- Researchers map ‘pharmacy deserts’, call for more equitable distribution
Grieve’s point isn’t that doctors need to change what they say to patients. It’s that this specific, and entirely plausible dynamic is arriving now, largely unexamined, at exactly the moment AI capability is moving on what Grieve feels is a cycle of significant week to week improvement.
Grieve thinks that the medium to long term will likely be OK because any “scary” dynamics like this will be eventually addressed by improvements in AI.
But for now we need to recognise that AI can’t solve the problem of not knowing specific information about the patient that the doctor has that it hasn’t.
And the solution to that is to “get all the data you can into the hands of the patient as soon as possible”.
A key problem today is that the institutions we are relying on for patient protection – our state and federal health departments and agencies, and our medical colleges – are still operating on 18 to 24-month planning cycles for a problem like this.
No one is positioned to respond to how fast AI is iterating and creating new, potentially dangerous dynamics like this one. Hence Grieve’s plea to put some more emphasis on getting “all” the data into the hands of the patient.
Grieve says that the government and its agencies have been doing a lot of good work with this objective in mind.
But the government and provider conversation about AI and trust and information sharing is still fighting the battle of 18 months ago – is AI unreliable?
In other words we aren’t thinking where AI is, we are thinking where it’s been.
Grieve’s personal experience, an experience now coloured by his experience now coding for FHIR using AI coding tools, is that AI has overtaken human performance in several specific domains and we need to get a handle on trying to understand and manage this speed and the issues it is rapidly throwing out sooner rather than later.
Related
Although there is no record in Australia of AI being involved in even one death to date, we aren’t even asking why this figure, so far at least, compares so starkly to the annual harm and death rate our largely human-centric medical decision-making model seems to generate – at its highest estimate it’s over 200,000 incidents of harm or death per year created by healthcare professional error.
Grieve isn’t getting into a debate here about the fidelity of AI vs human doctors. He in fact points out why humans are so vital in the patient management relationship now and moving forward.
He is pointing to how we are so far failing spectacularly to work out how to keep up. How to we get out of our old models of thinking through governance and establish a framework for thinking at the speed AI is throwing these challenges in front of us.
Who takes the reins on this problem?
Grieve thinks it probably should be the medical colleges.
But isn’t the government best positioned to pull its finger out on this one?
The colleges are constrained by old board governance structures and a fiercely tribal membership trying to protect “traditional” paradigms of medical power.
OK, the speed of government and managing the speed of AI do not seem like two reconcilable dynamics.
But pointing to the amazing work all levels of government and our medical institutions did during covid, Grieve thinks there’s hope we can.