The TGA just threw a spanner into one of the fastest-moving ecosystems in Australian healthcare and called it a screening exercise. It’s a bad start.
Emma Hossack from the Medical Software Industry Association says last week’s TGA digital scribe review is a “responsible preliminary market scan”, not a verdict, and that criticising the regulator for publishing it is “unfair and illogical.”
She’s being kind even calling it a “screening” exercise, really. Of the more than 170 AI scribe type products out there for Australian doctors to choose from, the TGA picked only 12 to look at and didn’t even tell us why they picked the ones they did. Although they named the 12, they didn’t tell us which ones reached the definition of a “medical device”.
For their first go at showing everyone they aren’t asleep at the wheel of trying to oversee and manage the medical AI software explosion, it’s a very big miss, for all stakeholders involved.
The only thing it makes clear is that, as suspected, AI has exposed the organisation as stuck somewhere in the 1950s in terms of its ability to manage what is the most important technology shift in the history of medicine.
They shouldn’t feel bad about that.
The TGA has very good people working very hard doing a lot of good work.
Before AI you could make a very good argument for the organisation being underfunded for what is expected to do.
With AI, it’s not just underfunded, it’s under thought.
The Department of Health Disability and Aged Care also has a lot on and it has to make choices as to where to deploy resources and funding.
It’s making that choice around AI and the TGA at the moment.
AI is so disruptive, so potentially transformational and moving so fast, no long-term established compliance body could hope to be up to speed on it and in a position to move with confidence on what is going on.
Which begs the question: why did the TGA even publish what they had done?
Why not just treat it for what it was: a very early and long overdue attempt to assess a small sample of scribes against their old-world regulatory settings. You know, just to get an idea of just how out of their depth they are.
By publishing it, the TGA has done something consequential with no ability to manage the consequences.
It has told every clinician in Australia who uses one of the 12 named scribes – Heidi Health, Lyrebird Health, Microsoft Dragon Medical One being among them (so most of the market) – that they may be using an unregistered medical device.
It has done this without telling them which eight products it has concerns about. It has done this without any regulatory framework yet in place that would tell them what to do about it. And it has done this in a market it estimates contains around 170 scribe suppliers, which means the chances that your scribe – whatever it is – is also in the frame are substantial.
So the message landing in every GP practice in the country is roughly this: we looked at 12 scribes and found eight we think are probably medical devices. There are about 170 out there. We’re not going to tell you which eight. We don’t have rules yet. Good luck.
Why good luck?
Well, to start with how does the TGA think the medical defence organisations are going to take this news?
The TGA can sit back and let its inadequate and ill-thought-out sample exercise sink in and wait to start some chats with the software vendors, which is happening sometime next week via the MSIA.
But what about the doctors using this stuff day-to-day at capacity?
That was not a screening exercise. It was an information vacuum with a confusing press release attached.
When we asked Heidi Health directly whether the TGA had advised them they were one of the eight, that didn’t answer the question.
Lyrebird Health has so far said they haven’t heard from the TGA at all.
What is the market to make of that?
Does the TGA think people don’t talk behind the scenes on this stuff?
At this point in time, that asymmetry is sitting in the market, potentially doing commercial damage.
The MSIA’s comfort with the TGA’s pace is understandable, if you read between the lines.
At the speed the TGA moves – drug approvals take years, medical device assessments aren’t fast either – by the time any formal regulatory framework for AI scribes exists, every doctor in the country will already be using one.
Ms Hossack is a very clever operator and her job is to protect her flock.
What better protection than to back the speed at which the TGA is capable of getting on top of this problem – a speed which most suspect is never going to be fast enough to actually catch it.
If the TGA can never actually catch it, that’s good for the AI software vendors so long as they don’t actually end up harming anyone.
Even if they did, and there was a firm audit trail to the event, you strongly suspect that it’s not actually going to be the software that we should be blaming.
After all, who among us think software is the infallible technology that we should be trusting the lives of our patients with?
Related
No doctor using AI does for a very simple reason. They know it’s fallible and they also understand that the patient’s safety is, in the end, their responsibility.
Users of Heidi will often say, “yep hallucinates a lot” (they won’t always use this term).
But whenever that user is interrogated a little further, if they haven’t already told you, they’ll say “I’m not much perturbed by that because it’s pretty easy to spot and for all the times it does do something silly it more often puts stuff in front of me that I hadn’t considered and would never have, and which is very relevant to treating my patient”.
At this point of time the market is self-regulating in terms of governance and guardrails because the people building these new AI tools are the only ones who can do that.
The TGA hasn’t got any experts to make any assessments on what’s new every day and even if they did, if they aren’t working at the coalface of this phenomenon, they aren’t going to understand it enough to proffer any sensible view of regulation and governance.
At least not for now.
None of this is ideal. Vendors need sensible oversight by government at some point of time.
But for now the safeguard is that the majority of these vendors – actually the biggest, because they are the ones with the most to lose – can’t afford to screw up in even a small way, so they need to spend a lot of time on guardrails and governance in terms of what they are doing.
One tiny screw-up could end Heidi in a week. The bosses and investors over there know this. And yet they do stuff like deliberately building out AI that can breach the Ts and Cs of the major general practice EMRs in the country.
You can think of why they choose to do this in a few ways:
- They’re crazy and out of control – no;
- They are throwing caution to the wind and taking a calculated risk – not quite;
- They have thoroughly assessed and mapped it all out, war-gamed it all, and understand that doing something so seemingly outrageous isn’t actually representing much of a risk if any to them, the doctors using it or the patients – probably.
This is not how the TGA thinks. They are that far off the mark of what is actually going on here.
This situation can’t go on forever obviously. The TGA or someone in government need to get their heads around it all quickly. Even more quickly now the TGA has stuck its head up and proven that beyond a doubt they are out of their depth big-time.
The only saving grace here is just how slow the TGA is signalling it is going to move on the problem and just how clunky they intend to be in how they move.
That’s why Ms Hossack is backing the TGA, an organisation she rarely has time for normally.
Back to a core issue here the TGA has completely missed.
AI scribes haven’t killed anyone. Millions of clinicians worldwide are using them. The better question – the one almost nobody in regulation is asking – is not how many patients they’ve harmed, but how many they’ve helped.
Doctors using AI are telling us that there seems to be a meaningful patient safety benefit, happening right now, outside any regulatory framework.
The TGA does have a job to do here.
AI scribes that generate clinical recommendations are meaningfully different from ones that just transcribe, and that difference matters for patients.
But the way to do that job is not to lob a half-finished market scan into the public domain and leave clinicians, vendors, and medical defence organisations to guess what comes next.
The TGA needs to sit down properly, map the ecosystem, co-design a regulatory approach with industry and clinicians, and publish a clear roadmap with a timeline – one that acknowledges the roadmap will change as the technology changes, and commits to regular transparent update so the market knows where it stands. Transparency about process, not just about what you’ve found so far.
Last week the TGA gave us something emblematic of an agency that is genuinely out of its depth on a fast-moving technology problem – under-resourced, under-experienced in AI, and not good at communicating with either clinicians or software vendors.
The MSIA thinks this is fine because slow is survivable. On past TGA form, they might be right.
That doesn’t make it good enough though.