Current AI laws are not adequate, the incoming president of the AMA has told a parliamentary inquiry.
The Australian Medical Association has warned that AI in healthcare should be considered high risk by default, warning that purely administrative tools carried the risk of “drifting” into clinical decision-making.
In its submission to the inquiry, the AMA recommended that AI in healthcare “be designated high risk by default”.
“The application of AI in healthcare should be treated as high risk in every case, given the potential for systemic error, privacy breach, algorithmic bias and irreversible patient harm,” the submission said.
“Australian frameworks have at times been equivocal on this point, leaving too much to case-by-case assessment.”
AMA president-elect Dr Michael Bonning appeared at the Joint Select Committee on Artificial Intelligence inquiry in Sydney today, where he was asked whether the recommendation of automatically considering AI healthcare tools as “high risk” should apply to both clinical and administrative tools.
“There’s nuance to that,” he said.
“As many of you will have seen, while a tool may well be originally identified for administration, we have seen that there … can be significant drift in and through actions by the software vendor to update, change, broaden their product offering to stay ahead in an arms race in their specific field.
“That can mean that something that was originally administrative now sits within the space of shaping clinical decisions.
“I would always be wanting to think about what something actually does, rather than the label that we give for it, because it’s about how these things are affecting …. people’s clinical outcomes and engagement.
“If something is purely administrative and remains purely administrative, sure, the environment in which it operates still needs to be reviewed, but there is probably less risk there, though not no risk.”
In his opening statement, Dr Bonning said: “Doctors are already using AI in consulting rooms across the country.
“We see that with the proliferation of AI scribes, and we know that decision support tools are informing diagnosis and treatment,” Dr Bonning said in his opening statement.
“We know that when applied safely, the AI can lift clinical productivity. It eases pressure on our workforce, it reduces the administrative burden and makes care much more consistent.
“We want Australia to capture those benefits on behalf of the population as a whole, noting that this gives us opportunities to improve care access and also care consistency across the whole country.”
But Dr Bonning said AI should be judged “against real clinical practice”.
“There are many papers that are then sensationalised, showing that AI exceeds doctors in certain tests or on certain papers.
“And while that is known and understood … that’s not the impossible standard that I think we apply.
“While AI output can look technically accurate, it can often still be wrong for the patient who is sitting in front of us.
“A treating clinician can reliably catch that, but clinical oversight must sit at the heart of how we govern AI in health, which is critically important to us.”
Dr Bonning said the AMA believed that Australia’s current laws were not adequate for AI in healthcare and proposed five priorities for reform:
- To treat all healthcare as a high-risk setting by default with oversight scaled to how autonomously a tool operates, how serious an error could be, and how readily a decision can be reversed
- A dedicated health AI governance body should be established that involves practicing clinicians to support regulators and identify emerging risks early
- Legislated and overarching framework working with the TGA and AHPRA to widen their scope so that general purpose tools used in clinical care, including scribes, are governed
- That liabilities should follow control: clinicians remain responsible for patient care, and software vendors must be accountable for the design, training, validation and security of their products.
- That patient data is protected: patients own their own health information, we have Australian and offshore data storage if appropriate, and models are trained on Australian data to reflect the population.
“The AMA very much welcomes the direction of the committee’s early advice, including its support for a risk-based framework, human oversight, incident reporting, and data sovereignty,” Dr Bonning said.
“If we can build health-specific expertise into those arrangements, and Australians can therefore adopt AI in healthcare with confidence, we’ll get a much better outcome, both for the population of Australia, but also for the interrelationship between that population and their healthcare providers.”
Related
Dr Bonning said AI tools should involve shared decision-making with the patient, explaining “this is what we know, this is what’s unclear or not entirely true, or this is where the AI and the clinician disagree.
“Open up that shared decision-making with the patient about what should we do.
“In lots of conditions that are progressive … that is often already a difficult question to know what to do until something fully declares itself.
“That will likely be improved by AI, but it will not remove some of the uncertainty associated with that.”
Train AI on local populations: radiologists
The Royal Australian and New Zealand College of Radiologists’ (RANZCR) AI committee chair, Dr Fahana Ali, was asked whether Australian health data should be used to train or validate AI models.
“Yes, absolutely,” she said.
“But it has to happen within proper safeguards.
“The reason that we should be using Australian health data is because we need to make sure that models are accurate on our unique population.
“What we know is that if the training data does not contain a particular type of population … when that model is applied to real-world populations … then the model is not going to be as accurate.
“We have to demonstrate accuracy by validating models in our Australian data. However, this needs to happen with the proper safeguards.”
Dr Ali said the responsibility for safe and effective implementation of AI tools should be shared.
“Firstly, the clinician is responsible for patient care, so they have their responsibility when an AI tool produces an output.
“The second responsibility … is for the health service organisation. And the third responsibility lies with the developers to make sure that their tool works as they say it does.”
Dr Ali said it was the responsibility of the organisation procuring and implementing an AI tool to basically validate it in the local population before it went live.
“It’s a lot more complex than just picking something off the shelf, plugging it in, testing that it works, and off you go.”
Dr Ali said RANZCR’s position was that clinicians were responsible for checking the output of the tool, but health services should be responsible if clinicians had not been trained or the tool had not been properly validated.
“I have the ability to edit and change those based on my years of training and experience.
“If there is a lack of adequate training on how to use the AI tool, what needs to be checked … or if the error has come from insufficient validation … before you deploy the AI tool, then that responsibility should lie with the health service organisation because that is not the clinician’s fault.”
Employers ‘expect higher output’
Earlier today, union representatives said there were a lack of productivity benefits being returned to workers from the use of workplace AI due to work intensification.
Assistant secretary of Unions NSW Todd Pinkerton said a survey of 6000 workers found that the majority reported either no change in levels of unpaid overtime, or an increase in unpaid overtime.
One fifth of respondents reported that their employers expected higher output as a result of AI integration with no change in work hours, Mr Pinkerton said.
“When AI is introduced to the workplace, the result is work intensification without direct benefit to the workforce, driven usually by employers whose expectations drastically outstrip the benefits from the technology in its current state.”
Mr Pinkerton recommended stronger requirements for employers to consult with workers and unions before AI systems are produced, procured or implemented.