What does ‘withdrawing consent’ mean when the thing you've consented to is no longer a database, but a model that has already learned from your data?
I spent an hour this week listening to one of the most fascinating discussions I’ve heard in years.
It took the form of a hypothetical board meeting with some well-known digital health leaders at the AIDH’s HIC2026 conference playing the parts of board members at a major hospital.
It was held under Chatham House Rules, so I can’t tell you who was in the room. But I can tell you the question they were trying to answer.
Here’s the scenario:
A hospital has trained an AI model to help identify sepsis in emergency departments. Four years later, a patient writes to the board, withdrawing her consent, and asking to have her data removed from the model.
What now?
You remove that patient’s data, right? And everyone moves on.
Except, it’s not that simple. That patient’s data has already influenced how the model works.
Before the hypothetical, Dr Anthony Porter, a plastic surgeon and clinical senior lecturer at the Australian Institute for Machine Learning, told delegates the challenge was that AI doesn’t simply store information the way a database does.
Once a model has been trained, a patient’s data has influenced thousands, sometimes millions, of mathematical relationships within the model.
The information has effectively been woven through the AI’s “memory”. Removing the original record from a database doesn’t remove what the model has already learned from it.
The obvious solution is to retrain the model from scratch without that patient’s data.
But, as Dr Porter pointed out, that can take weeks, cost enormous amounts of money and, paradoxically, leave the AI performing less well than before.
If patients began withdrawing consent regularly, hospitals could find themselves constantly retraining clinical models rather than using them.
Back to the hypothetical board meeting.
The Informatician on the board pointed out that removing a patient’s details from the dataset wasn’t enough.
“The data is baked into the model,” they said.
“The analogy I use is a blueberry cake. If I’m going to be using a blueberry cake and blueberries are in the cake, and somebody comes to me says, ‘remove the blueberries’; it’s already baked into it.
“In this case, if the data has influenced the weights and the distribution, it’s pretty much baked into it.”
Related
In other words, once a patient’s data has influenced the AI model, removing the original record doesn’t necessarily remove what the model has learned from it.
It’s like asking a human to unsee something. Good luck.
Imagine being the chair of that board. You have one patient insisting her data be removed. Hundreds of future patients relying on the AI continuing to detect sepsis. Retraining the model may take weeks. Leaving it running may offend everything the patient understands about consent.
Which ethical obligation wins?
How does a hospital’s board explain that to a patient – or a patient’s lawyer? How does a board justify using an AI model that was bought four years ago, before these issues had occurred to anyone?
Who is liable? Who is responsible?
The board? The vendor? The regulator?
Suddenly, withdrawing consent has become technical, ethical, legal, clinical, financial – all at the same time.
The question is no longer whether AI works, but “if something goes wrong, can we undo it”.
Dr Porter said in his presentation prior to the hypothetical:
“AI that cannot forget is incompatible with modern healthcare governance.”
The more I thought about it, the more I realised this wasn’t really a column about AI. It was a column about consent.
And I’m not sure our consent frameworks have caught up with what AI has become.