We must have humans in the loop, not because it leads to better healthcare, but so human decisions and ‘rational’ actions are visible and accountable.
“Human in the loop” (HITL) has become a mantra, almost a cliché, when talking about processes involving artificial intelligence (AI) in healthcare.
When we’re told there’s a human in the loop, it commonly refers to a system design where people are active participants in the operations, supervision and decision-making of automated systems, including machine training.
According to IBM: “HITL means that humans are involved at some point in the AI workflow to ensure accuracy, safety, accountability or ethical decision-making”.
The Stanford Institute for Human-Centered AI states, in relation to HITL, said:
“Humans may provide guidance, correct errors, or make final decisions to improve the accuracy and reliability of AI. This approach combines the strengths of both humans and machines, ensuring that complex or high-stakes tasks benefit from human judgment and oversight.”
There’s an inherent assumption that human judgement and oversight in complex decision-making improves accuracy, can discern errors; and that human judgement is needed, or rational.
I burrowed into the frightening rabbit hole of reports on medication-related errors, clinical and hospital mistakes, misdiagnoses and medical calamities.
Humans make mistakes – in healthcare, in the workplace, in personal relationships, in leadership, on the roads, and definitely in refereeing and umpiring sport.
Health practitioners make mistakes, and like everyone else, they bring prejudices, biases, political, religious, cultural and historical opinions and beliefs to their workplace.
Racism, misogyny, discrimination and ignorance in healthcare are known and documented.
Unlike AI and machines, humans experience fatigue, distractions and “emotional baggage”. Family breakdowns, marriages dissolving, children with myriad issues and problems; debt and financial anxiety; sex-related concerns; and physical and mental health problem like burnout can affect rational decision-making.
We know about human frailty.
My approach to “humans in the loop” is influenced by the groundbreaking documentary Fog of War, (Errol Morris, 2003).
In the segment titled Lesson #2: Rationality will not save us, the former US Secretary of Defence Robert McNamara, who devised the logistics of carpet bombing and led the American war against the Vietnamese people, noted that rational decision-making during the Cuban missile crisis almost led to nuclear war.
Related
“I want to say,” said McNamara, “and this is very important, at the end, we lucked out. It was ‘luck’ that prevented nuclear war. We came that ‘close’ to nuclear war at the end. Rational individuals. Kennedy was rational. Khrushchev was rational. Castro was rational. Rational individuals came that close to total destruction of their societies. And that danger exists today.”
It may seem a stretch to associate nuclear war with decision-making in healthcare, yet killing people and saving lives both require humans in the loop and are based on rational decision-making.
Our fear of AI and algorithms is a fear of loss of accountability and blame.
We accept that as humans, mistakes, errors, bad judgements and poor decisions are an entrenched part of who we are as a species, but AI making mistakes is frightening, a scenario from the now dated Terminator movies or pulp science fiction novels.
AI and algorithms cannot be empathetic; cannot read body language and nuances. Machines can literally but not emotionally “walk a mile” in a human’s shoes.
See Robodebt. See the controversy over the Integrated Assessment Tool (IAT) in evaluating Support at Home programs for the elderly. Rational decisions are made by programmed algorithms, yet as we saw with Robodebt, a human in the loop must be held accountable.
The 4 Corners exposé of the IAT in aged care highlights our need to have a face, like the minister for aged care and seniors Sam Rae, to be held accountable for black-box decisions.
Blaming others has existed since the Bible. Our society and structures are based on directing blame, anger or grief at those we identify as responsible.
Courts and legal systems can’t find an algorithm guilty. Parliamentary inquiries cannot make a black box appear. Detectives cannot interrogate AI and charge it with criminal negligence.
We can’t trust our health system if we can’t blame politicians, doctors, allied health experts, nurses and administrators when things go wrong.
And this is the dilemma of not having humans in the loop.
Humans don’t necessarily make rational decisions. We make mistakes, sometimes lethal and deadly mistakes. Oftentimes they aren’t deliberate mistakes; humans have a desire to kill and harm one another.
Modern warfare is rational men (it’s almost always men!) making rational decisions to kill as many of the “other” as possible; now aided by AI technology.
But we need the human in the loop because that’s who we hold responsible.
In Scapegoat: A History of Blaming Other People, Charlie Campbell writes that humans have a pattern of behaviour that refuses to accept ourselves as we are – which is that we’re far from perfect – so we seek explanations for what went wrong.
We find a scapegoat, then we execute, imprison, fire or publicly shame those we hold responsible.
Assigning blame is the defining feature of ancient and modern societies. The anger over the Robodebt inquiry was the absence of “heads” rolling; of names and faces of those punished.
We must have humans in the loop, not because it leads to better healthcare, better processes and functions, better management of sustainable agriculture and improved building, transport and manufacturing, but so human decisions and “rational” actions are visible and accountable.
Robert McNamara might be right – those rational decisions made by humans risks our total destruction. Perversely, we understand and fear this less than we do decisions made by AI.
Simon Tatz was director of communications for Mental Health Australia, director of policy for Mental Health Victoria and co-author of The Sealed Box of Suicide: The Contexts of Self-Death. He is currently the AIDH’s general manager of policy, advocacy and workforce advancement.