Good governance should help worthwhile innovation proceed with clear conditions and credible support. Its practical test is whether it improves decisions throughout use.
Healthcare organisations can approve an AI system without being ready to govern what happens after it arrives.
A procurement process may be complete. Technical assessments may be reassuring. A policy may require clinicians to check every output.
Yet essential questions can remain unresolved: who investigates a pattern of errors, who decides whether a new use is acceptable, and who can suspend the system when its continued use becomes difficult to defend?
At the Australian Institute of Health Executives, this is the problem that led us to develop the AIHE AWARE Framework™, launched this week.
Our starting point is that safe adoption depends on the capability of the organisation around the technology. The evidence leaders require must therefore extend beyond what an AI product can do to how people will use, question, monitor and control it.
This paper explains that reasoning ahead of the framework’s release. It sets out why we believe healthcare needs a more practical connection between AI principles, clinical governance and everyday leadership decisions.
The gap between principles and practice
In our conversations with clinicians, executives, board directors and digital health leaders, a recurring difficulty has been translating broad commitments into decisions.
Safety, transparency, fairness and human oversight are sensible principles. Their meaning becomes much harder to establish when a service must decide whether to introduce a particular system, under particular conditions, for particular patients.
Published research supports the need for that translation. A 2026 scoping review by Wang, Freeman and Magrabi examined 77 healthcare AI governance frameworks. Only 10 included all four components assessed: principles, assessment methods, lifecycle stages and oversight mechanisms. Oversight mechanisms appeared in just 15. The review concerned frameworks relevant to acute care; these figures describe the literature reviewed, not the proportion of hospitals governing AI effectively.
For AIHE, the implication is practical.
A framework earns its value when it helps people make a difficult decision and establish what happens next. A commitment to safety should lead to an agreed evidence threshold. A commitment to oversight should identify someone with the time, information and authority to act.
Without those connections, an organisation can produce an impressive account of its intentions while remaining uncertain about its responsibilities.
Clinical governance provides the organisational home
AI can influence care without making the final diagnosis.
A summary shapes what the next clinician notices. A prioritisation tool affects who is seen first. A prediction changes which patient receives attention. Documentation can influence later treatment, even when the software is described as an administrative aid.
Our position is that governance should follow the consequences of use. Where AI affects patient care, its oversight should connect directly to clinical governance. Digital, cyber security, privacy and procurement expertise remain essential to that oversight. Their findings need to inform an integrated judgement about clinical use.
Consider a hypothetical documentation tool.
It might meet technical security requirements and reduce typing time yet omit a relevant detail from a consultation summary. Assessing the product’s security would not establish whether the clinical review process can reliably catch that omission before another practitioner acts on the record.
This is why an organisation’s AI inventory should identify what each system influences, where it is used and who depends on its outputs. A list of software licences tells leaders much less than a map of the clinical decisions and workflows those products affect.
Regulatory status and local readiness answer different questions
Australia already has relevant regulation and professional guidance.
The Therapeutic Goods Administration regulates AI software when it meets the definition of a medical device. Its approach is based on intended purpose. Medical devices generally require inclusion in the Australian Register of Therapeutic Goods before legal supply, unless an exclusion or exemption applies. It would therefore be inaccurate to describe every healthcare AI tool as a TGA-approved medical device.
The Australian Commission on Safety and Quality in Health Care’s AI Clinical Use Guide addresses responsibilities before, during and after use. It covers evidence, limitations, patient communication, review of outputs and ongoing monitoring.
Our interpretation is that organisations must connect these requirements and expectations to their own operating conditions. Regulatory status is relevant evidence; leaders also need to understand the proposed patient population, workflow, staffing and safeguards.
A practical framework should help make those connections. AWARE™ is intended to support that work alongside applicable requirements and guidance. Its purpose is to improve organisational decision-making, and its use should not be presented as certification or a guarantee of safe performance.
Related
Accountability needs to be specific enough to use
The statement that humans remain accountable is necessary, but insufficiently precise to organise a health service.
A clinician may review an individual recommendation. An executive may authorise deployment. A service lead may monitor operational performance. Procurement and digital teams may manage different aspects of the supplier relationship. The board needs assurance that these arrangements are coherent and effective.
AIHE’s view is that these responsibilities should be explicit and connected. A named owner should be able to obtain evidence, resolve concerns and secure action. Assigning a name without the corresponding authority creates an appearance of accountability that may fail when challenged.
The clinician’s role deserves particular care. The Commission’s guidance makes clear that clinicians retain responsibilities for reviewing AI outputs and the records or decisions informed by them.
That should prompt leaders to ask whether staff can realistically fulfil the review expected of them. Have they received relevant training? Can they recognise the tool’s limitations? Does the workflow allow a meaningful check?
These are questions about governance design. The allocation of legal liability in an individual case is a separate matter, dependent on its circumstances and applicable law. A framework should clarify operational responsibility without pretending to settle every legal question.
Human oversight must work under real clinical conditions
Imagine a service introducing AI-generated summaries to reduce documentation time. Staff are instructed to check every summary, while appointment schedules are shortened in anticipation of the time saving.
The implementation has now changed the conditions under which its central safeguard must operate. Whether the review remains credible depends on the work clinicians actually have to do, the errors they can detect and the time available to correct them.
This hypothetical example illustrates why we regard workforce readiness as a governance issue. A training attendance record cannot demonstrate that a team can use a system safely during a busy shift. Leaders need to understand how staff respond when an output is plausible but wrong, when it conflicts with their judgement, or when the technology is unavailable.
They also need to know whether raising a concern will be welcomed. If questioning the system is treated as resistance to innovation, relevant information may never reach the people responsible for oversight.
In our view, credible human oversight requires both practical capability and organisational permission to disagree.
Approval begins an ongoing obligation to look again
A decision to introduce AI rests on assumptions about its intended use, supporting evidence and operating environment. Those assumptions can become outdated.
A vendor may change a product. A service may extend its use to another patient group. Staff may begin relying on outputs in ways the original assessment did not anticipate. Even a model that does not learn or update during use can encounter different data, patients or workflows.
The TGA specifically identifies the need for manufacturers to monitor changes in software functionality and intended purpose, including scope creep. At the organisational level, we believe leaders need an equally deliberate approach to recognising when the basis for local approval has changed.
This is the reasoning behind our emphasis on ongoing assurance and trigger-based reassessment. A scheduled review creates a regular opportunity to examine performance. A defined trigger brings that examination forward when something material changes.
The important question is what a trigger causes. Who receives the concern? Who assesses its significance? Can use continue during the review? Who can restrict or pause it, and what alternative care process is available?
A requirement to review is incomplete until it is connected to a response. This connection matters particularly once a service has become operationally dependent on the technology.
Boards need evidence they can interrogate
Boards require sufficient understanding to test the quality of management’s assurance. They need reporting that makes the organisation’s position clear: what is in use, what matters most, what is uncertain and where intervention is required.
A dashboard showing high uptake and time saved may describe implementation success while leaving clinical risk largely unexplored. A statement that no incidents have been reported also needs interpretation. Leaders should ask how concerns would be recognised, recorded and connected to the relevant system.
Our work on AI assurance reporting has therefore focused on the relationship between information and decisions. Useful reporting should make material changes visible, explain the evidence supporting continued use and show whether unresolved concerns have an owner and a response.
The board should be able to trace a reassuring conclusion back to its basis. Where evidence is incomplete, the limitation should be visible. A green indicator should not conceal an unanswered question.
For management, this requires a reporting process that draws together clinical experience, technical performance, patient feedback and organisational risk. The value lies in what those sources reveal when considered together.
Patient trust belongs in the design
An organisation may regard an AI application as efficient while patients experience its use as confusing or intrusive. Governance should make room for that difference before implementation decisions become difficult to reverse.
The Commission’s guidance addresses explaining AI use and establishing proportionate consent processes, with the approach depending on the application and context. AIHE’s broader position is that patient perspectives should help shape what acceptable use looks like.
For example, a service introducing a documentation tool should consider what patients will be told, how questions will be answered and how concerns about the record will be corrected. A system affecting access or prioritisation warrants attention to whose experience may be poorly captured by overall performance figures.
Consumer participation can expose assumptions that an internal implementation team has missed. In Australian healthcare, that includes taking cultural safety seriously and engaging appropriately with Aboriginal and Torres Strait Islander people when decisions affect their care or data.
Trust becomes more defensible when the organisation can explain its choices and demonstrate how patient concerns have influenced them.
Governance must remain proportionate and useful
The argument for stronger governance is also an argument for using organisational effort well.
A low-risk internal application and a tool influencing urgent clinical prioritisation should not automatically face identical scrutiny. The required evidence and oversight should reflect the potential consequences, scale of use, uncertainty and ability to detect and correct failure.
Nor should leaders assume that establishing a new committee will resolve fragmented responsibility. A committee is useful when it closes a defined gap and has a clear relationship with existing decision-makers. Otherwise, it can add another place where an issue is discussed without anyone acquiring authority to resolve it.
We also believe the benefit case belongs within ongoing assurance. Leaders should establish what improvement justifies introduction and whether that improvement occurs in practice. Claimed time savings warrant examination if they create additional checking work elsewhere. Continued use deserves reconsideration when the expected value does not emerge.
Good governance should help worthwhile innovation proceed with clear conditions and credible support. Its practical test is whether it improves decisions throughout use.
Why we developed AWARE
AWARE™ grew from these organisational questions. We wanted a healthcare-specific approach that clinicians, executives and boards could use to connect AI principles with their responsibilities in practice.
Its intended contribution is to bring accountability, workforce capability, ongoing assurance and the consequences for patients into a shared governance conversation. It places AI influencing care within clinical governance while recognising the digital, cyber, privacy and commercial expertise that responsible use requires.
The reasoning presented here is a statement of AIHE’s approach. The framework will need to earn confidence through its usefulness in practice and through evaluation; publication alone cannot demonstrate improved safety.
For healthcare leaders, the next step is to examine how that approach becomes a practical structure. The AIHE AWARE Framework™ will develop the connection between the questions raised in this paper and the governance decisions organisations need to make.
Before its release, consider one AI system already influencing care in your organisation. Could the people responsible explain the evidence supporting its continued use, the changes that would prompt reassessment and who could act if that evidence became less reassuring?
That is the conversation AWARE™ is designed to make more practical.
Dr Sidney Chandrasiri is the CEO of the Australian Institute of Health Executives.
Professor Luis Prado is the chief academic officer of the AIHE.
This article was first published by the AIHE. Read the original here.