Picture a chief operating officer at a regional health system. Her board has asked a plain question: where are we using AI, and where should we not be? She knows the honest answer is not “nowhere.” Somebody in revenue cycle is pasting appeal letters into a chatbot, a vendor has quietly turned on a new feature in a clinical system, and a physician is dictating notes into a tool no one approved. She also knows that if she says “everywhere, and it is going fine,” she will be wrong about something that matters.
This article is for that leader, and for the many leaders in healthcare who feel pulled between two fears: the fear of moving too fast with technology that touches patients, and the fear of falling behind. We think both fears are reasonable. We also think the way out is neither a ban nor a free-for-all, but a governance design that protects patients and raises productivity at the same time.
Caution Is the Right Instinct
In most industries, a bad AI output costs a company money or embarrassment. In healthcare, it can cost a person their health, and the information involved is among the most sensitive that exists. A tool that sees patient records is subject to privacy law, to contracts with the vendors that run it, and to the trust of patients who never agreed to be a test case. A clinician who acts on a wrong suggestion carries the consequences.
So when a physician, a compliance officer, or a nurse leader says “slow down,” we do not read that as resistance to change. We read it as the same professional habit that makes healthcare safe in the first place. Prudence is the correct default anywhere a decision touches life and death or handles protected patient information. Our argument is not that healthcare should relax. It is that prudence needs a design, because a general sense of caution does not tell a team which tool to approve, which to stop, and what evidence to ask for.
Where Hospitals Stand Today
Two surveys give a rough picture, and both come with limits. The Healthcare Financial Management Association, working with a market research firm, surveyed 233 health system respondents in the second quarter of 2025. It found that 88 percent used AI in some form and that 36 percent had AI tools fully deployed, but only 18 percent had mature programs with developed governance and strategy, and more than a third of those investing in AI had no formal AI data policy. The respondents are association members, so they may not represent every health system, and the results are self-reported.
Black Book Research surveyed 182 hospital leaders from October 15 to November 8, 2025. It reported that 22 percent were highly confident they could produce a complete AI audit trail within 30 days, that 29 percent had implemented and enforced policies on model inventory, lineage, and sign-offs while 48 percent were still drafting them, and that 41 percent named limited vendor documentation as their top audit barrier. We could not determine who funded the survey, the results were distributed through a news release, and “highly confident” is a feeling and not a test. We read both as a signal that use is running ahead of governance, and not as a measurement of any particular hospital.
Systems Written by Humans Fail Too
Here is the point we would add to the case for caution. Healthcare has never operated in a world where software and automation were reliably safe, and the risks of trusting a system are not unique to AI. Three examples, none of them about generative AI, show the pattern.
The Therac-25 was a radiation therapy machine. According to the investigation by Nancy Leveson and Clark Turner, it was involved in six known accidents with massive overdoses between 1985 and 1987, in part because its maker relied more heavily on software for safety and chose not to duplicate many of the hardware interlocks. It is one device from decades ago, but it shows what happens when a team trusts code without an independent check.
A more recent case is a widely deployed sepsis prediction model built into an electronic health record. A 2021 study in JAMA Internal Medicine examined nearly 40,000 hospitalizations at Michigan Medicine and found that the model sorted patients by risk correctly about 63 percent of the time, below the 76 to 83 percent range its developer had reported. It alerted on nearly one in five patients, and roughly one alert in eight pointed to sepsis. Part of the gap reflects different definitions of when sepsis begins, and the study covers one health system. But the lesson holds: a conventional model written by experienced engineers still needs to be tested on your own patients.
And on July 19, 2024, a faulty update to security software took Windows systems offline around the world. Many hospitals canceled nonurgent surgeries and visits and fell back on paper downtime procedures. No AI was involved. A routine update did it.
We draw two conclusions, and they are our own reading. First, the right comparison for any AI tool is not perfection but the alternative: the manual process, the older software, the unreviewed spreadsheet. Second, what these cases share is not a technology but a habit, trusting a system without validating it locally, without a fallback, and without a named person responsible. Governance exists to break that habit, and it should apply the same standard to AI as to everything else.
The Rules Are Moving, and Not in One Direction
It would be convenient if regulators settled the question. They are not doing so in a single direction. The Joint Commission and the Coalition for Health AI published voluntary guidance on the responsible use of AI in 2025, with seven elements that include governance, privacy and transparency, data security, ongoing monitoring, safety event reporting, bias assessment, and training. The Joint Commission then announced a voluntary certification for responsible AI use on June 1, 2026, which evaluates organizations and not individual products and is open to hospitals, critical access hospitals, and health systems.
At the same time, federal rules are shifting. One advisory briefing summarizing federal policy as of October 2026 reports that a December 2025 proposal from the national health IT coordinator’s office would remove the AI model card transparency requirements for certified clinical decision support tools, that FDA guidance issued in January 2026 may let more single-recommendation software reach clinicians without FDA review, and that the major update to the HIPAA Security Rule proposed in January 2025 now has a target for final action in July 2027. That briefing is a secondary source, parts of it are proposals and not final rules, and you should confirm each item with counsel before relying on it.
The practical reading, and it is a judgment, is that you cannot wait for outside rules to tell you what to do. If vendors disclose less, you will need to ask for more and test more yourself. If certification is voluntary, then the work of governing falls to you whether or not you pursue it.
A Different Balance, the Same Machinery
We have written about a two-part design for governing AI in companies of any kind. The first part is the Citizen AI to Production AI pipeline described in The Best of Both Worlds: an open lane where people can experiment with assistive tools, a front door where promising uses are nominated, a regular triage, and a gate that a workflow must pass before it becomes a governed production capability with a named owner. The second is Project Vista, our cadence for surfacing, scoring, sequencing, and proving AI investments, so that money follows evidence and every project has a record of why it was funded and what it was supposed to produce. The governance page on our site shows how they fit together.
Healthcare uses the same machinery with the dials set differently. Our suggestions, which are our judgment and not a standard, look like this.
The open lane is narrower. It is limited to approved tools and to work that involves no protected patient information, such as drafting from public or de-identified material, summarizing policies, and analyzing operational data. Anything that touches patient data requires a tool covered by the contracts and safeguards your privacy rules demand, which is a question for your compliance team and counsel before it is a question for IT. We covered the general problem of unsanctioned use in The Rack of Reinvented Wheels, and in a hospital the stakes of that problem are higher.
The gate to production is higher. We would ask for what the Governance Freeway describes, an owner, an audit record, testing before release, a fallback, and a review date, and in healthcare add three things the sepsis and outage stories suggest: validation on your own patient population before go-live, monitoring after go-live with a stated trigger for pulling the tool, and a downtime procedure that works without it. Where a tool informs a clinical decision, a named clinician stays accountable for that decision. Our companion piece on AI agents under SOC 2 was written for companies without HIPAA exposure; for those that have it, the perimeter is wider and the default for anything touching patient records is stricter.
The scoring is weighted differently. Project Vista scores and sequences candidate projects, as described in our piece on the scoring method. In a regulated clinical setting we would weigh patient-safety exposure and data sensitivity heavily in that sequencing, so that a high-value project with a high-risk profile waits behind a modest, safe one. That ordering is how you earn the right to the harder work.
Where AI Is Already a Reasonable Place to Start
None of this means healthcare should wait. Much of what burdens a health system is administrative: documentation, scheduling, coding support, correspondence, supply and staffing analysis, and the paperwork that surrounds payers. Picture a revenue cycle team that uses an approved tool to draft appeal letters from templates and de-identified summaries, with a person reviewing every letter before it goes out. The risk is bounded, the output is checked, and the time saved is real. We offer that as an illustration of the category, not as a measured result.
Clinical uses can follow once the pipeline has proven itself on the lower-risk work, with the validation, monitoring, and accountability described above. Our piece on the future of care, Healthcare in 2040, argues that a named, licensed professional must stand behind life-or-death decisions; the governance should be built around that expectation and not in spite of it.
What This Will Not Do
This is not legal or clinical advice, and it will not make you compliant with any particular rule or earn any certification. Healthcare regulation is changing quickly, our summary is as of late October 2026, and some of what we cite is proposed and not final. The examples are single cases, and the surveys have the limits we named. We have not measured how this design performs in a health system, so treat our adjustments for healthcare as informed judgment to test, not as a proven method. And if you would like a view of the same machinery applied without the clinical settings, our 90-day audit pathway lays it out.
Where to Start
Start with a list of where AI is already in use, including the tools clinicians and staff brought in themselves and the features vendors added to systems you already license. Mark which ones touch patient information and which touch a clinical decision, and name a person next to each. That list is the front door to everything else. If it would help to have an outside view, our independent assessment starts with exactly that list, and our healthcare practice works with provider and health-services organizations.
If you would rather think it through first, we are glad to talk, with no agenda beyond hearing where you are and what worries you most. You can reach us through our contact page. Caution got healthcare this far. A little design will let it go further.
