Governing Shadow AI
in Clinical Environments

1. Introduction

A practice administrator drafts a prior-authorization appeal letter using a consumer chatbot, pasting in the patient’s diagnosis history to give the model context. A physician dictates a clinical summary into a general-purpose transcription application recommended by a colleague. A billing coordinator uploads a denial spreadsheet to a free analysis tool to identify patterns. None of these individuals intends a compliance violation; each is solving a real problem with a tool that works well. In each case, protected health information has been disclosed to a third party with which the covered entity has no business associate agreement.

This is shadow AI. It resembles the shadow IT phenomenon that preceded it — unsanctioned tool adoption driven by capability gaps in officially provided systems — but differs in three respects that materially change the risk profile. First, the barrier to adoption is nearly zero: no installation, no procurement, frequently no payment. Second, the natural input to these tools is unstructured narrative text, which is precisely the form in which the most sensitive clinical information exists. Third, the disclosure is often invisible to the organization: no data-loss prevention rule fires when a user pastes text into a browser, and no log records what was pasted.

The governance response most commonly attempted — a prohibition circulated by email — addresses none of these properties. This article proposes an alternative structure. It proceeds by locating the exposure within existing HIPAA obligations rather than treating AI as a novel regulatory category (§2), mapping the governance task to an established reference framework (§3), and specifying a control sequence a small practice can actually execute (§4–5).

2. The Exposure Is Not New Law

A persistent misconception holds that AI adoption in healthcare awaits new regulation. In the specific matter of PHI disclosure to AI vendors, this is inaccurate. The governing obligations already exist and have for two decades.

Under the HIPAA Privacy and Security Rules, a vendor that creates, receives, maintains, or transmits protected health information on behalf of a covered entity is a business associate, and the covered entity must obtain satisfactory assurances — ordinarily a business associate agreement — before disclosure. Nothing in that construction turns on the technology involved. A large language model provider that receives PHI in a prompt occupies the same position as a transcription service or a cloud storage provider. The absence of a BAA is the violation; the sophistication of the tool is immaterial.

Two further existing obligations attach. The Security Rule requires covered entities to conduct an accurate and thorough risk analysis of potential risks to electronic PHI — an obligation that extends to systems the organization does not know it is using only in the sense that failure to know is itself a finding. And the Security Rule’s information system activity review provisions presuppose an ability to detect anomalous access and disclosure, which shadow AI usage by design evades.

Enforcement adjacent to this space has clarified regulator posture. The Federal Trade Commission’s 2023 actions against GoodRx and BetterHelp concerned the disclosure of consumer health information to third parties for advertising purposes without adequate consent, and established that the Commission would pursue health-data disclosure under the FTC Act and the Health Breach Notification Rule against entities outside HIPAA’s direct scope. The mechanism differs from OCR enforcement; the signal is convergent. Health information disclosed to third parties without a governing agreement is a supervised risk regardless of which regulator holds jurisdiction.

The practical consequence for a small practice is clarifying. The question to answer is not “what will AI regulation require of us?” but “where is PHI going today, and do we have an agreement with the recipient?” That question is answerable now.

3. Mapping to the NIST AI Risk Management Framework

NIST published the AI Risk Management Framework (AI RMF 1.0) in January 2023 as a voluntary reference for organizations developing or deploying AI systems. It is organized around four functions — Govern, Map, Measure, and Manage — and, like the Cybersecurity Framework, positions Govern as the function that contextualizes the others.

The framework was designed with AI developers substantially in view, and a small medical practice deploying third-party tools occupies a different position than an organization building models. Three of the four functions nonetheless translate directly, and the translation is useful because it supplies vocabulary that auditors and insurers recognize.

Table 1. NIST AI RMF functions translated to deployment-side obligations for a small covered entity. The framework’s development-oriented Measure activities compress substantially for organizations that consume rather than build models.

The Map function deserves particular emphasis because it is both the most neglected and the most tractable. An organization cannot govern tools it has not enumerated, and enumeration in a small practice is achievable in a week — not through technical discovery, which is unreliable for browser-based tools, but through structured disclosure, discussed below.

4. Why Prohibition Fails

The instinctive governance response is a blanket prohibition on AI tools. It fails predictably, for reasons worth stating explicitly because the failure is systematic rather than a matter of poor implementation.

Prohibition does not remove the underlying demand. Staff adopted these tools because they compress genuinely burdensome work — documentation, correspondence, summarization — in environments where administrative load is a principal driver of clinician dissatisfaction. A prohibition that supplies no alternative leaves the burden in place and the incentive intact.

Prohibition is unenforceable through technical means in the relevant channel. Browser-based tools accessed on personal devices, or on practice devices through ordinary web traffic, are not reliably distinguishable from other browsing. Organizations that believe they have blocked AI usage have typically blocked a handful of well-known domains while the actual usage migrates.

Most consequentially, prohibition destroys visibility. Once usage is a disciplinary matter, disclosure stops. The organization loses the information it most needs — what is actually being used and what data has already been exposed — and its risk analysis becomes formally accurate and substantively fictional. This is the worst available outcome: the exposure persists and the organization has lost the ability to see it.

The governance literature on shadow IT reached this conclusion earlier, and the remedy transfers: sanction a workable alternative, make disclosure safe, and reserve enforcement for use that persists after a compliant path exists.

5. A Pre-Deployment Control Sequence

The following sequence is ordered so that each step produces a usable artifact and so that visibility is established before restriction is imposed.

Step 1 — Disclosure amnesty (week 1). Circulate a brief, explicitly non-punitive survey asking staff which AI tools they use, for what tasks, and whether patient information has been involved. The framing must be credible: state plainly that the purpose is to build a compliant path and that no disciplinary action attaches to disclosure during the window. The output is the tool inventory required by the Map function, and it will be more accurate than any technical discovery method available at this scale.

Step 2 — Triage by data sensitivity (week 2). Sort disclosed usage into three categories: no PHI involved and none plausible; PHI involved; and ambiguous. The ambiguous category is usually the largest and typically includes tasks where PHI is not required but is convenient. Many ambiguous cases resolve through a small workflow change — de-identifying inputs, or drafting on templates rather than records — at negligible cost.

Step 3 — Qualify vendors for sanctioned use (weeks 2–4). For tasks that genuinely require PHI, identify vendors that will execute a business associate agreement. This is now a meaningfully populated market: several major providers offer enterprise or healthcare tiers with BAA availability, and a growing set of clinical documentation vendors are purpose-built for it. Three diligence questions matter most and are frequently skipped: whether the vendor will sign a BAA covering the specific service tier being purchased (consumer tiers are typically excluded even where an enterprise BAA exists); whether customer data is used for model training and whether that can be contractually disabled; and where data is retained, for how long, and under what deletion commitment.

Step 4 — Publish the sanctioned list and the policy (week 4). The policy should be short enough to be read. It needs to state which tools are approved for which data classes, who approves additions, what is prohibited without exception, and what a staff member should do after an inadvertent disclosure. A policy that specifies only prohibitions gives staff no route to compliance and reproduces the failure described in §4.

Step 5 — Address prior exposure (week 4–6). Disclosures surfaced in Step 1 that involved PHI to a non-BAA vendor require evaluation under the Breach Notification Rule’s four-factor risk assessment. Some will not rise to reportable breaches; that determination should be documented rather than assumed. This step is uncomfortable and routinely deferred, and deferral converts a manageable finding into an aggravating factor if the matter later surfaces through another channel.

Step 6 — Establish review cadence (ongoing). Tool capabilities, terms of service, and staff practices all change faster than annual policy cycles accommodate. A quarterly review of the sanctioned list, with a standing agenda item for newly requested tools, is proportionate for most practices.

6. The Transparency Dimension

A distinct issue arises where AI is embedded in certified health IT rather than adopted independently by staff. The Office of the National Coordinator’s HTI-1 final rule, published in December 2023, established transparency requirements for predictive decision support interventions in certified health IT, obliging developers to make available specified source attributes describing how such interventions were developed, validated, and should be used.

The practical significance for a practice is that information formerly unavailable is now obtainable. Where an electronic health record vendor supplies predictive functionality — risk scoring, deterioration prediction, coding suggestion — the practice can and should request the corresponding transparency documentation and retain it. This is a low-cost governance artifact that demonstrates diligence regarding embedded AI, which risk assessments frequently omit entirely because it does not present as a discrete tool.

Clinical decision support software also intersects with FDA jurisdiction, and the boundary between regulated device software and exempt clinical decision support is genuinely intricate. Practices deploying diagnostic or treatment-recommendation functionality should not resolve that question informally; it warrants specific regulatory or legal input.

7. Limitations

This analysis addresses the deployment posture of small covered entities and does not treat the obligations of AI developers, health systems building internal models, or business associates offering AI-enabled services, each of which carries materially different duties. The control sequence in §5 assumes a practice small enough that structured disclosure reaches substantially all staff; in larger organizations, survey-based enumeration degrades and technical discovery becomes necessary. The HIPAA discussion reflects the Security Rule as currently in force; proposed modernization published in January 2025 would alter several relevant provisions if finalized. Vendor-market observations in Step 3 describe conditions at the time of writing in a market changing rapidly, and specific claims about any vendor’s BAA availability or training practices must be verified contemporaneously rather than relied upon from secondary description.

8. Conclusion

Shadow AI in clinical settings is not principally a technology problem, and it is not awaiting a regulatory framework that does not yet exist. It is a visibility problem operating under obligations that have been in force since the Security Rule took effect. The organizations that will handle it well are not those that move fastest to prohibit, but those that establish honestly what is already happening, supply a compliant path for the work that motivated the adoption, and document their reasoning as they go.

The alternative posture — a circulated prohibition, an unexamined risk analysis, and an assumption that staff have complied — produces a compliance record that reads adequately until the first time it is examined closely. That examination is more likely to occur during a breach investigation than at a moment of the practice’s choosing.