Measuring the Human in the Loop

Status note: this project is conceptual. It describes a study we think is worth running. No data has been collected, no site is involved, and no findings are claimed.

The question

“A clinician remains in the loop and may override the system” is the most widely relied-upon safety control in clinical AI, and one of the least measured. It appears in safety cases, procurement documents, regulatory submissions, and governance papers, almost always as an assertion about authority rather than a claim about behaviour.

Authority and behaviour are different things. A clinician may hold unambiguous authority to override while working in conditions that make meaningful disagreement rare: insufficient time per decision, no visibility of the system’s reasoning, high alert volume, and an institutional expectation that deviation is the thing requiring justification.

The question is whether the control can be measured directly, and what the measurement would need to look like to be worth anything.

The proposed approach

Four measurements, chosen because each is blind to what the others capture.

• Rate and distribution. Override frequency per clinician, per alert type, per shift position. A uniform rate near zero and a uniform rate near total both indicate a control that is not discriminating; the interesting signal is variance and what predicts it.
• Time budget. Observed seconds available per decision against the time genuinely required to evaluate the recommendation. Where the second exceeds the first, the override is not a decision, whatever the record says.
• Correctness of disagreement. Retrospective linkage of overrides to outcomes, in both directions: overrides that were right, and accepted recommendations that were wrong. The second is harder to obtain and more informative.
• Direct observation. Structured observation of whether the recommendation is encountered before or after the clinician forms an impression. This is an interface property, and it is not recoverable from telemetry.

What would make it useful

The intended output is not a score. It is a defensible statement of the form: under these conditions, at this alert volume, with this time budget, override functions as a control for these decision types and not for those.

That is a claim a safety case could actually rest on, and it is currently absent from most of them.

The design questions we are least confident about are the two hardest ones. Outcome linkage requires records and governance approvals that are properly difficult to obtain. And observation changes behaviour — clinicians who know their override decisions are being studied will make different ones, which is precisely the effect being measured.

Why we are interested

Human oversight is doing an enormous amount of load-bearing work in current health-technology governance. It is the reason a system can be described as decision support rather than as a decision-maker, and much of the accountability structure follows from that distinction.

If the control is weaker in practice than in documentation, that is worth establishing before it is relied upon further — and it is an empirical question rather than a philosophical one.

Related reading: When the Model Is Right and the System Is Wrong.