The Intelligence Layer

For roughly a decade, machine learning was a feature. It sat inside a product, improved a specific task, and could be removed without the product ceasing to exist. Recommendation got better. Search got better. Fraud detection got better. If the model was switched off, something degraded, and everyone knew which thing.

That relationship is inverting. Models are increasingly the layer that other things are written against — the component that reads the document, decides the routing, drafts the response, and calls the next service. When that happens, the model stops being a feature of a system and becomes part of the substrate the system assumes.

This is a different kind of dependency, and it is worth being precise about why.

Infrastructure is defined by what fails without it

A useful test for whether something has become infrastructure is not how important it seems, but how much stops working when it is unavailable — and whether anyone can enumerate that list in advance.

By that test, a model integrated into a single workflow is a feature. A model that four teams have independently built against, that handles the unstructured middle of a dozen processes, and that no one has a complete inventory of, is infrastructure. The transition usually happens without a decision being made. Nobody approves the promotion. It occurs because the capability was reliable, cheap, and available, which is exactly how every previous layer of infrastructure arrived.

The consequence is that the failure modes change. Feature failures are local and legible. Infrastructure failures are diffuse: the outputs are still produced, they are still plausible, and the degradation shows up somewhere other than where it originated.

The capabilities that get absorbed are the unmeasured ones

Organisations are reasonably good at instrumenting the tasks they already measured. If a model takes over a step that had a throughput metric and an error rate, the metric usually survives the transition, and a regression is visible.

The tasks most readily handed to a general model are the ones that were never measured, because they were performed by a person exercising judgment in a way nobody thought to quantify: deciding which exceptions to escalate, noticing that a request is unusual, choosing what to leave out of a summary. These are absorbed quietly, and their absorption is not recorded anywhere, because there was no record before.

This produces a specific and awkward situation. The capability an organisation has become most dependent on is frequently the one it has the least ability to evaluate, because the baseline it should be compared against was tacit and is now gone.

A model does not need to be autonomous to become load-bearing. It only needs to be reliable enough that people stop checking.

What follows

Three consequences seem to us to follow from treating this as an infrastructure question rather than a model question.

Inventory precedes evaluation. Before asking whether a model is good enough, it is worth establishing where it is actually being relied upon. In most organisations of any size this is not known, and the answer is not obtainable from the procurement record, because much of the adoption did not go through procurement.

Evaluation has to reflect the deployment, not the benchmark. A model’s published performance describes its behaviour on a distribution that is not yours, under conditions no adversary selected. The relevant question is narrower and harder: how does this behave on our inputs, including the unusual ones, and how would we detect a change.

Degradation needs a detector. Infrastructure fails silently more often than it fails loudly. If the answer to “how would we know this got worse” is that someone would eventually notice the outputs seemed off, then the system has no detector, and the honest description of the current control is hope.

The uncomfortable part

None of this argues against building on models. The capability is real and, in many cases, the dependency is worth taking on deliberately.

The argument is against taking it on accidentally. The organisations that will handle the next few years well are not the ones that adopted fastest or slowest. They are the ones that can answer, on any given day, what their systems have delegated, to what, and what would happen if it stopped being reliable.

That is an unglamorous question. It is also the one that determines whether the intelligence layer is an asset or an unlogged dependency.