Horizon 04
Autonomous Systems
Perception, decision, and action in systems that no longer wait to be told.
When software begins to act.
The shift
Automation executes a decision that has already been made. Autonomy makes the decision. Most of the difficulty lives in the distance between those two sentences.
As perception and planning improve, systems are trusted with wider scope: not one action under supervision, but sequences of actions under exception-based review. Oversight becomes sampling. Sampling becomes ceremony.
The engineering question — can it act correctly — is tractable. The harder questions concern authority: what the system is permitted to do, how that permission is bounded, and what remains true about accountability when it acts within its bounds and the outcome is still wrong.
When software begins to act, the question stops being what it knows.
What we ask
- What is this system permitted to do without a human in the loop?
- How is that boundary enforced, rather than merely documented?
- Which of its actions are reversible, and which are not?
- What signal would cause it to stop, and who is able to send it?
- When it acts correctly and the outcome is harmful, where does responsibility sit?
Lines of inquiry
- 01
Scope and authority
Autonomy is less a property of a system than a permission granted to it. We examine how those permissions are defined, constrained, escalated, and withdrawn.
- 02
Human oversight in practice
Oversight that exists on paper and oversight that would actually catch an error are different things. Rate, latency, and attention determine which one an organisation has.
- 03
Reversibility
Actions that can be undone tolerate far more uncertainty than actions that cannot. Designing for reversibility is frequently cheaper than designing for correctness.
- 04
Accountability under machine action
Liability and assurance models generally assume a decision-maker. We are interested in what those models do when the decision-maker is a process.
Authority is granted, not assumed.
If something in your organisation has begun to act rather than recommend, it is worth establishing precisely what it is permitted to do.