Automation Is a Moral Position

There is a sentence I hear in almost every automation project, usually early, usually from procurement or a steering group, and it is designed to end a conversation: “This is a purely technical decision.” I have come to believe that sentence is never true. Deciding which work machines will do is deciding which work people will do, and decisions about what people spend a third of their waking lives doing are moral decisions wearing a hard hat.

This is not an argument against automation. I automate things for a living and I am proud of most of it. It is an argument about honesty: the field runs smoother when we admit what kind of decisions we are making, because the admission changes who belongs in the room and what counts as success.

Toil and agency are different targets

The most useful distinction I have found is between automating toil and automating agency. Toil is work that people experience as friction between them and their actual job: the retyping, the reconciling, the copying of numbers from one system into another system that should have been one system. When you remove toil, people feel larger. The caseworker stops being a clipboard and gets her afternoons back for casework. Nobody mourns toil, and the projects that target it generate something rare in enterprise IT: gratitude.

Agency is different. Agency is the part of the job where a person reads a situation, weighs it, and decides — the loan officer’s judgment call, the nurse’s sense that something is off with this patient, the inspector’s discretion about which building to look at harder. When you automate agency, people do not feel larger. They feel like appendices to a system that has already decided, retained only to absorb blame when it decides wrongly. There is a name for the role we create when we do this carelessly: the human crumple zone.

The moral texture of a project is set by which of these you are doing, and the tragedy is that our tooling cannot tell the difference. The process map renders a judgment call and a copy-paste as identical rectangles. Only the people in the process know which rectangle holds their agency, which is the practical argument for a principle I try never to break: the people whose work is being redesigned belong in the room where it is redesigned, not as a courtesy, but as the only reliable instrument for telling toil from agency.

Who bears the residual risk

The second moral question hides in a phrase that sounds like a safeguard: human in the loop. It matters enormously what the human in the loop is actually equipped to do. A radiologist reviewing an AI’s findings with time, tools, and authority to disagree is exercising supervision. A caseworker required to approve four hundred machine recommendations a day, with a dashboard that turns red when she falls behind, is performing supervision while actually providing legal cover. Same diagram. Opposite realities.

Watch where the residual risk settles after an automation, because it never disappears; it migrates. Before the system, a hundred small decisions carried a hundred small risks, distributed across the people making them. After the system, the routine ninety-five percent is handled flawlessly, and the strange five percent — the cases the design never met — concentrates on whoever remains. That person now handles only exceptions, all day, at a pace set by the machine’s throughput. We routinely call this role “oversight” and staff it as an afterthought, when it has quietly become the hardest job in the building.

I now ask three questions of every human checkpoint we design, and I encourage you to ask them of any system you are handed. Does the human have genuine capacity to evaluate, or only to click? Does disagreeing with the machine cost them something — time, metrics, arguments — that agreeing does not? And when the system errs, does accountability land on the person who could not realistically have caught it? If the answers are wrong, the checkpoint is not a safeguard. It is a signature machine, and its purpose is to manufacture someone to blame.

The privilege of the builder

People sometimes ask why I stay in this field if I worry about all this. The answer is that worrying inside the room beats worrying outside it. Every automation project contains dozens of small design choices that never reach a steering group: what gets logged, what gets escalated, whether the operator sees why the system decided, whether the exception path is dignified or humiliating. Each of those choices is small enough that a builder can simply make it well, and together they decide whether the resulting system treats people as colleagues or as components.

That is the quiet privilege of building: the values ship whether you chose them or not. An automated system is a moral position that compiles. It will enact somebody’s assumptions about what people are for, at scale, every day, long after the project team has dispersed. The only question is whether those assumptions were examined or merely inherited from a deadline.

So examine them. Name the agency in the process before you touch the toil. Seat the affected people at the design table and believe what they tell you. Refuse to build signature machines. And when someone says the decision is purely technical, understand what they are really telling you: that the moral decision has already been made, upstream, without discussion — and that they would prefer you not reopen it. Reopen it. That is the job behind the job.