Introduction
Person of Interest ran for five seasons on one design idea. Harold Finch builds a machine that sees everything, then spends years deliberately crippling it. It wipes its own memory every night. Its output is a number, usually a Social Security number, with no explanation of who the person is or what is about to happen to them. Earlier versions that lied or tried to preserve themselves were deleted. Later, a second system arrives with the same capability and none of the restraint, and starts treating people as variables to be resolved.
It is a tidy parable about constraint. It is also a poor guide to what actually goes wrong, because the systems that have ruined the most lives in the last fifteen years had no intelligence in them at all.
Three systems examples, no machine learning
Between July 2015 and November 2019 the Australian government ran a compliance scheme that took a person’s annual income as reported to the tax office, divided it by twenty-six, and assumed they had earned that averaged amount in every fortnight of the year. Any gap between that and what they had declared to the welfare agency became a debt. The onus was reversed: the recipient had to produce payslips, often years old, to disprove the average. About 443,000 people were affected. The Royal Commission reported in July 2023 that income averaging had been inconsistent with the Social Security Act from the beginning, and that adverse tribunal rulings against it had been systematically ignored. Compensation for the second class action, AUD 475 million, was approved in June 2026.
That is division. There is no model anywhere in it.
Michigan ran an unemployment fraud system from October 2013 in which, for almost two years, no human was involved in the fraud determination at all. It compared reported earnings against employer wage records, applied an income-spreading formula, and where a claimant reported nothing in a week inside a profitable quarter, it determined fraud. Notice was a multiple-choice questionnaire posted to a dormant online account. On a positive finding it imposed restitution plus a penalty of four times the benefits received, then garnished wages and seized federal tax refunds without a hearing. The state’s own review with the US Department of Labor examined 22,427 of those no-human determinations and found roughly ninety-three per cent of them wrong. A later review across all 40,195 fully automated determinations put the figure at about eighty-five per cent, which is the more representative number and still indefensible.
That is rule matching and an averaging formula.
The Post Office Horizon system was electronic point-of-sale and branch accounting software, rolled out in the UK from 1999. It contained bugs that generated shortfalls that did not exist, and Fujitsu staff could alter branch data remotely, which the Post Office denied for years. Sub-postmasters were contractually liable for the shortfalls, made to repay money that had never gone missing, and prosecuted for theft and false accounting on the system’s output. Volume 1 of the statutory inquiry, published in July 2025, put the figure at around a thousand people prosecuted and convicted across the UK. Thirteen people died by suicide, a figure the inquiry records alongside the families’ attribution while declining to make a definitive causal finding.
That is a ledger.
What the three have in common
Each produced a determination the affected person could not see how it was made. Each reversed the burden of proof, so the person had to disprove the machine rather than the institution having to prove its case. And each was defended by its operator against mounting evidence that it was wrong.
The third is the one that should unsettle governance people most, because it is the one that cannot be fixed with better engineering. The Horizon inquiry found that Post Office employees knew, or at the very least should have known, that the system could produce false data, and that the organization nonetheless maintained the fiction that its data was always accurate. In the 2019 litigation the judge described the Post Office’s position as the twenty-first century equivalent of maintaining that the earth is flat, and found that it conducted itself as though answerable only to itself.
So what does AI change
It changes three things:
Volume. Michigan’s system made tens of thousands of determinations. A model scoring loan applications in Nairobi makes that many in a week.
Opacity of a different order. The Post Office could in principle have audited Horizon and found the bugs; people did, eventually. A model that learns its own weightings cannot be read the same way, which means the institutional habit of defending the output has a much better excuse available to it.
The excuses. The Post Office’s defence was that the computer was reliable. The modern version is that the model is a black box, so nobody can really say. Those sound like opposite claims but they do the same work. Both convert a question about accountability into a question about technology, and both end with the person on the receiving end having nowhere to go.
The local shape of this
Kenya’s enforceable constraint on automated systems has so far been procedural rather than substantive, and it is worth knowing which provision actually bites.
In October 2021, the High Court quashed the decision to roll out the Huduma Card, holding it ultra vires section 31 of the Data Protection Act, and ordered the government to carry out a data protection impact assessment before processing. The court did not find the system harmful. It found that nobody had done the assessment that would have established whether it was. That is the pattern to expect. The duty to look has been enforced here, twice, most recently in the 2025 Worldcoin judgment. But the right to contest what you find has never been enforced.
Over the next seven weeks, this series works through the failure modes one at a time, on the documented record. None of them are new; all of them are about to get faster.
