Introduction
In Chen Qiufan’s story The Golden Elephant, a Mumbai family signs up to Ganesh Insurance, whose deep learning models set premiums and nudge policyholders towards lower-risk behavior. The system begins steering a teenage girl away from a boy she likes. She asks him how the AI could possibly know about caste, which has been unlawful for decades. His answer: the AI does not need to know the definition of the castes, all it needs is its users’ history, and our data is a shadow that nobody escapes.
The above is fictitious, but real-world examples exist.
Cost as a proxy for need
In October 2019, Science published an analysis of a commercial risk-prediction tool used to select patients for high-risk care management programs. The tool assigned each patient a risk score. Black patients at any given score turned out to be substantially sicker than white patients at the same score, and were therefore systematically under-referred for the extra care.
In this scenario, Race was not an input, and the model had never seen it.
It was trained to predict future healthcare cost. Because less is spent on Black patients at the same level of illness, for reasons of access, referral patterns, and trust, equal predicted cost meant unequal actual sickness. In the study’s sample of 49,618 patients, Black patients made up 17.7 percent of those flagged at the automatic enrolment threshold; a simulation substituting a health-based label instead of a cost-based one raised that to 46.5 percent. The researchers worked with the manufacturer afterward, and the reported fix was not a change to the model architecture but a change to what the model was asked to predict.
The cruder version
From 2013, the Dutch tax administration ran a risk classification model over childcare benefit applications. High scores were routed to a civil servant for manual review, and the reviewing official was given no explanation of why. One of the risk factors was whether the applicant held Dutch citizenship.
The Dutch data protection authority fined the tax administration EUR 2.75 million for three violations, one of which was using nationality as a risk indicator in an automated selection system, and a further EUR 3.7 million over a separate fraud blacklist holding around 270,000 people. About 26,000 parents were wrongly accused of fraud. The cabinet resigned in January 2021.
One case used a proxy without knowing it, while the other used the thing itself and wrote it into the specification. The governance failure is the same failure at different resolutions: nobody asked what the system had been told to optimize, and nobody had to answer.
What governance would have required
There is a governance question here and it is what, precisely, this model was trained to predict, and how far that sits from the thing we actually care about. If nobody in the room can state the target variable in one sentence, the system is not governable, whatever else has been documented about it. Cost is not health, clicks are not interest, prior arrests are not crime, and repayment history in a market where credit has been unevenly available is not creditworthiness. Every one of those gaps is a place where the social structure of the training data walks in wearing a neutral label.
The target variable is a governance artefact, not an engineering detail, and the frameworks already treat it that way. The NIST AI Risk Management Framework asks for the intended purpose and context of use to be documented before anything is built, and for fairness to be assessed against outcomes, which is what the Science team did when they compared illness by race at equal score. Article 10 of the EU AI Act requires training data to be examined for possible biases that are likely to affect health, safety or fundamental rights, in view of the intended purpose. ISO/IEC 42001 requires an assessment of the system’s impact on individuals before it is used.
The DAMA-DMBOK supplies the piece the AI frameworks assume: a business glossary. Need and cost are different business terms with different definitions and owners, and the model confused them because nobody had written down which one it was predicting. A data steward who owned the definition of need would have been the first person to object. The fix at the manufacturer was not an architectural change but a specification change, which is why this is a governance matter, not a research problem. The defect was in a sentence somebody failed to write, and the control is making somebody write it and sign it.
Where this lands in Kenya
Kenya’s Data Protection Act defines profiling in section 2 by reproducing, almost verbatim, the Article 27 list of prohibited grounds of discrimination: race, sex, pregnancy, marital status, health status, ethnic social origin, color, age, disability, religion, conscience, belief, culture, dress, language, birth. The European definition does not name the protected characteristics at all. Whether a definition framed that way accidentally produces a model that predicts cost and creates a racial gap is an open, untested question.
Meanwhile, the Office of the Data Protection Commissioner’s guidance note for digital credit providers expressly contemplates, in its introduction, that lenders may use mobile phone usage, social media activity and transaction history to assess creditworthiness. It does not then say which lawful basis that processing rests on.
Read the gap alongside The Golden Elephant. A model trained on repayment outcomes, fed behavioral data from a phone, in a market with a documented history of uneven access to formal credit, is not a hypothetical but the default architecture of the Kenyan lending sector, and no Kenyan regulator has yet asked a single provider what its models were trained to predict.
