An AI system doesn’t need to be told to discriminate to produce a discriminatory outcome. That’s the uncomfortable core of this problem, and it’s why “we didn’t design it to be biased” isn’t a defence under UK discrimination law. Intent has never been the test. Effect is.

This series has already covered the clearest UK example of this in practice: Manjang v Uber Eats, where facial recognition software was alleged to be less accurate for Black drivers. This piece goes further, into the actual mechanics of how AI systems produce biased outcomes in the first place, what the Equality Act 2010 actually requires when they do, and where employers’ defences genuinely hold up and where they don’t.

Explore: AI Grievances and Employment Tribunals: A Guide for HR Teams

ICO’s own guidance uses a simple example to explain how this happens without anyone intending it to. A bank builds an AI system to assess credit risk, trained on a large dataset of past borrowers. During testing, the bank finds the system gives women lower credit scores. Nobody told it to do that. It happened because men were overrepresented in the training data, so the model paid closer attention to the patterns that predicted repayment for men, and less to the patterns that applied to women, even where women in the data were, on average, more likely to repay.

Removing an obvious characteristic like gender doesn’t fix this. ICO calls this “fairness through unawareness,” and it doesn’t work, because other data points often stand in for the characteristic that was removed:

• Postcode can correlate with race
• Working hours can correlate with gender, since women are still more likely to work part-time

A system never given someone’s gender directly can still reproduce gender-based outcomes through these proxies, with high statistical accuracy, without ever being told what it’s actually detecting.

A well-known real-world example, though a US one, illustrates the same mechanism clearly:

• Amazon built an AI tool to screen job applications, trained on a decade of resumes submitted mostly by men
• The system taught itself male candidates were preferable, and began penalising resumes that included the word “women’s” and downgrading graduates of certain all-women’s colleges
• Engineers tried to correct for the specific terms flagged, but couldn’t be confident it wasn’t finding other ways to reproduce the same bias
• Amazon scrapped the tool entirely

Nobody built it to discriminate. The training data did the rest.

The Equality Act 2010 draws a clear line between two kinds of discrimination, and understanding which one applies matters, because they work differently.

• Direct discrimination (Section 13): treating someone less favourably specifically because of a protected characteristic. It admits almost no defence, intent or good motive doesn’t save the employer.
• Indirect discrimination (Section 19): the one that applies to almost every AI discrimination claim. A provision, criterion, or practice, applied the same way to everyone, ends up disadvantaging people who share a protected characteristic. An AI scoring tool is exactly this kind of practice. It doesn’t need to target anyone. It just needs to produce statistically unequal outcomes for a protected group.

There’s a genuinely new wrinkle here too, since January 2024. Section 19A allows someone to bring an indirect discrimination claim even if they don’t personally hold the protected characteristic being disadvantaged, as long as they can show they suffered “substantively the same disadvantage.” For AI-driven decisions applied at scale, where the same scoring system touches a large, mixed workforce, this genuinely widens who can bring a claim.

Unlike direct discrimination, indirect discrimination can be defended. If an employer can show the practice was “a proportionate means of achieving a legitimate aim,” the claim fails. Whether that defence actually holds up is where the next section goes.

The mechanics described above aren’t hypothetical. They show up in specific, recurring places across the employment lifecycle:

• Recruitment and hiring screening tools. CV-scoring systems trained on past successful hires can learn to favour whatever patterns defined those hires, including patterns that track race, gender, or educational background rather than genuine ability.
• Performance scoring and monitoring systems. A system that flags “underperformance” based on output volume or activity levels can disproportionately penalise disabled employees whose conditions affect pace, or employees on flexible or part-time arrangements.

Manjang v Uber Eats UK Ltd remains the clearest UK example of this actually reaching a tribunal. The claim centred on facial recognition verification software allegedly performing less accurately for Black drivers, a system trained on data that didn’t represent the population it was later used on, producing a disparate outcome without anyone designing it to. The tribunal allowed the claim to proceed past an early strike-out challenge; the case later settled before a full hearing decided the underlying question.

Explore: Tribunal Rulings, Regulatory Action and Claim Trends on AI in the Workplace

The most common instinct when something goes wrong is to point at the tool. “We didn’t build it, the vendor did” feels like it should matter. Legally, it doesn’t.

Janus Compliance and multiple other UK legal commentators are consistent on this point: a vendor contract does not move Equality Act exposure to the vendor. The employer is the one applying the provision, criterion, or practice to its own staff or candidates, and that’s what the law looks at. A tribunal isn’t going to accept “the algorithm produced the shortlist” as a defence, any more than it would accept “a manager made the call” as a defence to a manager’s discriminatory decision.

Fieldfisher notes this gap is significant enough that the TUC has proposed amending the Equality Act 2010 directly to codify employer liability for AI-driven decisions, though this remains a proposal rather than current law.

The one defence that does exist, for indirect discrimination specifically, is objective justification: showing the practice was a proportionate means of achieving a legitimate aim. But this bar is set high, and case law has generally made it harder to clear than employers expect. Cost or convenience alone rarely qualifies as a legitimate aim. Demonstrating proportionality typically requires showing that less discriminatory alternatives were genuinely considered and rejected for good reason, not just that the tool was efficient or that it hadn’t caused a problem before.

In practice, this means vendor due diligence has legal weight, not just operational value. Understanding what data a tool was trained on, and whether it’s been tested for disparate impact, isn’t a nice-to-have before deployment. It’s the evidence an employer would need to mount a justification defence later, if it ever comes to that.

None of this is unmanageable, but it does require treating bias as something to actively test for, not something to assume away. ICO’s own guidance recommends:

• Testing before deployment, comparing training data against a genuinely representative benchmark, such as census data for the population the system will actually be used on
• Ongoing monitoring, with clear thresholds set in advance for how much disparity is acceptable before the system gets pulled and reviewed, since a tool that looked fine at launch can drift as the population using it changes

The standard that ties all of this together is one this series has returned to repeatedly: meaningful human involvement. Not a person glancing at an output and approving it by default, but someone with genuine authority to catch and correct a disparate outcome before it becomes a pattern. Bias testing tells you where the risk is. A real human review point is what actually stops it from reaching someone unfairly.

Explore: Building an AI Grievance Policy Before You Need One

How Avado Can Help

Understanding how AI can quietly produce a discriminatory outcome, and knowing what a defensible response looks like when it does, is exactly the judgement HR needs to bring to every decision a system influences. Avado’s HR Compliance for Managers course, presented by employment law specialist Amanda Chadwick, builds that judgement directly, covering discrimination law alongside the wider disciplinary and grievance landscape managers now operate in.

Explore HR Compliance for Managers and make sure your organisation can defend every decision a system helps make!