Blog

GesellschaftJeff Watkins

Jeff Watkins

Founder „North Star Intelligence“ and Founding Member „AI Alliance“, and Lianne Potter, digital anthropologist and Chief AI Security & Ethics Officer „North Star Intelligence“

Citizens Under Algorithmic Rule: How the UK is building an Architecture of Suspicion

The UK is one of the most surveilled democratic nations in the world, and certainly the most surveilled country in Europe. Surveillance is no longer exceptional; it is embedded in the fabric of everyday life, evident in CCTV, the expanding use of facial recognition, the introduction of online age verification, and the broader push toward digital identification. This condition did not emerge suddenly. It developed incrementally, normalised through a series of partial justifications: security, efficiency, public good. What might once have been resisted as intrusive has, over time, come to appear routine. There are ways to push back, at least on the surface. You can wear a mask. Avoid certain online platforms. Refuse to hand over more data than you need to. But some systems do not give you that choice. If you interact with the UK welfare system, opting out is not an option. It is a space in which algorithmic governance is not simply encountered but imposed.

In the UK, as across much of Europe, the welfare state has long been underpinned by the idea that the state bears a duty of care toward its citizens. However imperfectly realised, this settlement rests on a simple premise: that support in times of need is not a concession, but an obligation. We see this as an integral part of the social contract.

Yet the administration of that support remains a persistent site of political contestation. In the UK, public discourse frequently reduces welfare to a moral binary: between the “hardworking” and the allegedly undeserving who are often cast as fraudsters, “scroungers”, or the work-shy. This narrative has proved politically expedient. Rather than challenging the misconception that most claimants are legitimate, successive governments have reinforced it by foregrounding fraud as a central concern, thereby enabling them to increase their surveillance capabilities under the guise of safeguarding public funds. The result has been a significant investment in systems designed to detect and deter abuse. Despite fraud representing a minority of cases, substantial resources have been directed toward developing algorithmic tools to identify it.

Since 2022, the Department for Work and Pensions has expanded its use of machine learning to monitor Universal Credit claimants, embedding automated decision-making more deeply within the welfare system. Even the cost of this infrastructure is difficult to determine: dispersed across contracts, justified through projected “savings”, and obscured by the same opacity that characterises the system itself.

But the reality behind this is not really about efficiency and fraud prevention, it is about what happens when the state begins to treat access to vital support as being conditional on subjects being exposed to systems of permanent scrutiny, trading data and surveillance for vital services. This is suspicion by default, and it affects those most vulnerable in our societies.

Big Brother Watch’s report, Suspicion by Design, examines the Department for Work and Pensions’ growing use of algorithmic tools, data matching, and machine-assisted profiling in the welfare system. The findings are troubling not simply because of the risk of bias or error, but because of what these systems normalise. Systems like this make suspicion an intentional feature by shifting the burden of proof onto the citizen. In doing so, departments are quietly redefining the social contract itself.

That challenge is not only legal or technical, but also a lived experience. Research on Universal Credit describes automated welfare as creating not just compliance demands but administrative burdens, costs that fall on claimants themselves. In other words, automation does not simply streamline the state; it also shifts complexity onto those with the least spare time, money and resilience.

The enforced surrender of data and privacy is especially significant because this is not a consumer platform, like social media, where one can opt out and leave. For many people, welfare systems are a vital lifeline and are thus not optional. Basic conditions for living, such as heating, rent, and food, may all depend on access to these services. When algorithmic profiling enters that space, there is no real opt-out. There is only compliance and an erosion of privacy and autonomy for those least able to refuse.

This is what makes the report’s warnings so significant. Big Brother Watch describes a welfare environment in which people may be subjected to mandatory profiling without fully understanding it. This creates a system in which intrusive checks can be triggered through opaque processes, and transparency is routinely resisted on the grounds that greater openness would allow the system to be “gamed”.

Arguments against transparency are especially worrying because they assume that scrutiny is a threat, while secrecy is a safeguard. Yet in democratic terms, the opposite should be the case. The Dutch SyRI judgment is a good example. It was one of the first major cases in which a court halted a digital welfare fraud system on human rights grounds, and the court found that the system failed to strike a fair balance between fraud detection and privacy. The wider significance of the case lay not only in its privacy ruling but also in its insistence that transparency matters because, without it, the rights to contest a decision and seek a meaningful remedy become nearly meaningless.

Across Europe, legal frameworks are beginning to respond to these developments. Data protection law, and more recently, regulatory efforts such as the EU’s AI Act, attempt to place limits on automated decision-making and introduce requirements around transparency, risk, and accountability. Yet these interventions remain uneven in practice. They are often reactive, contested, and dependent on enforcement that is itself opaque and under-resourced. The result is a growing gap between the formal protections afforded to citizens and the systems that govern them in practice. Rights may exist on paper, but their exercise becomes difficult when the mechanisms of decision-making are difficult to see, challenge, or even fully understand.

The problem, then, is not only whether a model is accurate or not. It is about whether a public body can rely on data-driven suspicion while withholding meaningful explanation from the people it governs. If a system can profile you, rank your risk, or trigger an investigation without you knowing how or why, opacity is not a neutral design choice. It is a political one.

These systems encourage a world in which the poor are more invasively knowable, more easily profiled, and more routinely asked to prove their legitimacy. In anthropological terms, this looks less like administration and more like social sorting: the division of populations into categories of trust and risk, with real consequences for how they are treated.

The real danger here is not only that these tools may be wrong. It is that they introduce us to a different moral framing of government, one in which people seeking help are first treated as line items on a risk register, in which group-level inference starts to displace individual circumstances, and where data exposure becomes the price of social protection.

That is why the central question is bigger than whether a single model is fair enough or a single department is transparent enough. It is whether we are comfortable allowing machine-mediated suspicion to become a default condition of citizenship. For the people who need these systems most, there is often no meaningful opt-out, only a forced accommodation to the terms of machine-mediated suspicion, without a simple path to recourse.

The UK is not an outlier. It is an early adopter. What is being constructed is an architecture of suspicion, one that embeds distrust into the ordinary functioning of the state and makes scrutiny a condition of care. Europe should not assume it will remain untouched. The same systems, logics, and incentives are already in motion. The danger is not only that these tools will spread, but that they will be accepted before they are understood. Because once suspicion becomes the default setting of citizenship, it is no longer confined to welfare. It becomes the way the state sees everyone. That would be an especially dark irony for the country that gave the world Nineteen Eighty-Four: not a sudden descent into totalitarian spectacle, but the gradual construction of suspicion through administrative systems, public justifications, and the quiet normalisation of being watched.

Back To Top

human ist das Leitmedium für strategische Entscheider:innen und alle, die mit KI arbeiten und leben werden. Wir stellen den Menschen in den Mittelpunkt. Für eine lebenswerte Zukunft aller.

Folgen Sie unserer Einladung zum Dialog – werden Sie Teil einer spannenden Reise…

Social