Algorithms Gone Wild: When Government Computes Misery

Automated systems wrongly flag social security debt, causing widespread harm.

A new study reveals Australia’s "Robodebt" system, an automated social security debt recovery program, often failed, causing significant hardship for citizens.

Researchers investigated the Robodebt system, which aimed to find overpayments by comparing income data. They wanted to understand its performance and its impact on people.

The study looked at how the Robodebt system worked within Australia’s social security setup. It drew on public reports, government inquiries, and known details about the system.

How Robodebt Worked

Robodebt automatically compared information from two government agencies. If it found a mismatch, it sent letters demanding people prove they didn't owe money.

Flawed Calculations

The automated system often miscalculated debt. For example:

It flagged people for debt even when they had reported income correctly, simply because employer names didn't perfectly match in different databases.
The system also used averaged income over time instead of actual, specific income.

This led to incorrect debt calculations for thousands.

Significant Negative Effects

This flawed system caused "significant negative effects." People suffered:

Financial hardship
Anxiety
Stress

One report from the Senate inquiry noted that people couldn't afford necessities like food for their children due to these enforced debts. The study also mentions one case where stress from the system allegedly led to suicide.

"Robodebt is a breathtaking example of government by algorithm in its audacious system-level automation, its encoded disruption of administrative principles, and the politicization it generated."

The system issued 20,000 'debt' letters each week, with over 217,403 notices sent by March 2017. Even if someone was cooperative, a 10 percent surcharge was automatically added if clients tried to dispute the debt.

Challenges and Future Research

The study admits challenges in looking into such systems, like how government processes and algorithms can be "black-boxed" or hidden. Future research might need to deeply explore people's real-life experiences with these systems.

This case shows that highly automated government systems, if not carefully designed, can cause public misery instead of public value.

Reference:

Henman, P. (n.d.). The computer says ‘DEBT’: Towards a critical sociology of algorithms and algorithmic governance. Retrieved from https://doi-org.ezp.lib.unimelb.edu.au/10.1177/14407833211015637