Who is accountable when an AI-assisted eligibility decision is wrong — the caseworker, the agency, or the vendor?

The shift

Reading a benefit application, classifying it against eligibility rules, and producing a determination goes from scarce caseworker-hours to near-free and instant. What stays exactly where it was is due process: a wrong denial that harms a citizen triggers a legal, public obligation to explain, review, and correct — and that obligation attaches to the state, not to whatever produced the determination.

The axioms

• Processing and classifying an application is the caseworker's core job — the thing they're staffed and trained to do. Rests on application-to-rule matching being scarce, expensive human labor.
• A decision is accountable because a specific human official made it and can be identified as its author. Rests on the human decision-maker and the accountable party being the same person by default.
• "Human review" of a determination is a real check because the reviewer has the time and information to actually reconsider it. Rests on review capacity being roughly matched to decision volume.
• The citizen's right to appeal means a responsible human re-examines the decision and can be held to it. Rests on there being a human who actually decided, to appeal to.
• The government owes due process — notice, a reasoned explanation, and a chance to contest — for any decision that denies a benefit. Rests on due process being a legal obligation of the state that can't be delegated away.
• Ambiguous cases — conflicting evidence, unusual circumstances, facts that don't fit the form — require a judgment call. Rests on judgment under genuine ambiguity being scarce.
• Public legitimacy of the benefits system depends on citizens believing a fair, answerable institution stands behind decisions. Rests on trust being tied to identifiable institutional accountability.

Invalid axioms

1. Processing and classifying an application is the caseworker's core job. Reading a claim, extracting the relevant facts, matching them against eligibility criteria, and drafting a determination is squarely what current models do well and cheaply. The core task the role was built around is now the abundant part. The habit-trap: agencies still describe, staff, and grade the job around throughput — cases closed per week — as if moving applications through the rules were the scarce, valuable act, when the scarce act has moved to catching the cases the model gets confidently wrong.
2. "Human review" is a real check because the reviewer has time and information to reconsider. This was true when decision volume was capped by how fast humans could produce decisions. AI uncaps production while leaving review capacity flat, so "a human reviewed it" quietly degrades from reconsideration to a signature on a queue moving faster than anyone can read. The habit-trap: agencies keep "human-in-the-loop" as the compliance answer and the political reassurance, without staffing review to a level where the loop does anything — the sign-off survives as ritual after it has stopped being a check.

Unchanged axioms

1. The government owes due process for a decision that denies a benefit. Notice, a reasoned explanation, and a real chance to contest are obligations the state carries because it is the state — they don't transfer to a vendor by contract or to a model by deployment. Cheaper determinations don't make this cheaper or smaller; more automated decisions make it more load-bearing, because there are more determinations that must each be explainable and contestable.
2. The citizen has a right to appeal to a responsible human who can be held to the decision. Appeal presupposes someone with the authority and the information to actually re-decide, and the standing to own the new outcome. A model can be re-run; it cannot be answerable. This survives untouched — and gets harder to honor, not easier, as the original decision becomes something no human meaningfully made.
3. Ambiguous cases require a judgment call. Conflicting evidence, facts that don't fit the form, circumstances with no clean precedent — this is where pattern-matching against past cases is weakest and confidently-wrong output is most likely. The determinations that most need a human are exactly the ones automation is worst at flagging as needing one.
4. Public legitimacy depends on an answerable institution standing behind decisions. Legitimacy is a relationship between citizens and a body they can hold to account, not a property of the output. A fluent, correctly-formatted denial carries authority only because an accountable institution is behind it; strip the accountability and it's plausible text with a letterhead.

New axioms

1. When the model produces the determination and a human signs it, accountability diffuses across caseworker, agency, and vendor with no one holding the whole of it. The caseworker didn't really decide; the agency deployed a system it doesn't fully understand; the vendor built a tool it disclaims liability for and calls decision-support. Each can point to another. The legal duty still lands on the state, but the internal locus of responsibility — whose fault, who fixes it, who is answerable to the citizen — has no default owner the way a named caseworker's decision once did.
2. When a citizen appeals a decision to a human who only rubber-stamped it, the appeal runs to someone with no independent basis to defend or overturn it. The reviewer's "yes" was itself an artifact of volume, not judgment, so the appeal re-enters the same undecided decision rather than reaching a person who can meaningfully reconsider. Government must solve for what a genuine second look means when the first look wasn't one.
3. When no official meaningfully decided, the legal duty to give a reasoned explanation has no author to draw the reasons from. Due process assumes a decision-maker whose actual reasoning can be stated and tested. If the reasons are reconstructed after the fact from a model's output — or generated to fit a determination nobody examined — the explanation is a plausible rationalization, not the basis on which the citizen was actually denied. The obligation persists; the thing it was designed to surface may no longer exist.

Where it breaks

Agencies redefine the caseworker's job away from deciding and toward clearing a queue the model now feeds (INVALID #1), and keep "human review" as the accountability answer (INVALID #2) — while due process still requires a responsible human the citizen can appeal to (STILL HOLDS #2) and there is no longer an official who meaningfully decided for that reviewer to be (NEW #2, NEW #3). The appeal right formally survives, but it now routes to a rubber-stamp; the citizen is told they can contest the decision to a human, and the human they reach never made it.

A second collision: the legal duty to explain a denial (STILL HOLDS #1) meets a determination no official examined (NEW #3). The system can always produce a reasoned-sounding explanation on demand — that's the cheap part now — so the explanation requirement gets satisfied on paper by generated text, precisely when it's least connected to why the citizen was actually denied. Whether that counts as due process is a question the deployment answers by default, before anyone asks it.

ai-axioms.vercel.app/axioms/government/government-accountability-ai-axioms