Who's accountable when an AI-flagged risk assessment misses a child- or elder-welfare warning sign?

The shift

Scoring and flagging risk from case data — pulling referral history, prior incidents, and known factors into a prioritized "this family needs attention now" signal — goes from scarce analyst and caseworker hours to abundant, instant, and cheap. The score is now the easy part; owning the life-safety call it feeds is not.

The axioms

• A licensed worker owns each life-safety decision and is answerable for it — scarce accountability, attached to a named person.
• Producing a risk score or triage ranking from a case's data took skilled human hours — scarce scoring capacity.
• Reading a home visit — the state of the house, a child's flinch, an elder's unexplained bruise, a caregiver's evasiveness — surfaces signals no record holds — scarce embodied judgment.
• A parent, child, or vulnerable adult discloses to a person they trust what they'll never enter into a system — scarce relationship.
• The duty of care is legal and moral, and it attaches to a human or an accountable organization, not to a tool — scarce liable actor.
• A worker can explain and defend the reasoning behind their own assessment when it's challenged — scarce, but the worker's own reasoning is inspectable to them.

Invalid axioms

1. Producing a risk score or triage ranking takes skilled human hours. Ranking a caseload by data-derived risk is pattern-matching against structured history — exactly what a model does instantly and at volume. Habit-trap: agencies still treat the score as the scarce, expert output and staff around generating it, when generating it is now free and the scarce work is deciding whether to trust it on the case in front of you.

Unchanged axioms

1. A licensed worker — or the accountable agency — owns the life-safety decision. A model can rank risk; it cannot be named in a serious case review, testify to why a warning sign was missed, or carry the legal and professional consequence of a child or elder harmed. When an AI-flagged assessment misses a sign, accountability lands where it always did: on the human who acted or failed to act, and the organization that deployed the tool and set the workflow. The model is not a defendant.
2. Reading a home visit surfaces signals no record contains. Whether a house is unsafe today, whether an elder's injury matches the caregiver's story, whether a child is frightened of the adult in the room — these are read in person, against contradictory and emotionally loaded cues, with no ground truth to check against. A risk model built on recorded data structurally cannot see what was never recorded, and much of what protects someone lives in exactly that gap.
3. The relationship is what surfaces the disclosure. A frightened child or a coerced elder tells a person who has earned their trust things that never reach a case file — and therefore never reach the model. This isn't slow because retrieval is slow; it's slow because trust is.
4. The duty of care attaches to a person or an accountable body, not a tool. Cheaper scoring doesn't create a new bearer of liability. The obligation to protect stays with the licensed worker and the agency, and no procurement contract moves it onto the vendor's model.

New axioms

1. Automation bias on the rare catastrophic miss. A risk model that is usually right trains workers to defer to it — so the dangerous failure is the one case the score rates low and the human, primed to trust it, doesn't look harder. The system is most trusted precisely where its miss is most fatal, and we have no reliable way to keep a usually-right score from suppressing the human second look on the exception. Fast-moving: as models get more reliable, this trust deepens and the trap gets worse, not better.
2. Liability for a miss by a model nobody in the room built or can interrogate. When an AI-flagged assessment misses a warning sign, the worker is held to account for a score they can't see inside, produced by a vendor's model trained on data they didn't choose. The duty of care still lands on the human (STILL HOLDS), but the reasoning behind the flag is opaque to the very person answerable for acting on it — a mismatch between where accountability sits and where the decision logic actually lives. Who is liable — worker, agency, or vendor — is unsettled in law and policy as of mid-2026 and moving as regulators catch up.
3. The worker defending a decision they can't reconstruct. Axiom (f) assumed a worker could explain their own reasoning under challenge. When the assessment is model-driven and the model is a black box, "why did you rate this family low-risk" has no answerable form beyond "the tool did" — which is not a defense a review will accept, and not a record that shows who actually judged what.
4. Signals only a human visit catches, now systematically deprioritized. If the score decides who gets seen, families whose risk lives in the unrecorded — the things a visit would surface but the data never captured — get ranked down and visited less, which means the signal never gets recorded, which keeps them ranked down. The tool routes human attention away from exactly the cases where human attention was the only thing that would have caught it.

Where it breaks

Agencies treat the risk score as the scarce expert product and build the workflow around trusting it to allocate scarce attention (INVALID: scoring is now cheap) — while the actual failure is automation bias suppressing the human second look on the one low-scored case that was catastrophic (NEW). The faster and more reliable the model gets, the more the workflow leans on it, and the more the rare miss is guaranteed to be the one nobody looked at twice.

Separately: the duty of care still lands on a named human and their agency (STILL HOLDS) — but the reasoning that drove the flagged assessment lives inside a vendor model that same human can't interrogate or reconstruct (NEW). We've kept accountability attached to the person while moving the decision logic somewhere they can't reach, so when a warning sign is missed, the one who answers for it is the one who could see it least.

ai-axioms.vercel.app/axioms/society/social-work-risk-accountability-ai-axioms