What changes for work when AI safety, alignment, and red-teaming become core functions?

The shift

The thing being tested stops being deterministic software with a spec and becomes a probabilistic system whose behavior is generated fresh each run, has no fixed ground truth, and shifts every time the model or its context changes. Classic QA rested on scarce, expensive engineering time to write tests against a known-correct answer; AI makes generating candidate behaviors and adversarial inputs abundant, but makes the correct answer itself unavailable — so the scarce good moves from producing tests to judging open-ended output nobody can fully specify.

The axioms

• You can test software against a spec: there is a defined correct output, and the test asserts against it.
• Behavior is deterministic and reproducible: the same input yields the same output, so a passing test stays passing.
• Evaluation coverage can be made complete: the input space is enumerable enough that you can drive coverage toward "we tested the paths that matter."
• A fix stays fixed: once a bug is closed and regression-tested, it does not silently reopen.
• Verification is a bounded, one-time gate before ship, because the artifact under test doesn't change after release.
• Someone accountable decides what's safe enough to ship — a human owns the risk call and is answerable for it.
• Finding the failure the tests missed takes adversarial creativity, which is scarce and human.
• The adversary you're defending against is a fixed target: it doesn't rewrite itself in response to your defense within the same engagement.

Invalid axioms

1. You can test software against a spec with a defined correct output. For an open-ended model, there is no single correct output for "summarize this," "is this response harmful," or "did the agent behave." The scarcity QA rested on — a cheap oracle telling you pass or fail — is gone; the output is plausible-by-default and judging it is the expensive part. The habit-trap: teams still staff and budget AI evaluation like a test-writing task ("write more test cases") when the bottleneck is grading, not generating, and still report a green suite as if it meant the same thing it meant for deterministic code.
2. Behavior is deterministic and reproducible. Sampling, non-determinism in serving, and context sensitivity mean the same prompt can pass a thousand times and fail the thousand-and-first. The habit-trap: pass/fail gates, flaky-test triage, and "it worked in the last run" reasoning are imported wholesale from deterministic CI, so orgs treat a passing eval as a guarantee rather than as one draw from a distribution, and treat a single caught failure as fixed.
3. Evaluation coverage can be made complete. The input space of natural language plus tool-use plus multi-turn context is effectively unbounded; you cannot enumerate it, and a benchmark score is a sample, not a certificate. The habit-trap: coverage metrics, "we tested the top intents," and eval-suite pass rates get read as completeness, when they measure only the slice someone thought to write down.
4. A fix stays fixed. A safety patch to one behavior can regress another, and a model or prompt update can silently reopen a failure that was closed, because there's no localized code path that was changed and can be diffed. The habit-trap: regression testing assumes a fix is durable and localized; here every model or context change is a potential re-opening of everything, and release notes rarely reflect that.
5. Verification is a bounded, one-time gate before ship. The system's behavior drifts post-release as the model, its context sources, tools, and the adversaries all change — so a point-in-time sign-off certifies a system that no longer exists. The habit-trap: ship/no-ship review boards, one-time certifications, and "we red-teamed it before launch" are still structured as a gate you pass once, not a function you run continuously.

Unchanged axioms

1. A named human owns the risk call and is answerable for it. A model cannot be liable, appear in a filing, or be fired. Deciding what's "safe enough to ship" when the failure space is unbounded is a judgment under exactly the novel, high-stakes ambiguity where no pattern-match substitutes for an accountable person — and it gets harder, not easier, when the thing you're signing off on has no complete spec.
2. Adversarial creativity — finding the attack nobody wrote down — stays partly scarce. Models now generate large volumes of candidate jailbreaks and edge cases cheaply, and automated red-teaming is real and improving fast. But the genuinely novel attack class, the one outside the training distribution of known attacks, still tends to come from human ingenuity working against an incentive; abundance of variations is not the same as the new idea. (Fast-moving: automated red-teaming is closing this faster than most other calls here — the honest position by mid-2026 is that volume is solved and novelty is contested, not that novelty is safe.)
3. Responsibility for what "safe enough" means is a taste-and-values call, not a measurement. Where to set the threshold — what harm rate is acceptable, which trade-off against usefulness is right — is a decision about what's worth doing, not a fact an eval returns. Evals can inform it; they can't make it, and offloading it to a metric is itself the failure mode.
4. A confidently-wrong safety verdict is more dangerous than a slow correct one. The evaluator is now often itself a model (LLM-as-judge), which means the grader is as probabilistic and gameable as the system under test. Trusting that output requires a scarce, human-gated verification step — you cannot fully verify a probabilistic system with another probabilistic system and call it settled.

New axioms

1. Evaluating open-ended behavior with no ground-truth spec is an unsolved measurement problem. When "did it behave" has no reference answer, you're building graders — rubrics, LLM-judges, human panels — that are themselves imperfect and drift. Nobody has a general answer for how to trust a grader you can't ground, or how to know your eval is measuring the harm you care about rather than a proxy that games easily.
2. Red-teaming an adversary that adapts breaks the fixed-target assumption. The attacker (and increasingly an automated one) rewrites its approach in response to your defense inside the same engagement, so a defense that held yesterday tells you little about tomorrow. Testing has to become adversarial-in-the-loop and continuous, and almost no org has sized what that costs or who runs it.
3. Accountability for a harm the red team missed has no settled home. When an unbounded space guarantees you didn't test everything, "we followed the process" and "we red-teamed it" collide with a real-world harm that slipped through. Who is answerable — the safety team, the deployer, the model provider — for a failure that was statistically inevitable and specifically unforeseen is unresolved, and regulation is arriving faster than the answer.
4. Safety has to be continuous, not one-time, and the org isn't built for it. The function has to run against every model update, context change, and new attack indefinitely — a standing operational capability, not a pre-launch project. Budgets, headcount, and review structures modeled on one-time QA gates have no line item for "forever."
5. Scaling human oversight past the volume machines produce is unsolved. When behaviors, agent actions, and eval results are generated far faster than humans can review them, the choice between human-in-the-loop (approve each action, doesn't scale) and human-on-the-loop (supervise aggregate, misses the specific bad case) is a live trade-off with no clean answer, and the review capacity it implies has not been sized.
6. This is becoming a mandated function and a profession, not a team's side project. Regulation, procurement requirements, and model-provider policy are turning AI safety and red-teaming into a required, staffed, credentialed role — which forces questions of standards, certification, independence from the ship-it incentive, and what "qualified" even means for a discipline whose ground rules are still being written. (Fast-moving: the regulatory picture is shifting quarter to quarter; the direction is clear, the specifics are not.)

Where it breaks

Teams still run safety as a pre-launch gate — "we red-teamed it, it passed, ship" (invalid: verification is a one-time gate) — while the system's behavior, its adversaries, and the models underneath all keep changing after release (new: safety has to be continuous against an adapting target). The sign-off certifies a system that no longer exists by the time the harm lands, and the org has budgeted for the gate, not the standing function.

Green eval suites get read as safety guarantees (invalid: coverage can be complete, a fix stays fixed) at the same moment accountability is hardening into a named, possibly regulated owner for harms the evals provably couldn't cover (new: no settled home for the missed harm). The person signing off is being handed a number that structurally cannot mean what a passing test meant in deterministic software, and is the one answerable when the gap it hid shows up in production.

ai-axioms.vercel.app/axioms/engineering/ai-safety-redteaming-work-ai-axioms