What changes for QA and testing with AI?

The shift

Generating test cases, test scripts, and plausible edge-case scenarios goes from scarce (tester-hours, one case at a time) to abundant (thousands of candidate tests, seconds, near-free). What stays scarce is knowing which failures actually matter and whether a suite that passes proves the system is actually correct.

The axioms

• Writing test cases and test scripts is slow, so coverage is gated by tester-hours. Rests on drafting being expensive.
• Exploratory testing — probing a system the way a confused or adversarial user would — requires a human's judgment and intuition. Rests on judgment under ambiguity being scarce.
• A green test suite is evidence the software works. Rests on the suite being a faithful, trustworthy proxy for correctness.
• Someone must be accountable when a shipped defect causes harm. Rests on accountability requiring a liable human.
• Reproducing and diagnosing a bug report requires a skilled engineer to trace cause from symptom. Rests on diagnostic synthesis being expensive.
• Test maintenance (updating tests as the product changes) is a tax paid in engineering time. Rests on translation/update work being manual and slow.
• Manual regression testing across many device/OS/locale combinations doesn't scale, so it's sampled rather than exhaustive. Rests on execution breadth being bounded by human throughput.

Invalid axioms

1. Coverage is gated by tester-hours. Generating unit tests, property-based tests, fuzz inputs, and boilerplate integration tests is now near-free — an LLM can draft hundreds of cases from a spec or a diff in seconds. Habit-trap: teams still staff and budget test-writing as if each case costs an engineer's afternoon, and treat "we don't have time to write more tests" as a real constraint when it's now a generation-and-review problem, not a generation problem.
2. Test maintenance is a manual tax paid in engineering time. Updating assertions, fixing selectors, and rewriting tests after a refactor is exactly the kind of pattern-matched translation work AI does well. Habit-trap: sprints still carve out fixed "test debt" time as if someone has to hand-edit every broken test, when most of that rewriting can be delegated and just needs review.
3. Manual regression testing is sampled because exhaustive coverage doesn't scale. Agentic tools can now drive UI flows across device/OS/locale matrices at a scale no manual team could match. Habit-trap: teams keep a fixed, historically-sized "regression pass" scope instead of asking why they're still sampling when broader execution is cheap.
4. Bug triage requires a skilled engineer to manually trace symptom to cause. AI can synthesize logs, stack traces, and recent commits into plausible root-cause candidates almost instantly. Habit-trap: on-call rotations still price first-pass triage as senior-engineer time when it's increasingly a draft an engineer verifies.

Unchanged axioms

1. A green suite is evidence the software works. This now depends on who wrote the assertions and what they actually check. An AI-generated test that asserts the function returns something rather than the right something is worse than no test — it manufactures false confidence. Verifying that tests encode real intent, not just plausible-looking coverage, stays scarce and human.
2. Someone must be accountable when a shipped defect causes harm. A model that wrote or approved a test can't answer for a production incident. This doesn't get cheaper or more distributed just because test volume did — it stays pinned to a person or team with the authority and consequence exposure to own the call.
3. Judgment about what's worth testing under novel, high-stakes ambiguity stays scarce. Deciding which edge case matters for a payments system, a medical device, or a one-shot migration isn't a pattern-match problem — there's no corpus of "what breaks in this specific untested integration with this specific customer's data." That call still needs a human who understands the actual stakes.
4. Exploratory testing driven by an adversarial, curious human mindset stays valuable. AI can execute exploratory scripts, but deciding what's suspicious enough to poke at — the tester's nose for "this feels wrong" — is taste and judgment, not pattern-matching against known bug shapes.

New axioms

1. Who verifies the verifier, when tests are generated as fast as code. If both the implementation and its tests are AI-drafted, a bug and a matching wrong test can ship together looking coherent. The field has no settled practice yet for independently checking that generated tests encode real requirements rather than just mirroring the generated code.
2. Test-suite bloat becomes its own liability. When generating tests is free, teams accumulate thousands of shallow, redundant, or brittle tests that inflate CI time and give false confidence without raising real defect-catch rate. Curating and pruning a test suite is now harder than writing one.
3. Coverage metrics stop meaning what they used to mean. Line/branch coverage was a rough proxy for testing effort when tests were expensive to write; now it's trivially gameable by generating volume. The field needs a new proxy for "this suite would actually catch a regression," and doesn't have one yet.
4. Agentic testing tools can now take real actions (submit forms, hit staging APIs, mutate data) at volume. This creates a class of risk — flooding third-party sandboxes, tripping rate limits, corrupting shared test data — that didn't exist when test execution was manual and slow.

Where it breaks

Teams treat "more tests" as unambiguously good because coverage used to be scarce (INVALID #1) — but at generated volume, an unverified glut of shallow tests actively erodes the signal a green build is supposed to give (NEW #2, #3). The team that scales up test generation without also scaling up review of what those tests actually assert ends up with a slower CI pipeline and less real confidence than before.

Separately, bug triage getting delegated to AI-drafted root-cause analysis (INVALID #4) collides with accountability staying human (STILL HOLDS #2): if the on-call engineer starts rubber-stamping a plausible-sounding AI diagnosis under time pressure, the accountability hasn't moved even though the habit of trusting the draft has — and nobody's redesigned the sign-off step to catch that gap.

ai-axioms.vercel.app/axioms/engineering/qa-testing-ai-axioms