What changes for data engineering with AI?

The shift

Writing and translating the code that moves, models, and documents data — pipelines, schema mappings, SQL, lineage explanations — goes from scarce, specialist-hours to abundant and near-instant. A model with long-context access to a warehouse's schemas, query logs, and ticket history can now draft transformations, infer table relationships, and explain what a pipeline does, not just autocomplete a query.

The axioms

• Writing ETL/ELT pipelines and transformation code is slow, specialist work, so headcount and timelines are sized to how much pipeline code a team can hand-write. Rests on: drafting integration code is scarce.
• Documentation of what a table, column, or pipeline actually means lags or doesn't exist, because writing and maintaining it loses to shipping the next pipeline. Rests on: documentation is scarce effort competing with delivery.
• Onboarding a new engineer to the data model takes months, because the "why this table looks like this" knowledge lives in a few people's heads. Rests on: institutional knowledge is scarce and undocumented.
• Debugging a wrong number in a dashboard or a broken pipeline requires manually tracing a complex DAG by hand — lineage archaeology. Rests on: tracing causality across scattered systems is scarce, slow expert work.
• Migrating between warehouses, schemas, or vendors is a large, risky, manual effort because rewriting queries and pipelines by hand is expensive. Rests on: translation between formats and systems is scarce labor.
• The person who wrote the SQL or pipeline is the one who understands the business logic buried inside it. Rests on: synthesizing business rules into code is scarce.
• Data quality problems (nulls, duplicates, schema drift, broken joins) are caught late because writing exhaustive checks upfront is expensive and gets deprioritized. Rests on: anticipatory verification code is scarce effort.
• Schema and model design requires deep upfront human judgment about how the data will be queried and by whom. Rests on: anticipating future usage patterns is a scarce judgment call.
• Someone is accountable when a bad number reaches a board deck, a dashboard, or a model trained on corrupted data. Rests on: accountability requires a liable, employable human.
• Deciding what data is worth collecting, retaining, and building infrastructure around is a cost and business-value call, not a technical one. Rests on: taste and prioritization about business value is scarce.
• Governance, access control, and compliance (PII, GDPR, retention policy) require a human who understands legal and business context to set the policy. Rests on: judgment on regulatory and trust tradeoffs is scarce.

Invalid axioms

1. Headcount and timelines are sized to how much pipeline code a team can hand-write. Drafting transformation logic, connectors, and boilerplate DAGs is now something a model does in minutes from a source schema and a target spec. The habit-trap: teams still scope data-eng capacity by pipeline count and staff-weeks-per-source, when the write step has stopped being the bottleneck.
2. Documentation of tables, columns, and pipelines lags because writing it is scarce effort that loses to shipping. A model can read the pipeline code, the query history, and the commit log and generate a first-pass description of what a table is, who touches it, and what breaks downstream. The habit-trap: teams still carry "someday" tickets for data-dictionary and catalog debt as if writing the first draft were the expensive part.
3. Onboarding to a legacy data model takes months of tribal-knowledge transfer. Most of that transfer is answerable from artifacts a model already has long-context access to — DDL, historical PRs, Slack threads, ticket history. The habit-trap: onboarding plans are still scheduled as a multi-month shadowing exercise instead of a retrieval-and-verification problem.
4. Tracing a wrong number back through a tangled DAG is slow, manual lineage archaeology. A model that can read pipeline definitions, transformation code, and column-level lineage metadata can trace and explain a chain of transformations in the time it takes to ask. The habit-trap: incident response still budgets hours or days for "find where this number went wrong" as if the trace itself were the hard part, rather than confirming the trace is right.
5. Migrating between warehouses, schemas, or vendors is a large manual rewrite effort. Translating SQL dialects, remapping schemas, and rewriting connector code is exactly the pattern-matchable translation work that's now cheap and fast to draft. The habit-trap: migration projects are still scoped and quoted as multi-quarter, headcount-heavy efforts sized to hand-translation speed.

Unchanged axioms

1. Someone is accountable when a bad number reaches a dashboard, a board deck, or a training set. A model can't be liable for a decision made on corrupted data — accountability still requires an identifiable human or team with the standing to be held responsible. Generation getting cheap doesn't relocate this.
2. Data quality and correctness require ground-truth verification, not confident-sounding output. A model can draft a transformation or a data-quality check just as plausibly whether the underlying logic is right or subtly wrong — join keys, timezone handling, deduplication logic, currency conversions are exactly the places "confidently wrong" bites hardest, because errors compound silently downstream into everything built on top of that table.
3. Schema and model design under real, evolving business ambiguity needs human judgment. Deciding how to model an entity that the business itself hasn't finished defining — what a "customer" or an "active user" means across three product lines — is a judgment call with no clean pattern to match, not a retrieval task.
4. Governance, access control, and regulatory compliance require a human who owns the policy tradeoff. Setting retention rules, PII handling, and access boundaries is a legal and trust decision with real consequences if wrong; a model can draft the policy language but can't be the accountable party when a regulator or a breach shows up.
5. Deciding what data is worth collecting and building pipelines for at all is a business-judgment call. Cheap pipeline-drafting doesn't tell you whether an event is worth instrumenting or a source is worth ingesting — that's a prioritization and cost call sitting upstream of any pipeline work.

New axioms

1. When pipelines and transformations are cheap to generate, the volume of pipeline code needing review and validation can outpace the team's capacity to check it. Review, historically sized to hand-written pipeline volume, now faces a firehose of AI-drafted DAGs, schema mappings, and queries — the open problem is building data-quality and lineage verification that scales with generation instead of becoming the next bottleneck.
2. When a model can draft a plausible transformation from an ambiguous spec, silently wrong business logic can ship at pipeline speed instead of code-review speed. A confidently wrong join or a misunderstood business rule now propagates into downstream tables and dashboards faster than teams are used to catching it, and nobody has agreed on who re-verifies generated transformation logic before it touches production data.
3. If onboarding and tribal-knowledge transfer shift to model-mediated retrieval, the pipeline that produces engineers who deeply understand the data model loses its training mechanism. It's an open problem how the next generation of data engineers builds the judgment that STILL HOLDS says remains scarce — modeling ambiguous entities, spotting quality issues — if the slow manual exposure that used to build that intuition is skipped.
4. Abundant pipeline and documentation generation increases the volume of data infrastructure faster than any one person's understanding of the whole system. More sources, more transformations, more auto-generated docs mean a data platform can grow past the point where any single human — or a single context window — holds a coherent model of how it all fits together, which is a different failure mode than debt from a slow, deliberate build.
5. Cheap migration and schema-translation raises the stakes of getting the translation subtly wrong at scale. Moving a warehouse or rewriting a SQL dialect at AI speed means an unnoticed edge-case mistranslation (a rounding rule, a null-handling difference, a timezone assumption) can replicate across an entire data estate before anyone checks it against ground truth — and how fast automated validation tooling closes that gap is still an open, fast-moving question.

Where it breaks

"Headcount and timelines scale with pipeline-writing throughput" (invalid) collides with "review and data-quality verification, not generation, is the new constraint" (new): teams that redirect data-eng capacity toward shipping more AI-drafted pipelines without resourcing verification are increasing the surface area for silently wrong numbers to reach dashboards and models, and won't notice until a metric that's been quietly broken for months gets challenged.

"Migrating and translating between systems is now cheap and fast" (invalid) collides with "a subtly wrong translation now replicates at scale before anyone checks it" (new): migration projects scoped for hand-rewrite speed but executed at AI speed compress the time available to catch an edge case, so the same mistake that used to surface one query at a time now surfaces across an entire warehouse at once.

ai-axioms.vercel.app/axioms/engineering/data-engineering-ai-axioms