Interactive Explainer

Anatomy of a Modern Data Pipeline — and Where Human Judgment Still Wins

AI assistants can now scaffold ingestion, write SQL transforms, and wire orchestration in minutes. So what is left for the engineer? Quite a lot — just not where it used to be. Click through the pipeline below.

"Building data pipelines is becoming the easiest part of data engineering... So if SQL and Python are becoming commoditized, how do data engineers stand out? A few squishy skills have become even more valuable."— @EcZachly, on X

The Six Stages

Each box shows how much of the stage is realistically automatable by prompt-driven tools today. Click or tab into a stage to see what AI handles versus where a human still earns their salary.

The Commoditization Slider

Drag AI capability forward and watch which parts of the pipeline get eaten first — and which parts barely move, because they depend on context, trust, and taste rather than syntax.

Squishy Skills Self-Assessment

Rate yourself honestly on the five skills that compound as code generation gets cheap. Then copy your result and share it.

Why the Squishy Stuff Wins

A pipeline is a promise, not a script. The code answers "how do rows move." A human answers "which rows matter, to whom, at what cost, and what happens when the promise breaks at 3 a.m." Prompt-generated SQL cannot negotiate an SLA with a skeptical finance team, notice that a metric definition quietly changed meaning after a product launch, or decide that a $40k/month warehouse bill is buying nothing. Requirements gathering, data modeling judgment, cost accountability, incident communication, and stakeholder trust are the durable moat — precisely because they are hard to specify in a prompt.

Enjoy this tool? Build your own with Super