Reproducible Python Pipeline Lineage & Caching Workbench

Pipeline DAG Dependency Graph (Click node to mutate)

Preset Workflow:

CACHED (Hash hit – 0s compute)

DIRTY (Invalidated downstream – recomputes)

EXECUTING (Live runner simulation)

Status: Idle / Ready

Compute Economics & Invalidation Ripple LINEAGE CHAIN VERIFIED

Total Time Saved 6.0s

Recompute Cost 49.4s

Compute Ratio Saved 10.8%

Invalidated Nodes 3 / 5

Preserved Cache 2 / 5

Node / Task	Content Hash (SHA)	Parent Digests	State	Duration

In traditional data science, caching is often bolted on as an afterthought using arbitrary file paths or timestamps, leading to cache invalidation bugs or silent training on stale data.

By enforcing cryptographic provenance, every task artifact is addressed by:

ContentHash = SHA256(UpstreamArtifactHashes + CodeBytecodeHash + Hyperparameters)

When you alter hyperparameter n_clusters: 5 → 8 in feature_pipeline, its hash changes. The downstream train_xgboost and eval_metrics depend on this hash, so their keys change automatically. Upstream nodes (raw_telecom_data, clean_transform) remain untouched and hit cache in 0.00 seconds.

Task Mutation Drawer

Select any node or edit the fields below. Changing parameters or code will immediately trigger a downstream cryptographic invalidation wave.

Active Task Node

Content Hash (SHA-256 Prefix)

3d71fa9e Copy

Primary Hyperparameter / Key

Hyperparameter Value Default fixture: was 5 → now 8 (triggers invalidation)

Function Code Fingerprint

Node Metadata

Compute: 6.5s Type: feature Status: DIRTY

Automatic Lineage Guarantee

Notice how Raw Telecom Data and Clean Transform never recompute. In an ML system with 4-hour raw extracts, you just avoided 4 hours of redundant processing with zero manual configuration.

Export Self-Contained Reproducible Pipeline