Machine Learning Decision & Leakage Auditor

Offline Training AUC

0.962 (inflated due to leakage)

Mirage created by future variables

Live Production AUC

0.714 (after leakage correction)

Actual real-world discriminative power

Optimal Threshold (τ*)
0.18
Maximizes net business expected value

Net Daily Business Value
$184,250
Daily benefit minus operational error costs

1. Task & Data Pipeline Architecture

Machine Learning Task

Data Splitting Strategy

Time-dependent problems must split chronologically to evaluate true out-of-time decay.

2. Feature Pipeline & Temporal Boundary

Toggle variables to audit temporal data leakage. Forbidden features look miraculous in backtests but fail in production.

logins_last_7d

Session count up to prediction point

Clean

support_tickets_30d

Customer complaints prior to decision

Clean

future_30d_revenue

Contains outcome occurring post-event

Include

account_tenure_months

Account age established at inference

Clean

3. Business Cost Matrix & Operational Scale

Daily Volume (Predictions / Day)

10,000

Base Positive / Churn Rate (π)

5.0%

Cost of False Positive ($FP - Customer Friction / Unneeded Discount)

$10

Cost of False Negative ($FN - Lost Customer / Missed Fraud)

$500

Current Decision Threshold (τ)

0.18

ROC & Leakage Visualizer Offline vs Live

Cost-Benefit Payoff Curve Expected Daily Value

4. Model School Matcher (Pedro Domingos' 5 Schools)

Match problem geometry to algorithm worldview rather than defaulting blindly to deep learning.

Symbolists

Rules, Decision Trees

Best for auditable workflows & hard boundaries.

Bayesians

Naive Bayes, PGMs

Belief updating under cold-start uncertainty.

Connectionists

Deep Neural Nets, CNNs

Unstructured perceptual images & language.

Evolutionaries

Genetic Alg., Auto-ML

Search non-differentiable ugly spaces.

Analogizers

SVMs, k-NN, Kernels

Geometric margins in high-dimensional text.

5. Production Machine Learning Operating Audit

Audit Dimension	Status	Engineering Recommendation
Temporal Data Leakage	Violated	Purge future_30d_revenue; restrict feature engineering strictly to historical t ≤ t_decision.
Evaluation Design	Sound	Chronological split faithfully models out-of-time distribution shifts.
Decision Threshold	Calibrated	Threshold set to 0.18 to minimize severe False Negative costs ($500).
Algorithm Archetype	Aligned	Symbolist / Ensemble (e.g. XGBoost / Logistic Baseline) fits structured tabular churn data.