Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Validation Strategies and Data Leakage › Visualizing Data Leakage

Michael BrenndoerferOctober 25, 20266 min read

Machine Learning from ScratchAll chapters

Part 4 of 15Model Evaluation and Selection

Chapter 31Validation Strategies and Data Leakage › Visualizing Data Leakage

Reading section 5 of 10 · Visualizing Data Leakage · 6 minSection 5 of 10 · 6 min

Section 5 of 10 · 6 min read

Validation Strategies and Data Leakage › Visualizing Data LeakageSection view · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

We first inspect row and group allocations, then examine a simulation where the numerical effect of customer overlap is small and nonmonotone. Together they separate a structural check from a score-based diagnostic.

The allocation diagrams contain six customers with four observations each. The row allocation alternates training and held-out observations within each customer; it is a deterministic illustration of overlap, rather than a random draw. The group allocation holds out customers E and F together.

PythonIn [3]
CollapseExpandrng_part = np.random.default_rng(11)
n_cust = 6
rows_per = 4
cust_value = rng_part.uniform(20, 80, n_cust)
cust_labels = [f"Customer {chr(65 + k)}" for k in range(n_cust)]
row_cust = np.repeat(np.arange(n_cust), rows_per)
row_val = cust_value[row_cust] + rng_part.normal(0, 2.5, n_cust * rows_per)
row_pos = np.arange(n_cust * rows_per)
row_is_train_random = (row_pos % rows_per) % 2 == 0
row_is_train_group = row_cust < 4
customer_tick_positions = [row_pos[row_cust == k].mean() for k in range(n_cust)]
allocation_y_limits = (row_val.min() - 5, row_val.max() + 5)
allocation_panels = [
    (row_is_train_random, "Row allocation"),
    (row_is_train_group, "Group allocation"),
]

VisualizationsOut [4]
CollapseExpand

Six customers A through F each have four observations, alternating filled training and open held-out markers within each customer. — Deterministic row allocation: alternating observations from every customer are assigned to training and hold-out. Filled markers are training rows and open markers are held-out rows. This diagram shows entity overlap; it does not fit a predictor or establish a score.

The four observations of customers A through D are filled training markers; all observations of E and F are open held-out markers. — Whole-customer allocation: A through D are training customers and E and F are held out together. Feature ranges can still overlap across customers; the identifier boundary, rather than feature separation, defines this split.

The diagrams show which customers occur on each side of the split. Their feature ranges can overlap even when the customer identifiers do not. No predictor is fitted in these diagrams, so they do not establish a numerical prediction error. The hand-worked example below supplies an explicit predictor and targets.

The next simulation varies how many held-out customers also have training observations. It uses a scalar feature correlated with the customer baseline, so observations from other customers can still be informative. The function also redraws the held-out customers and changes the number of held-out rows at different overlap settings; this is a descriptive simulation, not a controlled estimate of an overlap effect.

VisualizationOut [6]
CollapseExpand

RMSE falls from about 3.47 at zero overlap to 3.16 at 40 percent overlap, then rises to about 3.67 at full overlap. — Test RMSE of a 1-NN pipeline as the fraction of test customers whose other rows also appear in training increases from 0 (group split) to 1 (row split). The curve is not monotone here: the scalar feature predicts customer baselines across groups, and the simulation also changes the selected customers and test-set size, so the endpoint gap is not an isolated causal effect of leakage.

The computed curve is not monotone: RMSE ranges from about 3.16 to 3.73 in this run and finishes above its zero-overlap value. Here the scalar feature is itself a noisy measurement of the customer baseline, so nearby observations from other customers can also predict the target. In addition, the function changes the selected customers and the number of test rows as overlap changes. This is a counterexample to treating overlap as a universal numerical penalty: the curve does not isolate a causal effect, and a small score gap does not establish that an evaluation matches deployment. For an unseen-customer task, group disjointness remains a design requirement regardless of this gap.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Comments

Add to the conversation

Stay up to date