Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Validation Strategies and Data Leakage › Example

Michael BrenndoerferOctober 25, 20266 min read

Machine Learning from ScratchAll chapters

Part 4 of 15Model Evaluation and Selection

Chapter 31Validation Strategies and Data Leakage › Example

Reading section 6 of 10 · Example · 6 minSection 6 of 10 · 6 min

Section 6 of 10 · 6 min read

Validation Strategies and Data Leakage › ExampleSection view · Sections in this chapter6 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

We now replay the same logic on a dataset small enough that every number can be verified by hand, which separates the mechanism from any library behavior. We use four customers, each contributing two rows. This is a smaller version of the setup in the first figure.

The four customers and their observations are:

Customer A: $(x, y) = (10, 5)$ and $(11, 6)$
Customer B: $(x, y) = (50, 25)$ and $(51, 26)$
Customer C: $(x, y) = (20, 10)$ and $(21, 11)$
Customer D: $(x, y) = (80, 40)$ and $(81, 41)$

For each customer, the second observation is one unit higher in feature space and one unit higher in the target, so a model that assigns each row its same-customer sibling will predict it almost exactly.

Out[8]:

	customer	row	x	y
0	A	1	10	5
1	A	2	11	6
2	B	1	50	25
3	B	2	51	26
4	C	1	20	10
5	C	2	21	11
6	D	1	80	40
7	D	2	81	41

Worked example dataset with four customers and two observations each.
customer	row	x	y
A	1	10	5
A	2	11	6
B	1	50	25
B	2	51	26
C	1	20	10
C	2	21	11
D	1	80	40
D	2	81	41

We use a 1-NN regressor on a single scalar feature $x$ . The prediction for a test observation is the target of the training observation with the smallest $|x_\text{train} - x_\text{test}|$ . For our data this rule is easy to evaluate by hand.

The row splitLink Copied

The leaking row split assigns the first row of every customer to training and the second to test. This deterministic allocation guarantees a training sibling for every test row. It illustrates overlap that a random row split can produce; a random split can also place both of a customer's rows in the test set.

Training set: $\{(10, 5), (50, 25), (20, 10), (80, 40)\}$ , one row per customer.

Test set: $\{(11, 6), (51, 26), (21, 11), (81, 41)\}$ , the sibling row of each customer.

For each test row, we find the nearest training $x$ and read off its $y$ .

Test row $(11, 6)$ : distances $|11-10| = 1$ , $|11-50| = 39$ , $|11-20| = 9$ , $|11-80| = 69$ . Nearest is $x=10$ with $y=5$ . Prediction $= 5$ . Squared error $= (5-6)^2 = 1$ .
Test row $(51, 26)$ : distances $|51-10| = 41$ , $|51-50| = 1$ , $|51-20| = 31$ , $|51-80| = 29$ . Nearest is $x=50$ with $y=25$ . Prediction $= 25$ . Squared error $= (25-26)^2 = 1$ .
Test row $(21, 11)$ : distances $|21-10| = 11$ , $|21-50| = 29$ , $|21-20| = 1$ , $|21-80| = 59$ . Nearest is $x=20$ with $y=10$ . Prediction $= 10$ . Squared error $= (10-11)^2 = 1$ .
Test row $(81, 41)$ : distances $|81-10| = 71$ , $|81-50| = 31$ , $|81-20| = 61$ , $|81-80| = 1$ . Nearest is $x=80$ with $y=40$ . Prediction $= 40$ . Squared error $= (40-41)^2 = 1$ .

Mean squared error $= \frac{1}{4}(1 + 1 + 1 + 1) = 1$ . RMSE $= \sqrt{1} = 1.000$ .

The group splitLink Copied

Now we assign whole customers to one side, enforcing the group-disjointness condition. We place customers A and B in training and customers C and D in test. All four test rows belong to unseen customers.

Training set: $\{(10, 5), (11, 6), (50, 25), (51, 26)\}$ .

Test set: $\{(20, 10), (21, 11), (80, 40), (81, 41)\}$ .

Test row $(20, 10)$ : distances $|20-10| = 10$ , $|20-11| = 9$ , $|20-50| = 30$ , $|20-51| = 31$ . Nearest is $x=11$ with $y=6$ . Prediction $= 6$ . Squared error $= (6-10)^2 = 16$ .
Test row $(21, 11)$ : distances $|21-10| = 11$ , $|21-11| = 10$ , $|21-50| = 29$ , $|21-51| = 30$ . Nearest is $x=11$ with $y=6$ . Prediction $= 6$ . Squared error $= (6-11)^2 = 25$ .
Test row $(80, 40)$ : distances $|80-10| = 70$ , $|80-11| = 69$ , $|80-50| = 30$ , $|80-51| = 29$ . Nearest is $x=51$ with $y=26$ . Prediction $= 26$ . Squared error $= (26-40)^2 = 196$ .
Test row $(81, 41)$ : distances $|81-10| = 71$ , $|81-11| = 70$ , $|81-50| = 31$ , $|81-51| = 30$ . Nearest is $x=51$ with $y=26$ . Prediction $= 26$ . Squared error $= (26-41)^2 = 225$ .

Mean squared error $= \frac{1}{4}(16 + 25 + 196 + 225) = \frac{462}{4} = 115.5$ . RMSE $= \sqrt{115.5} \approx 10.747$ .

Reading the two estimates side by sideLink Copied

The row-split RMSE is 1.000 and the group-split RMSE is about 10.747, a difference of about 9.747 target units. The row split tests siblings of training observations; the group split tests different customers on a different set of target values. This illustrates how task and partition choices can change the error. It does not estimate a universal leakage penalty or imply that every learner would make the same errors.

PythonIn [9]
CollapseExpanddef one_nn_predict(train_df, x_test_values):
    preds = []
    for x_val in x_test_values:
        distances = np.abs(train_df["x"].values - x_val)
        nearest = train_df.iloc[np.argmin(distances)]
        preds.append(nearest["y"])
    return np.array(preds)


row_train = example_df[example_df["row"] == 1].reset_index(drop=True)
row_test = example_df[example_df["row"] == 2].reset_index(drop=True)
row_preds = one_nn_predict(row_train, row_test["x"].values)
row_rmse = np.sqrt(np.mean((row_preds - row_test["y"].values) ** 2))

group_train = example_df[example_df["customer"].isin(["A", "B"])].reset_index(
    drop=True
)
group_test = example_df[example_df["customer"].isin(["C", "D"])].reset_index(
    drop=True
)
group_preds = one_nn_predict(group_train, group_test["x"].values)
group_rmse = np.sqrt(np.mean((group_preds - group_test["y"].values) ** 2))

ConsoleOut [10]
CollapseExpandRow split RMSE:   1.000
Group split RMSE: 10.747
Score difference: 9.747

In this tiny row allocation, every nearest training observation is the held-out row's same-customer sibling. In the grouped allocation, C and D have no training siblings, and the stated 1-NN rule selects rows from A or B. These are properties of this constructed dataset, not guarantees for all repeated-customer data.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

The row splitLink Copied

The group splitLink Copied

Reading the two estimates side by sideLink Copied

Comments

Add to the conversation

Stay up to date