Validation Strategies and Data Leakage › Disadvantages

Holding out complete entities can leave few independent units for fitting and evaluation. For example, five equal group folds over 100 customers train on 80 customers at a time. Cross-validation then assesses fits trained on fewer customers than a final fit using all 100. If additional training data improve that learning procedure, this can make CV pessimistic for the final fit; neither improvement nor the bias direction is universal.

Grouped splits require identifiers that represent the entity boundary of the task. A customer should retain the same identifier across relevant tables and visits. Inconsistent identifiers or joins can hide overlap, while assigning one identifier to different customers can create false overlap. Temporal evaluation also needs event times, feature availability times, and, when relevant, the time at which each target becomes known.

With few or heterogeneous groups, results can depend strongly on which groups are held out. Increasing the fold count changes the training sizes and evaluation partitions, but does not create new customers. Report the limited evidence, compare models on common partitions, and use uncertainty methods whose sampling assumptions fit the groups and time structure. Resampling a handful of observed groups cannot supply diversity that was never collected.