Held-out test sets: why the split matters more than the score

A vendor's score for a language system is only as good as the data behind it. A held-out test set is data kept out of training and out of every tuning decision, so that a score on it estimates performance on data the system has not seen. A number measured on data that leaked into training, or that was used to choose the model, no longer measures what its reader assumes, which is why the split matters more than the score.

Training, validation, and test

The scikit-learn documentation states the basic rule. Evaluating a model on the data it learned from is a methodological mistake, because a model that memorized its training labels would score perfectly and still fail on new data. The remedy is to hold out part of the data as a test set.

The documentation then names a second problem. If settings such as hyperparameters are adjusted until the model performs well on the test set, knowledge of the test set can leak into the model, and the metric stops reflecting generalization. A separate validation set absorbs the tuning, and the final evaluation runs on the test set. The same page says preprocessing steps such as scaling should be learned from the training portion only and then applied to held-out data.

The GPT-3 paper (Brown et al., 2020) uses the same division with the term development set. It tried a few values of K, the number of conditioning examples, on the development set and ran the best value on the test set. It reports final results on the test set when that set is publicly available. Where the test set is private, it reports development-set results instead, and it submitted to a test server for only a small number of datasets. A development-set score and a test-set score are therefore different claims, and a report has to say which one it gives.

Leakage is broader than reuse

Kapoor and Narayanan surveyed machine-learning-based science and found 17 fields where leakage errors had been identified, collectively affecting 329 papers. Their taxonomy has eight types:

• No test set
• Pre-processing on training and test set
• Feature selection on training and test set
• Duplicates in datasets
• Model uses features that are not legitimate
• Temporal leakage
• Nonindependence between train and test samples
• Sampling bias in test distribution

Their reproduction of civil war prediction studies shows the stakes. Papers claiming that complex models beat older methods failed to reproduce because of leakage, and once the leakage was corrected the complex models did not perform substantively better than decades-old logistic regression models. The authors propose model info sheets, reporting documents that record the details needed to detect leakage.

Contamination in language-model evaluation

Large pre-training corpora add another route to leakage. A survey by Xu and colleagues defines benchmark data contamination as exposure of a model to benchmark data during training, which distorts evaluation results, and names the composition of large pre-training datasets as one of the primary sources. Detection by n-gram overlap has limits: GPT-3 used a 13-gram overlap as the indicator, and the survey notes that this approach can miss subtle differences and that simple variations such as paraphrasing can bypass it.

The GPT-3 paper documents its own case. A bug left some overlaps with benchmark data in the training set, and the cost of training made retraining unfeasible. The authors instead compared performance on the full benchmark with performance on a clean subset, meaning the examples without the flagged overlap. Two results from that analysis, both for GPT-3:

• PIQA: 29 percent of examples were flagged, and performance on the clean subset was 3 percentage points lower, a 4 percent relative decrease.
• Winograd: 45 percent of examples were flagged, and performance on the clean subset was 2.6 percent lower.

The authors marked both results with asterisks. Their overall reading was that either their conservative method substantially overestimated contamination or contamination has little effect on performance, even though a quarter of the benchmarks scored over 50 percent potential contamination.

What reusing a test set costs

Repeated checking against the same test set is the worry behind the scikit-learn warning. Two studies measured it, in settings other than language models.

Recht and colleagues built new test sets for CIFAR-10 and ImageNet, two image-classification datasets. Accuracy dropped by 3 percent to 15 percent on CIFAR-10 and 11 percent to 14 percent on ImageNet. Their results suggest that the drops were not caused by adaptivity but by the models' inability to generalize to slightly harder images than those in the original test sets.

Roelofs and colleagues analyzed more than 100 Kaggle competitions, in each of which participants repeatedly checked a holdout set that produced a public leaderboard while a separate test set, scored once, set the final ranking. They found little evidence of substantial overfitting.

Together they show two separate things: neither found substantial overfitting from repeated reuse, and a new test set still gave lower scores in the image-classification case.

What to ask

• Which split produced the number, development or test?
• Did any choice, such as hyperparameters, prompt examples, or model selection, use the test set?
• How was overlap between the benchmark and the training data checked, and what threshold was used?
• Where overlap was found, what is the gap between the full benchmark and the clean subset?
• Does the report document the details needed to detect leakage, as model info sheets propose?

Azcuna's ELLMENT, the E-2D Large Language Model Entity, was proposed to NAVAIR PMA-231 under Navy SBIR Phase I topic DON26BZ01-NV010. The prototype, built at private expense on publicly available doctrine, answers with a page-level citation and refuses when the documents do not support an answer. The proposal has been submitted.

Sources

1. 1Cross-validation: evaluating estimator performance · scikit-learn project documentation
2. 2Language Models are Few-Shot Learners · arXiv (Brown et al.)
3. 3Leakage and the Reproducibility Crisis in ML-based Science · arXiv (Kapoor and Narayanan)
4. 4Benchmark Data Contamination of Large Language Models: A Survey · arXiv (Xu et al.)
5. 5Do ImageNet Classifiers Generalize to ImageNet? · arXiv (Recht et al.)
6. 6A Meta-Analysis of Overfitting in Machine Learning · Advances in Neural Information Processing Systems 32 (NeurIPS 2019)