Open LLM Wave (2023) › Training Data and Curation
History of Language AIAll chapters Open LLM Wave (2023) › Training Data and CurationOne section · Sections in this chapter5 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
A critical aspect of the open model wave was the emphasis on training data quality and curation. Organizations releasing open models invested significant effort in collecting, filtering, and composing training datasets that would enable strong performance despite smaller model sizes or more limited computational budgets.
The Falcon models' RefinedWeb dataset demonstrated the importance of data quality. Rather than simply scraping the web indiscriminately, the Falcon team implemented sophisticated filtering and deduplication procedures that produced a high-quality dataset from trillions of tokens of raw web text. This careful curation enabled the Falcon models to achieve strong performance despite having fewer parameters than many proprietary systems trained on larger but less carefully filtered datasets.
The success of Falcon models highlighted one insight: smaller models trained on carefully curated, high-quality data could match or exceed the performance of larger models trained on less filtered datasets. This principle would influence subsequent model development, shifting focus from simply scaling dataset size toward better curation and greater data diversity.
MPT models emphasized the importance of diverse training data composition. In addition to web text, the MPT training datasets included substantial portions of code, mathematical content, and other specialized text types. This diversity helped the models develop capabilities across a wide range of domains, from general language understanding to code generation and mathematical reasoning.
The open nature of these models also enabled transparency about training data composition. Unlike proprietary models where training data remained confidential, open model developers often released information about their data sources, filtering procedures, and composition strategies. This transparency helped researchers understand the relationship between training data characteristics and model capabilities, informing future data curation efforts.
Data curation techniques developed for open models also advanced the field more broadly. Procedures for deduplication, quality filtering, toxicity detection, and content balancing that were refined for open model training found applications in proprietary model development and in downstream fine-tuning datasets. The open model wave contributed new models and improved methodologies for training data preparation.