Qwen Architecture › Training Methodology
Michael BrenndoerferUpdated August 7, 20253 min read
Language AI HandbookAll chapters Part 29 of 67Modern Decoder Models
Qwen Architecture › Training MethodologyOne section · Sections in this chapter6 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Training Data Composition
Qwen is trained on approximately 3 trillion tokens, materially more than LLaMA's 1-1.4 trillion:
Out[19]:
Visualization
Key data characteristics:
- Multilingual balance: Substantial Chinese content alongside English and other languages
- Code inclusion: Programming languages for coding capabilities
- Quality filtering: Extensive deduplication and quality scoring
- Domain diversity: Academic papers, books, web text, and specialized corpora
Training Configuration
In[20]:
Code# Qwen training hyperparameters
training_config = {
"optimizer": "AdamW",
"adam_beta1": 0.9,
"adam_beta2": 0.95,
"adam_eps": 1e-8,
"weight_decay": 0.1,
"gradient_clip": 1.0,
# Learning rate
"peak_lr": 3e-4,
"warmup_steps": 2000,
"lr_schedule": "cosine",
"min_lr_ratio": 0.1,
# Precision
"precision": "bfloat16",
# Context length
"context_length": 8192, # Qwen supports longer context
# Training tokens
"total_tokens": "3T",
}Out[21]:
ConsoleQwen Training Configuration ======================================================= Optimizer: AdamW β₁ = 0.9, β₂ = 0.95 Weight decay = 0.1 Gradient clipping = 1.0 Learning Rate Schedule: Peak LR = 0.0003 Warmup = 2,000 steps Schedule = cosine Training Scale: Context length = 8,192 tokens Total training tokens = 3T Precision = bfloat16
Qwen's training follows similar principles to LLaMA but with extended context length (8192 vs 2048) and more training tokens. The longer context is enabled through careful RoPE scaling and NTK-aware interpolation.
Context Length Extension
Qwen supports 8192 token context through training, with the ability to extend further using position interpolation techniques:
Out[22]:
Visualization