Qwen Architecture › Model Variants and Family

Michael BrenndoerferUpdated August 7, 20253 min read
Language AI HandbookAll chapters
Part 29 of 67Modern Decoder Models
Chapter 212Qwen Architecture › Model Variants and Family
Reading section 8 of 11 · Model Variants and Family · 3 minSection 8 of 11 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 212 of 513

Section 8 of 11 · 3 min read

Qwen Architecture › Model Variants and FamilyOne section · Sections in this chapter8 / 11 ↓
Sections in this chapter11
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Qwen Model Family

Out[34]:
Visualization
Hierarchical diagram showing Qwen model variants branching from base models to specialized versions.
Qwen model family tree showing base models, chat variants, and specialized versions. Each base model has corresponding instruction-tuned (Chat) and quantized variants.

The Qwen family has expanded materially since its initial release:

Base models (various sizes):

  • Qwen-1.8B, Qwen-7B, Qwen-14B, Qwen-72B
  • Pre-trained on diverse multilingual data
  • Strong foundation for fine-tuning

Chat variants:

  • Instruction-tuned using SFT and RLHF
  • Optimized for conversational use cases
  • Safety-aligned for deployment

Specialized variants:

  • Qwen-VL: Vision-language model for image understanding
  • Qwen-Audio: Audio understanding and generation
  • CodeQwen: Specialized for code generation and understanding
  • Qwen-Math: Enhanced mathematical reasoning

Qwen 2 Improvements

Qwen 2, released in 2024, introduced several architectural refinements:

In[35]:
Code
# Qwen 2 architectural improvements
qwen2_improvements = {
    "architecture": {
        "gqa_all_sizes": "GQA used across all model sizes",
        "yarn_rope": "YaRN RoPE for better length extrapolation",
        "sliding_window": "Optional sliding window attention",
    },
    "training": {
        "more_data": "Training on 7T+ tokens",
        "better_filtering": "Improved data quality filtering",
        "longer_context": "Native 32K context in some variants",
    },
    "efficiency": {
        "flash_attention": "Native FlashAttention-2 support",
        "quantization": "Optimized for INT4/INT8 inference",
    },
}
Out[36]:
Console
Qwen 2 Improvements over Qwen 1
=======================================================

Architecture:
  • GQA used across all model sizes
  • YaRN RoPE for better length extrapolation
  • Optional sliding window attention

Training:
  • Training on 7T+ tokens
  • Improved data quality filtering
  • Native 32K context in some variants

Efficiency:
  • Native FlashAttention-2 support
  • Optimized for INT4/INT8 inference