Qwen Architecture › Architecture Overview
Language AI HandbookAll chapters Qwen Architecture › Architecture OverviewOne section · Sections in this chapter3 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Before examining specific components, let's place Qwen among modern LLMs and identify its architectural differences.
Foundation: Decoder-Only Transformer
Qwen adopts the decoder-only Transformer architecture, following the successful pattern established by GPT and refined by LLaMA. Like its predecessors, Qwen uses:
- Causal (left-to-right) attention masking for autoregressive generation
- Single unified architecture for all tasks
- Input and output sharing the same embedding space
Key Architectural Differences
Qwen combines proven innovations from LLaMA with additional refinements:
| Component | LLaMA | Qwen | Rationale |
|---|---|---|---|
| Normalization | RMSNorm (pre-norm) | RMSNorm (pre-norm) | Training stability |
| Activation | SwiGLU | SwiGLU | Model quality |
| Positional Encoding | RoPE | RoPE | Length extrapolation |
| Attention | Multi-head (7B) / GQA (70B) | GQA (all sizes) | Memory efficiency |
| Attention Bias | No bias | QKV bias | Improved attention quality |
| Vocabulary Size | 32,000 | 151,936 | Multilingual optimization |
| Embedding Tying | Tied | Untied | Better output distribution |
The most specific differences are the addition of bias terms in attention projections, the much larger vocabulary optimized for Chinese, and the use of Grouped-Query Attention across all model sizes.