RNN Architecture › Training Loss and Convergence

A character-level RNN trained on a short corpus learns letter-transition probabilities, common bigrams and trigrams, and some word-level structure, all from the sequential character stream. The hidden state is doing the heavy lifting of remembering which characters have appeared and in what pattern. The steepest descent in loss typically occurs in the first few epochs as the model moves from near-uniform predictions to learning the most frequent character transitions. Later epochs fine-tune the model's handling of less frequent patterns.

It is worth pausing on what the cross-entropy loss value means in practical terms. A random predictor over a vocabulary of size $V$ has expected cross-entropy $\log_2 V$ bits per token. For our character vocabulary of about 30 characters, that is roughly $\log_2 30 \approx 4.9$ bits. If our trained model achieves a loss of around 2.0 nats ($\approx 2.9$ bits), it has learned meaningful structure that compresses the character stream significantly below the random baseline.

The concept of perplexity converts cross-entropy to a more interpretable scale. Perplexity is $\text{PPL} = e^{\mathcal{L}}$ (in nats) or $2^{\mathcal{L}}$ (in bits). A model with perplexity 10 is, on average, as uncertain between 10 equally likely options at each step. Lower perplexity means the model is more confident and more accurate. For reference, well-tuned word-level language models on large corpora achieve perplexities in the range of 20 to 100, while simple baselines start near the vocabulary size.