LSTM Architecture › Character-Level Language Model with LSTM
Language AI HandbookAll chapters LSTM Architecture › Character-Level Language Model with LSTMOne section · Sections in this chapter9 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Language modeling is among the most natural applications for LSTMs. The model reads characters one at a time and learns to predict the next character. Because individual characters carry minimal meaning on their own, the model must internalize word-level, phrase-level, and sentence-level patterns to make accurate predictions.
Character-level modeling was historically significant because it required no tokenization decisions and could handle any input text without preprocessing. The famous 2015 blog post by Andrej Karpathy, "The Unreasonable Effectiveness of Recurrent Neural Networks," demonstrated that LSTM character models trained on large corpora could generate surprisingly coherent text, structured code, and even plausible mathematical notation. This work helped establish LSTMs as the dominant sequence modeling architecture before the transformer era and inspired many researchers to explore what kinds of structure an LSTM could learn to represent internally.
Dataset Preparation
# Sample text corpus for demonstration
corpus = (
"To be, or not to be, that is the question: "
"Whether 'tis nobler in the mind to suffer "
"The slings and arrows of outrageous fortune, "
"Or to take arms against a sea of troubles "
"And by opposing end them."
)
# Build character-level vocabulary
chars = sorted(set(corpus))
char_to_idx = {c: i for i, c in enumerate(chars)}
idx_to_char = {i: c for c, i in char_to_idx.items()}
vocab_size = len(chars)
encoded = [char_to_idx[c] for c in corpus]Corpus length: 197 characters Vocabulary size: 30 unique characters First 30 chars: 'To be, or not to be, that is t'
Model Architecture
The character model uses a small embedding layer before the LSTM. Although character indices could be passed directly as one-hot vectors, an embedding layer allows the model to learn a dense representation for each character. In a well-trained model, characters with similar roles (all lowercase vowels, all punctuation marks) tend to cluster together in the embedding space, making the downstream LSTM's job easier.
class CharLSTM(nn.Module):
"""Character-level language model using LSTM."""
def __init__(self, vocab_size, embed_dim, hidden_size, num_layers=1):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTM(
input_size=embed_dim,
hidden_size=hidden_size,
num_layers=num_layers,
batch_first=True,
)
self.output_layer = nn.Linear(hidden_size, vocab_size)
def forward(self, x, hidden=None):
# x: (batch, seq_len) integer character indices
embeds = self.embedding(x) # (batch, seq_len, embed_dim)
lstm_out, hidden = self.lstm(embeds, hidden)
logits = self.output_layer(lstm_out) # (batch, seq_len, vocab_size)
return logits, hiddenNote that the LSTM returns the hidden state, which is passed back into the model during generation. This stateful generation pattern, where the hidden state is carried from one call to the next, is what enables the model to generate coherent text: each new character prediction takes into account the most recent character and the full history encoded in the recurrent state.
Preparing Training Batches
def make_char_batches(encoded, seq_len=25):
"""Create (input, target) pairs where target is input shifted by 1 position."""
inputs, targets = [], []
for i in range(0, len(encoded) - seq_len - 1, seq_len):
inputs.append(encoded[i : i + seq_len])
targets.append(encoded[i + 1 : i + seq_len + 1])
return torch.tensor(inputs), torch.tensor(targets)
seq_len_lm = 25
inputs_lm, targets_lm = make_char_batches(encoded, seq_len=seq_len_lm)Number of training sequences: 7 Sequence length: 25 Example: Input: 'To be, or not to be, that' Target: 'o be, or not to be, that '
The input-target offset by one position is the standard autoregressive training setup. The model sees a character and must predict the next one. By training on all consecutive pairs in the corpus, the model learns the statistical regularities of the text: which characters commonly follow which others, which word patterns are frequent, and which punctuation conventions appear.
Training the Character LM
torch.manual_seed(42)
char_model = CharLSTM(
vocab_size=vocab_size, embed_dim=32, hidden_size=64, num_layers=2
)
char_optimizer = optim.Adam(char_model.parameters(), lr=0.005)
char_criterion = nn.CrossEntropyLoss()
char_losses = []
for epoch in range(150):
char_model.train()
epoch_loss = 0.0
for inp, tgt in zip(inputs_lm, targets_lm):
inp_b = inp.unsqueeze(0) # (1, seq_len)
tgt_b = tgt.unsqueeze(0)
char_optimizer.zero_grad()
logits, _ = char_model(inp_b)
loss = char_criterion(logits.view(-1, vocab_size), tgt_b.view(-1))
loss.backward()
# Gradient clipping prevents occasional exploding gradients in stacked LSTMs
torch.nn.utils.clip_grad_norm_(char_model.parameters(), max_norm=1.0)
char_optimizer.step()
epoch_loss += loss.item()
char_losses.append(epoch_loss / max(len(inputs_lm), 1))Initial loss: 3.2606 Final loss: 0.0108 Improvement: 99.7%
Generating Text
def generate_text(model, seed_text, num_chars=100, temperature=0.8):
"""
Generate characters autoregressively from a seed string.
Temperature scales the logits before softmax sampling.
Lower values produce more predictable output; higher values more varied.
"""
model.eval()
generated = seed_text
hidden = None
with torch.no_grad():
# Prime the LSTM hidden state with the seed characters
for char in seed_text[:-1]:
idx = torch.tensor([[char_to_idx[char]]])
_, hidden = model(idx, hidden)
# Generate new characters one at a time
next_char = seed_text[-1]
for _ in range(num_chars):
idx = torch.tensor([[char_to_idx[next_char]]])
logits, hidden = model(idx, hidden)
probs = torch.softmax(logits[0, -1] / temperature, dim=0)
next_idx = torch.multinomial(probs, 1).item()
next_char = idx_to_char[next_idx]
generated += next_char
return generatedSeed: 'To be' Generated: To be, or not to be, that a s eo thet aks againd take arms againd a sea of troubles And againd the be, th
Even on a small corpus with brief training, the LSTM generates plausible character sequences. The temperature parameter plays an large effect in generation quality. Low temperatures (0.3 to 0.5) make the model more conservative: it selects characters with high probability more consistently, producing repetitive but well-formed text. High temperatures (1.2 to 1.5) make the model more exploratory: it sometimes picks lower-probability characters, producing more varied but potentially less coherent output. Temperature 1.0 corresponds to unmodified sampling from the model's distribution. In practice, values around 0.7 to 0.9 often produce a good balance of coherence and diversity.
Given more data and training, the model would learn word boundaries, punctuation conventions, and longer structural patterns. This is the foundation underlying language models that operate at the subword level, though modern systems have replaced LSTMs with transformers as the underlying architecture.
Key Parameters
When configuring nn.LSTM:
- input_size: Dimensionality of the input at each timestep. For embedded text, this is the embedding dimension.
- hidden_size: Dimensionality of both the hidden state and cell state. Larger values (128 to 1024) provide more capacity but increase computation.
- num_layers: Number of stacked LSTM layers. More layers can model more complex patterns but require more gradient clipping and careful tuning.
- batch_first: When
True, expects input shape(batch, seq_len, features). UsingTruealigns with standard data loaders. - dropout: Applied between LSTM layers when
num_layers > 1. Values in the range 0.1 to 0.5 reduce overfitting. - bidirectional: When
True, processes the sequence in both directions simultaneously, doubling the effective hidden size. Useful for classification but inapplicable for generation.