Fine-tuning Data Efficiency › Data Augmentation for NLP

Data augmentation artificially expands the training set by creating modified versions of existing examples. While augmentation has been especially effective for computer vision (random crops, rotations, color changes), text augmentation is more challenging because small changes can alter meaning sharply. Changing a single word can flip the sentiment of a sentence, and rearranging phrases can make text ungrammatical. The goal is to introduce real variation while preserving the label's validity.

The simplest augmentation replaces words with synonyms:

This preserves the overall meaning while introducing lexical variation. The model learns that both "quick" and "fast," or "jumps" and "leaps," can express similar concepts, reducing its dependence on specific vocabulary. However, synonym replacement has limitations: not all words have good synonyms, context matters (a "bank" isn't always replaceable with "shore"), and some domains have specialized vocabulary where synonyms don't exist.

Back-translation generates paraphrases by translating text to another language and back:

The translation process naturally introduces variation in word choice and sentence structure while typically preserving meaning. Each translation step makes decisions about how to express concepts, and these decisions may differ from the original phrasing. Using multiple intermediate languages generates more diverse paraphrases, as each language brings its own structural biases and vocabulary choices to the translation process.

Several random operations can augment text data, each introducing a different kind of variation.

Random insertion adds random words from the vocabulary into sentences. While this might seem likely to introduce noise, it encourages the model to focus on key words rather than word order. The model learns to identify important content words even when surrounded by distractors.

Random swap exchanges the positions of two words in a sentence. Again, this seems potentially harmful, but it helps models tolerate minor ordering variations. Many classification tasks do not depend on exact word order, and this augmentation reflects that fact.

Random deletion removes words with some probability. This simulates incomplete or noisy input and prevents over-reliance on specific words. If a model can still classify correctly with some words missing, it has learned more reliable patterns than one requiring all words present.

These random operations work best when applied conservatively. Augmenting too aggressively can create examples that are too different from real data, training the model on a distribution that doesn't match what it will encounter at test time.

Mixup, originally developed for images, creates new training examples by interpolating between existing ones. The intuition behind mixup is simple but powerful: by training on blended examples with soft labels, we encourage the model to behave linearly in the space between training points, leading to smoother decision boundaries. For text, this is less straightforward since discrete tokens cannot be interpolated directly. You cannot average the word "happy" and "sad" to get a word that is half-positive. Instead, text mixup operates in embedding space, where continuous representations make interpolation real:

Given two examples $(x_1, y_1)$ and $(x_2, y_2)$, we create a synthetic example by interpolating their embeddings and labels:

$$\begin{aligned} \tilde{e} &= \lambda \cdot e_1 + (1 - \lambda) \cdot e_2 \\ \tilde{y} &= \lambda \cdot y_1 + (1 - \lambda) \cdot y_2 \end{aligned}$$

where:

• $\tilde{e}$: the embedding vector for the new synthetic example
• $\tilde{y}$: the label for the new example (a "soft" target)
• $\lambda$: the mixing coefficient sampled from a Beta distribution ($\lambda \in [0, 1]$), controlling the ratio between the two examples
• $e_1, e_2$: the embeddings of the original examples (typically [CLS] token embeddings or pooled representations)
• $y_1, y_2$: the labels of the original examples

The first equation creates a new embedding that lies somewhere on the line segment connecting the two original embeddings in the high-dimensional space. When $\lambda = 1$, we recover the first example exactly; when $\lambda = 0$, we recover the second. Intermediate values produce embeddings that blend characteristics of both inputs.

The second equation applies the same interpolation to the labels. This is what makes mixup so distinctive: rather than assigning a hard label to the synthetic example, we assign a soft target that shows the mixture. This process smooths the decision boundary in the embedding space by teaching the model that intermediate regions should have intermediate predictions. For instance, if a positive example ($y=1$) is mixed with a negative example ($y=0$) with $\lambda=0.7$, the target label becomes $0.7$, encouraging the model to reflect this ambiguity in its prediction confidence. The model learns that it should be more confident near pure examples and less confident in the interpolated regions between classes.

The Beta distribution for sampling $\lambda$ is chosen because it naturally produces values between 0 and 1, and its shape parameter controls how often extreme values (near 0 or 1) versus moderate values (near 0.5) are sampled. Lower shape parameters favor interpolations closer to the original examples, while higher values encourage more balanced mixing.

More advanced augmentation uses language models themselves to generate variations. Given a sentence, mask out one or more words and use a masked language model to predict replacements:

This approach generates contextually appropriate substitutions that maintain grammaticality better than random synonym replacement. The language model considers the full context when selecting a replacement word. This keeps the augmented word fits naturally in its surroundings. A word like "dull" is chosen because the model has learned that it appears in similar contexts to "boring."

Generative models can create even more diverse augmentations. Given a few examples of a class, a language model can generate entirely new examples in the same style, potentially multiplying the effective dataset size many times over. This approach can create examples that express the same sentiment or class membership through entirely different phrasings. This gives richer training signal than modifications of existing examples.