Watermarking › Watermark Evaluation

Evaluating a watermarking scheme requires balancing three properties that are often in tension: detection reliability, text quality, and robustness to attack. A scheme that achieves near-perfect detection at the cost of garbled text is useless in practice. A scheme that preserves perfect quality but degrades after a single paraphrase is also insufficient. Good evaluation methodology quantifies all three dimensions and reports the Pareto frontier of achievable tradeoffs.

The core metrics for detection quality come from the hypothesis testing framework.

True positive rate (TPR) at a fixed FPR: The fraction of watermarked texts correctly identified. Reporting TPR at a standard FPR (such as 0.01 or 0.001) allows comparison across schemes, because different schemes optimize the TPR/FPR tradeoff differently. A scheme with high TPR at FPR 0.01 might have lower TPR than another scheme at FPR 0.001, depending on the shape of the ROC curve.

Area under the ROC curve (AUC): A threshold-free summary of detection performance. AUC $= 1.0$ means perfect separation; AUC $= 0.5$ means chance performance. AUC is useful for comparing schemes across all operating points, but it can obscure important differences in the tails. Two schemes with the same AUC might have very different performance at very low FPR values, which is the region that matters most for high-stakes deployments.

Average token count for reliable detection: How many tokens are needed to achieve, say, TPR $= 0.95$ at FPR $= 0.01$. Lower is better, because it extends the scheme's utility to shorter texts. This metric is especially important for comparison between soft and hard watermarking and between logit-bias and distortion-free schemes.

These metrics should be reported separately for different text types, including long-form prose, dialogue, code, non-English text, and mixed-language content, because detection power varies with domain. Code, for example, has a highly constrained vocabulary in many contexts (keywords, common identifier names), which affects how often the natural token choice happens to be on the green list.

The watermark must not noticeably degrade the quality of generated text. Quality is measured at multiple levels, and different metrics capture different aspects of degradation.

Automatic metrics:

Perplexity measures the watermarked text's unexpectedness under a reference language model. A text with higher perplexity contains more unusual token choices, which may signal watermark-induced distortion. Perplexity is sensitive to the strength of the logit boost: at $\delta = 2.0$, perplexity increases measurably compared to the unwatermarked baseline, while at $\delta = 0.5$, the increase may be negligible. The reference model for perplexity evaluation should ideally be different from the watermarked model to avoid measuring the original model's own preferences.

BLEU and ROUGE scores measure n-gram overlap between watermarked and unwatermarked outputs on the same prompts. If the watermarked model is forced to use unusual tokens, overlap with the natural output will be lower. These metrics are blunt because high-quality paraphrases can have low n-gram overlap with the original while conveying the same meaning. They are useful as a quick sanity check but should not be the primary quality measure.

Semantic similarity using sentence embeddings captures meaning preservation better than n-gram overlap. By embedding both the watermarked and unwatermarked outputs in a high-dimensional semantic space and measuring cosine similarity, you can assess whether the watermark forced the model to change the meaning of its outputs. A watermark that changes words but not meaning should show high semantic similarity despite low BLEU.

Human evaluation:

Automatic metrics do not always correlate with human perception. Human judges assess fluency (does the text read naturally?), coherence (does it stay on topic and make logical sense?), and relevance (does it faithfully address the prompt?). A watermark that increases perplexity by 10% may still produce text that humans find perfectly natural. Conversely, a small perplexity increase concentrated on a few unnatural word choices may be immediately obvious to human readers.

Human evaluation is the gold standard but expensive to run at scale. Practical evaluation combines automatic screening to identify conditions where watermarks may cause visible degradation, followed by targeted human evaluation of those cases.

Robustness is measured by the TPR of the detector after various attack budgets are applied.

Paraphrase robustness: TPR after applying a paraphrase model, where the budget is constrained so that the paraphrased text maintains semantic similarity above some threshold (typically 0.8 cosine similarity with the original). This measures how much watermark signal survives fluent paraphrasing. A scheme is considered robust to paraphrase if TPR remains above 0.8 at FPR 0.01 after paraphrasing with a constraint of semantic similarity 0.8.

Word substitution robustness: TPR after randomly substituting a fraction $p$ of tokens. This is a controlled experiment that characterizes how many token substitutions the scheme can tolerate. Results should be reported at multiple substitution rates (10%, 20%, 30%, 50%) to reveal the robustness profile rather than just the failure point.

Cropping robustness: TPR when only a prefix or suffix of the text is evaluated. This matters because attackers may crop text to reduce $T$ and thus statistical power, or because legitimate users may quote only part of a watermarked document.

The tradeoff between robustness and detection power is a Pareto frontier. A scheme with high detection power at $T = 100$ tokens may have low robustness to paraphrase; a scheme with high paraphrase robustness may require $T = 500$ tokens for reliable detection. Evaluation must characterize this frontier rather than reporting a single operating point.

Distortion-free watermarks require a different evaluation approach because they do not change the output distribution. Perplexity-based quality metrics will show no degradation by construction. The relevant quality metric is instead semantic fidelity: do the watermarked outputs carry the same meaning and satisfy the same preferences as the unwatermarked outputs? Since the marginal distribution is unchanged but specific samples differ, this is a subtle question. Two texts drawn from the same distribution may differ considerably in quality due to sampling variance.

Detection metrics are the same (TPR/FPR) but typically show lower efficiency (higher required $T$) compared to logit-bias schemes, showing the weaker signal per token that comes from not modifying the output distribution. Robustness comparisons between distortion-free and logit-bias schemes must account for this difference in baseline detection efficiency: it is not fair to compare robustness at the same text length if the two schemes have very different detection power at that length.