Watermarking › Statistical Detection

Detection is a hypothesis testing problem. The detector receives a piece of text and must decide: was this generated by the watermarked model or not? The quality of the detector is measured by its false positive rate and true positive rate.

The challenge is that the same test statistic must work for text about any topic, written in any style, at any level of formality. A watermark scheme that only works on formal prose would be easy to evade by switching styles. The robustness of the statistical test across diverse text types is an important practical concern.

Define:

• $H_0$: the text was not watermarked (null hypothesis)
• $H_1$: the text was generated by the watermarked model (alternative)

The detector computes a test statistic $S$ from the text and compares it to a threshold $\tau$. If $S > \tau$, the detector rejects $H_0$ and claims the text is watermarked.

The two error types are:

• False positive (Type I error): Declaring human-written text as watermarked. This harms innocent people and destroys trust in the detector. If a university relies on a watermark detector to identify academic dishonesty, a false positive could result in a student being wrongly disciplined.
• False negative (Type II error): Failing to detect watermarked text. This lets attackers evade detection. Depending on the application, false negatives may be more or less acceptable than false positives.

The FPR is controlled by setting $\tau$ at the appropriate quantile of the null distribution. If $S$ under $H_0$ follows a standard normal distribution, then $\tau = 2.33$ gives FPR $= 0.01$, meaning 1 in 100 human-written texts will be falsely flagged. The null distribution does not depend on the text's content, only on the assumption that token choices are independent of the key-derived partition. This assumption holds when the text was written without any knowledge of the key.

The TPR depends on how far the watermarked distribution has shifted relative to the null. More signal (higher $\delta$, more tokens) means higher TPR at the same FPR. The formal tool for quantifying this tradeoff is the Receiver Operating Characteristic (ROC) curve, which plots TPR against FPR across all threshold values. A strong watermark pushes the curve toward the upper-left corner, indicating high TPR at low FPR. A weak watermark, insufficient token count, or heavy attack places the curve close to the diagonal, indicating near-chance performance.

A useful single-number summary is the area under the ROC curve (AUC). AUC $= 1.0$ means perfect separation between watermarked and unwatermarked text; AUC $= 0.5$ means the detector is no better than random. In practice, AUC values above 0.95 at text lengths above 100 tokens are achievable with soft watermarking at $\delta = 2.0$.

Watermark detection is a function of the number of tokens $T$. With very few tokens (say $T = 20$), the green token count has high variance under both hypotheses, and the z-score is noisy. Detection becomes reliable only when enough evidence accumulates.

The statistical argument is straightforward. Under the watermarked distribution, the expected green count is $\mu_1 T$ where $\mu_1 > \gamma$ (because green logits are boosted). Under the null, it is $\gamma T$. The z-score is approximately:

$$z \approx \frac{(\mu_1 - \gamma) \sqrt{T}}{\sqrt{\gamma(1-\gamma)}}$$

This grows as $\sqrt{T}$. To achieve a threshold z-score of $z_\alpha = 2.33$ at significance level $\alpha = 0.01$, you need:

$$T \geq \frac{z_\alpha^2 \cdot \gamma(1-\gamma)}{(\mu_1 - \gamma)^2}$$

A higher logit boost $\delta$ increases $\mu_1 - \gamma$, reducing the required $T$ quadratically. Doubling $\delta$ approximately quarters the required token count for reliable detection.

The minimum text length required for reliable detection also depends on:

• $\gamma$: the green list fraction. Asymmetric partitions ($\gamma \neq 0.5$) reduce variance under $H_1$ but may degrade quality more noticeably for tokens in the disadvantaged list.
• Target FPR/TPR: More stringent requirements demand more tokens. Going from FPR $= 0.01$ to FPR $= 0.001$ increases the required z-score from 2.33 to 3.09, requiring roughly 75% more tokens.

As a rough heuristic, soft watermarking with $\delta = 2.0$ and $\gamma = 0.5$ achieves TPR $\approx 0.99$ at FPR $= 0.01$ with about $T = 200$ tokens. For shorter texts, power drops significantly. This is one of the scheme's principal limitations in practice.

The chart below illustrates how the z-score distributions under $H_0$ and $H_1$ shift apart as the logit boost $\delta$ increases. The overlap between the two distributions corresponds to the unavoidable tradeoff between false positive rate and false negative rate.

When $\delta = 0.5$, the two distributions overlap almost completely, and detection is barely better than chance at most thresholds. At $\delta = 1.0$, a clear gap opens, and the fraction of the $H_1$ distribution to the right of the threshold grows substantially. At $\delta = 2.0$, the watermarked distribution has shifted so far right that most of its mass lies above the detection threshold, yielding high TPR. The cost is that the logit boost at this level is strong enough to measurably distort the output distribution, a tradeoff that practitioners must account for.

In practice, a watermark detector might be applied to millions of texts. With FPR $= 0.01$ and one million texts, approximately 10,000 innocent texts will be falsely flagged. This is a multiple testing problem, and failing to account for it leads to a flood of false accusations.

Correcting for multiple testing with a Bonferroni correction tightens the per-test significance threshold. For 1 million tests and a family-wise error rate of 0.05, the per-test threshold becomes:

$$\alpha_\text{corrected} = \frac{0.05}{1{,}000{,}000} = 5 \times 10^{-8}$$

This corresponds to a z-score of about 5.5 rather than 2.33. Achieving that z-score reliably requires significantly longer text than the 200-token rule of thumb at the standard threshold.

The Bonferroni correction is conservative. False discovery rate (FDR) methods like Benjamini-Hochberg are less stringent while controlling the expected proportion of false positives among all detected positives. For a watermarking deployment where the proportion of watermarked texts is substantial (say, 10% of submissions), FDR control may be more appropriate than family-wise error rate control. The choice depends on the application's tolerance for false positives versus the statistical power needed.

Practical deployments must balance detection sensitivity against the false positive burden imposed by the scale of deployment. A school that processes 1,000 submissions per term can afford a tighter threshold than a platform that evaluates 100 million posts per day.

When the goal is to determine whether an AI system (rather than a specific document) is generating watermarked text, evidence can be pooled across multiple short texts. This is especially valuable when individual documents are too short for reliable individual detection: a social media post of 30 words falls well below the minimum length, but if you observe 1,000 such posts from the same account, you can test whether they collectively show the watermark signal.

A meta-analysis approach tests whether the distribution of z-scores across a corpus is consistent with the null or shows systematic enrichment. Fisher's method combines independent p-values:

$$F = -2 \sum_{i=1}^{n} \ln(p_i)$$

where:

• $p_i$: the p-value from the z-test on the $i$-th document
• $n$: the number of documents in the corpus
• $F$: follows a chi-squared distribution with $2n$ degrees of freedom under $H_0$

Under $H_0$, each $p_i$ is uniformly distributed on $[0, 1]$, so $-\ln(p_i) \sim \text{Exponential}(1)$ and $F$ follows a $\chi^2(2n)$ distribution. If the texts are watermarked, p-values will be systematically smaller than uniform, making $F$ larger than expected and yielding a significant combined test.

This allows detection of watermarked content even when individual documents are too short for reliable individual detection. The cost is that you can only attribute the collection to the watermarked model, not any specific document. This is often sufficient for the use case: you want to know whether an account is primarily using a specific AI system, not whether any single post was AI-generated.

Stouffer's z-score method offers an alternative that is more interpretable in terms of the watermark z-score framework:

$$Z_\text{combined} = \frac{\sum_{i=1}^{n} z_i}{\sqrt{n}}$$

where $z_i$ is the watermark z-score for document $i$. Under $H_0$, each $z_i$ has mean 0 and variance 1, so $Z_\text{combined} \sim N(0, 1)$ under $H_0$. Under $H_1$, the mean z-score is positive, and the combined statistic grows as $\sqrt{n}$. This has the appealing interpretation that you are accumulating token evidence across documents as if they were a single long text.