Monitoring › Sampling Strategies for Production Traffic

At scale, it is neither practical nor necessary to analyze every request with the same depth. A tiered sampling strategy concentrates coverage where it matters without the storage and compute overhead of full analysis everywhere.

Uniform random sampling at a low rate (1-5%) provides a statistically representative sample for quality analysis and trend monitoring. If you sample 2% of requests uniformly, you can estimate the population mean quality score with high confidence given sufficient volume. For a service handling 100,000 requests per day, 2% sampling yields 2,000 samples, which is more than enough for reliable statistical analysis.

Stratified sampling over-represents the edges of the distribution. Sample at a higher rate for requests with very long contexts (where quality is more variable), very short contexts (where quality might degrade differently), and requests from new users (who are most sensitive to first impressions). Stratified sampling lets you allocate your analysis budget where it provides the most signal.

Anomaly-triggered sampling captures requests that look unusual according to some fast heuristic: quality score below a threshold, latency in the top 1%, or input length in the top 0.1%. These are the requests most likely to contain interesting failure modes. Saving all anomalous requests rather than just a fraction ensures that rare but important failure modes are not lost.

Feedback-triggered sampling captures all requests that received explicit negative feedback from users. If a user clicks "bad response" or "regenerate," that request should be saved in full for analysis. These are your ground-truth labels for quality, and every one is valuable.