Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Human Evaluation Design › Code Implementation: Simulating Evaluation Design

Michael BrenndoerferMarch 7, 202610 min read

Language AI HandbookAll chapters

Part 55 of 67Human and Model Evaluation

Chapter 423Human Evaluation Design › Code Implementation: Simulating Evaluation Design

Reading section 8 of 12 · Code Implementation: Simulating Evaluation Design · 10 minSection 8 of 12 · 10 min

Section 8 of 12 · 10 min read

Human Evaluation Design › Code Implementation: Simulating Evaluation DesignOne section · Sections in this chapter8 / 12 ↓

Sections in this chapter12

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's implement a simple framework for calculating evaluation costs and determining sample sizes. This isn't a full annotation interface (which would require web development), but rather a toolkit for planning your evaluation study. These functions allow you to explore cost-quality tradeoffs before committing to expensive data collection.

We'll create functions to estimate costs and perform power analysis for comparative evaluation studies.

In[6]:

Code

def calculate_evaluation_cost(
    n_items: int,
    n_annotators_per_item: int,
    minutes_per_item: float,
    hourly_wage: float,
    platform_fee_percent: float = 0.15,
    qualification_time_minutes: float = 10,
    n_qualification_attempts: int = 50,
) -> dict:
    """
    Calculate total evaluation cost including qualification and main task.

    Parameters:
    -----------
    n_items : int
        Number of items to annotate
    n_annotators_per_item : int
        Number of annotators per item (for reliability)
    minutes_per_item : float
        Estimated annotation time per item
    hourly_wage : float
        Annotator hourly wage in dollars
    platform_fee_percent : float
        Platform fee as decimal (0.15 = 15%)
    qualification_time_minutes : float
        Time to complete qualification test
    n_qualification_attempts : int
        Number of people who will attempt qualification

    Returns:
    --------
    dict : Cost breakdown
    """
    # Main annotation cost
    total_annotations = n_items * n_annotators_per_item
    hours_per_annotation = minutes_per_item / 60
    base_cost = total_annotations * hours_per_annotation * hourly_wage

    # Platform fees
    platform_cost = base_cost * platform_fee_percent

    # Qualification costs (pay everyone who attempts)
    qual_hours = (qualification_time_minutes * n_qualification_attempts) / 60
    qual_cost = qual_hours * hourly_wage * (1 + platform_fee_percent)

    total = base_cost + platform_cost + qual_cost

    return {
        "main_annotation_cost": base_cost,
        "platform_fees": platform_cost,
        "qualification_cost": qual_cost,
        "total_cost": total,
        "total_annotator_hours": (total_annotations * hours_per_annotation)
        + qual_hours,
        "cost_per_item": total / n_items,
    }

Now let's calculate the cost for our chatbot evaluation example:

In[7]:

Code

costs = calculate_evaluation_cost(
    n_items=500,
    n_annotators_per_item=3,
    minutes_per_item=1.0,
    hourly_wage=20.0,
    platform_fee_percent=0.20,
    qualification_time_minutes=10,
    n_qualification_attempts=50,
)

Out[8]:

Console

Evaluation Cost Breakdown:
Main annotation: $500.00
Platform fees: $100.00
Qualification: $200.00
Total cost: $800.00
Cost per item: $1.60
Total annotator hours: 33.3

Out[9]:

Visualization

Stacked horizontal bar chart showing breakdown of evaluation costs across main annotation, platform fees, and qualification. — Cost breakdown for evaluating 500 chatbot conversations with three annotators per item. Main annotation labor constitutes the largest share of expenses, while qualification testing adds significant fixed overhead proportional to the number of workers screened. Platform fees scale linearly with labor costs, illustrating how the replication factor and workforce size together determine total study cost.

The qualification phase adds substantial fixed costs to our study, as shown in the breakdown above. This shows why it's important to budget for qualification upfront rather than treating it as an afterthought. Now let's examine sample size requirements for detecting differences between models.

In[10]:

Code

from scipy import stats


def sample_size_for_comparison(
    effect_size: float,
    alpha: float = 0.05,
    power: float = 0.80,
    test_type: str = "paired",
) -> int:
    """
    Calculate required sample size for comparing two systems.

    Uses Cohen's d for effect size. For paired tests (within-subjects),
    this assumes we're comparing the same items across systems.

    Parameters:
    -----------
    effect_size : float
        Expected Cohen's d (0.2=small, 0.5=medium, 0.8=large)
    alpha : float
        Significance level
    power : float
        Desired statistical power (1 - beta)
    test_type : str
        'paired' or 'independent'

    Returns:
    --------
    int : Required sample size per group
    """
    # For paired t-test
    if test_type == "paired":
        z_alpha = stats.norm.ppf(1 - alpha / 2)
        z_beta = stats.norm.ppf(power)
        n = ((z_alpha + z_beta) / effect_size) ** 2
        return int(np.ceil(n))
    else:
        # For independent samples (conservative estimate)
        z_alpha = stats.norm.ppf(1 - alpha / 2)
        z_beta = stats.norm.ppf(power)
        n = 2 * ((z_alpha + z_beta) / effect_size) ** 2
        return int(np.ceil(n))

In[11]:

Code

import pandas as pd

# Calculate sample sizes for different effect sizes
effect_sizes = [0.2, 0.3, 0.5, 0.8]
scenarios = []

for d in effect_sizes:
    n_paired = sample_size_for_comparison(d, test_type="paired")
    n_indep = sample_size_for_comparison(d, test_type="independent")
    scenarios.append(
        {
            "Effect Size (Cohen's d)": d,
            "Interpretation": {
                0.2: "Small",
                0.3: "Medium-Small",
                0.5: "Medium",
                0.8: "Large",
            }[d],
            "Paired Test N": n_paired,
            "Independent Test N": n_indep,
            "With 3x Replication": n_paired * 3,
        }
    )

df = pd.DataFrame(scenarios)

Out[12]:

Console

 Effect Size (Cohen's d) Interpretation  Paired Test N  Independent Test N  With 3x Replication
                     0.2          Small            197                 393                  591
                     0.3   Medium-Small             88                 175                  264
                     0.5         Medium             32                  63                   96
                     0.8          Large             13                  25                   39

Notice how quickly required samples grow for small effect sizes. Detecting a small improvement (Cohen's $d = 0.2$ ) requires many more items than detecting a medium effect, as shown in the table above. This directly affects evaluation cost. If you need to detect small improvements (common in mature systems where large gains are rare), your evaluation budget must scale accordingly. Conversely, if you only care about large improvements, you can run smaller, cheaper studies.

Let's visualize the cost-effectiveness tradeoff:

Out[13]:

Visualization

Line plot showing statistical power versus sample size for three effect sizes, with cost on the secondary right axis. — Statistical power as a function of sample size for three effect sizes (Cohen's d = 0.3, 0.5, and 0.8), with total evaluation cost shown on the right axis for three annotators per item at \$0.40 per annotation. Larger effects reach 80% power at much smaller and cheaper samples. The horizontal dashed line marks the conventional 80% power threshold used to determine minimum study size.

The visualization reveals the economic reality of evaluation design. Achieving 80% power to detect a medium effect (Cohen's $d = 0.5$ ) costs substantially less than detecting a small effect (Cohen's $d = 0.3$ ), as illustrated by the cost axis in the figure. This is why pilot studies to estimate effect sizes are important; they prevent underpowered studies that waste money on inconclusive results. The curves also show diminishing returns: increasing sample size from 50 to 100 gives substantial power gains, but increasing from 400 to 450 gives minimal benefit at significant cost.

Finally, let's simulate annotator agreement to understand how replication affects reliability:

In[14]:

Code

def simulate_annotator_agreement(
    n_items: int,
    n_annotators: int,
    true_quality: np.ndarray,
    annotator_noise: float,
) -> dict:
    """
    Simulate annotations with random noise to show reliability concepts.

    Parameters:
    -----------
    n_items : int
        Number of items to simulate
    n_annotators : int
        Number of annotators per item
    true_quality : np.ndarray
        True quality scores (0-1 scale)
    annotator_noise : float
        Standard deviation of annotator error

    Returns:
    --------
    dict : Simulated data and agreement metrics
    """
    annotations = np.zeros((n_items, n_annotators))

    for i in range(n_annotators):
        noise = np.random.normal(0, annotator_noise, n_items)
        annotations[:, i] = np.clip(true_quality + noise, 0, 1)

    # Calculate inter-annotator correlation (average pairwise)
    correlations = []
    for i in range(n_annotators):
        for j in range(i + 1, n_annotators):
            corr = np.corrcoef(annotations[:, i], annotations[:, j])[0, 1]
            correlations.append(corr)
    avg_correlation = np.mean(correlations)

    # Reliability of mean rating (Spearman-Brown prophecy)
    reliability = (n_annotators * avg_correlation) / (
        1 + (n_annotators - 1) * avg_correlation
    )

    return {
        "annotations": annotations,
        "avg_pairwise_correlation": avg_correlation,
        "reliability_of_mean": reliability,
        "std_dev_of_means": np.std(np.mean(annotations, axis=1)),
    }

The Spearman-Brown prophecy formula used inside this function deserves an explanation. Given a set of $n$ annotators with average pairwise correlation $\bar{r}$ , the reliability of their aggregated mean rating is:

\rho_n = \frac{n \bar{r}}{1 + (n - 1)\bar{r}}

where:

$n$ : the number of annotators being aggregated
$\bar{r}$ : the average pairwise correlation between individual annotators
$\rho_n$ : the reliability (intraclass correlation) of the mean of all $n$ annotators

This formula tells you how much adding more annotators improves reliability. When $\bar{r}$ is low (noisy annotators), each additional annotator gives more marginal improvement than when $\bar{r}$ is already high. You can use this formula before data collection, given an assumed pairwise correlation from similar work, to determine how many annotators you need to reach a reliability target such as 0.80.

In[15]:

Code

np.random.seed(42)

# Simulate 200 items with varying true quality
true_quality = np.random.beta(2, 2, 200)  # Beta distribution centered at 0.5

# Run simulations for different numbers of annotators
simulation_results = []
for n_ann in [1, 2, 3, 5]:
    result = simulate_annotator_agreement(
        n_items=200,
        n_annotators=n_ann,
        true_quality=true_quality,
        annotator_noise=0.15,
    )
    simulation_results.append((n_ann, result))

Out[16]:

Console

Effect of Annotator Replication on Reliability

------------------------------------------------------------
1 annotator(s):
  Avg pairwise correlation: nan
  Reliability of mean rating: nan
  Std dev of aggregate scores: 0.255

2 annotator(s):
  Avg pairwise correlation: 0.729
  Reliability of mean rating: 0.843
  Std dev of aggregate scores: 0.244

3 annotator(s):
  Avg pairwise correlation: 0.672
  Reliability of mean rating: 0.860
  Std dev of aggregate scores: 0.222

5 annotator(s):
  Avg pairwise correlation: 0.695
  Reliability of mean rating: 0.919
  Std dev of aggregate scores: 0.218

Out[17]:

Visualization

Line plot showing Spearman-Brown reliability coefficient versus number of annotators per item, with target reliability marked. — Reliability of mean ratings as a function of annotator count per item, calculated using the Spearman-Brown prophecy formula with an assumed pairwise correlation of 0.65. Reliability increases substantially with the first two or three annotators but produces diminishing returns beyond that point. The dashed red line marks the 0.80 reliability target commonly used in psychometrics, which is reached with approximately three annotators given this pairwise agreement level.

The simulation shows the Spearman-Brown prophecy formula in action: while individual annotators show moderate agreement, aggregating three annotators yields high reliability. However, diminishing returns set in after three annotators; moving from three to five annotators gives minimal reliability improvement while materially increasing cost. This finding supports the common practice of using three annotators per item as a cost-effective balance between reliability and expense.