Evaluation, Factuality, and Reliability

Covers perplexity calculation, perplexity interpretation, perplexity limitations, comparing perplexities.

Covers cross-entropy definition, bits-per-character, cross-entropy vs perplexity, loss curves.

Covers n-gram precision, brevity penalty, BLEU formula, BLEU limitations, corpus vs sentence BLEU.

Covers ROUGE-N, ROUGE-L, ROUGE-W, ROUGE interpretation, ROUGE limitations.

Covers BERTScore computation, token alignment, BERTScore variants, BERTScore vs BLEU.

Covers exact match scoring, token-level F1, normalization for matching, metric selection.

Covers calibration definition, expected calibration error, calibration plots, calibration methods.

Covers evaluation design principles, metric selection, evaluation pitfalls, evaluation frameworks.

Covers benchmark construction, dataset collection, annotation guidelines, benchmark validity.

Covers benchmark contamination, evaluation brittleness, gaming metrics, evaluation best practices.

Covers MMLU structure, subject coverage, MMLU evaluation protocol, MMLU limitations.

Covers HellaSwag task design, adversarial filtering, HellaSwag evaluation, HellaSwag saturation.

Covers GSM8K problem types, chain-of-thought evaluation, GSM8K accuracy metrics, math reasoning assessment.

Covers HumanEval structure, functional correctness, pass@k metric, HumanEval limitations.

Covers MBPP dataset, MBPP vs HumanEval, code evaluation challenges.

Covers TruthfulQA design, truthfulness vs informativeness, TruthfulQA evaluation methods.

Covers contamination problem, contamination detection methods, n-gram overlap analysis, contamination mitigation.

Covers ceiling effects, benchmark retirement, dynamic benchmarks, benchmark evolution.

Covers evaluation interface design, task instructions, annotator selection, evaluation cost.

Covers Cohen's kappa, Fleiss' kappa, Krippendorff's alpha, handling disagreement.

Covers A/B comparison design, Elo rating systems, preference aggregation, statistical significance.

Learn how to build LLM-as-Judge evaluation pipelines: prompt design, judge model selection, calibration against human annotations, and bias mitigation.

Covers position bias measurement, bias mitigation (swapping), verbosity bias, sycophancy.

Learn how to design reliable LLM judge prompts using explicit criteria, few-shot examples, and chain-of-thought formatting to maximize evaluation accuracy.

Learn how language models hallucinate: intrinsic and extrinsic hallucination, factual errors, fabrication, and inconsistency with NLI-based detection.

Learn four methods for detecting LLM hallucinations: entailment-based scoring, knowledge base verification, self-consistency checks, and learned detection models.

Covers training data issues, exposure bias, knowledge gaps, generation pressure.

Learn how to reduce LLM hallucination using retrieval augmentation, self-consistency decoding, DPO training, and calibrated uncertainty expression.

Learn how language models link generated claims to source documents, evaluate citation accuracy with NLI, and measure attribution precision and recall.

Covers confidence calibration, verbalized uncertainty, sampling-based uncertainty, uncertainty communication.

Fresh questions, private test sets, rolling releases, objective scoring, and contamination controls.

Versioned prompts, model settings, graders, confidence intervals, artifacts, and comparable reports.

Stateful environments, functional outcomes, partial credit, recovery, efficiency, and trajectory analysis.

Bias, self-preference, verbosity effects, rubric validity, judge ensembles, and agreement with experts.

Shadow traffic, interleaving, A/B tests, guardrail metrics, drift, and linking offline scores to user outcomes.

Benchmark access, disclosure, privacy, gaming, deprecation, audit trails, and responsible leaderboard use.