712 items · Page 3 of 15
Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.
Free to join · Takes less than a minute
Why
I created this space so readers can learn together, ask questions, and make sense of difficult ideas.
Design reliable LLM judge prompts using explicit criteria, few-shot examples, and chain-of-thought formatting to maximize evaluation accuracy.
How language models memorize training data, methods for measuring extractable memorization, PII risks in web-scale corpora, and practical privacy mitigations.
Explains how position bias, verbosity bias, and sycophancy distort LLM evaluation. Measure swap consistency, detect length effects.
How RETRO trains language models with retrieval from scratch, using chunked cross-attention to integrate a 2T-token database and cut parameter needs 25x.
Build LLM-as-Judge evaluation pipelines: prompt design, judge model selection, calibration against human annotations, and bias mitigation.
Process reward models score individual reasoning steps instead of final answers alone. Covers training data, credit assignment, math tasks, and limitations.
Explains how Constitutional AI trains safer LLMs using constitutional principles, AI-driven critique and revision, and RLAIF preference labeling.
Evaluate language models with pairwise comparisons and Elo ratings, including preference aggregation, confidence intervals, bias, and significance tests.
Covers chance-corrected agreement metrics for NLP annotation reliability. Calculate Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha with Python examples.
Examines o1-style reasoning models, test-time compute scaling, process reward models, and open research questions shaping the frontier of AI reasoning.
Design rigorous human evaluations for language models. Topics include interface design, annotation paradigms, annotator selection.
Covers benchmark saturation in AI evaluation. Explains why static metrics hit ceiling effects, lose statistical power, and how dynamic benchmarks solve this.
Examines systematic reasoning failures in LLMs including spurious correlations, reasoning shortcuts, negation failures.
Explains how benchmark contamination affects LLM evaluation accuracy. Topics include detection methods including n-gram matching, MinHash.
Explains how LLMs solve math problems, from grade-school word problems to competition math. Topics include chain-of-thought, process reward models, GRPO.
Explains how TruthfulQA tests AI truthfulness by identifying imitative falsehoods.
Explains how process reward models score each reasoning step, how verification-guided search selects correct chains.
Explains how self-consistency, tree of thought, least-to-most prompting, and decomposition strategies improve language model reasoning accuracy and reliability.
Examines the MBPP benchmark for Python code generation evaluation. Topics include crowd-sourced programming tasks, pass@k metrics, execution-based testing.
Explains how chain-of-thought prompting enables language models to reason step by step. Topics include few-shot CoT, zero-shot CoT, self-consistency.
Explains how HumanEval evaluates LLM code generation using functional correctness and the pass@k metric.
Examines LLM text generation for content creation, writing assistance, and code. Topics include quality dimensions, constraint verification, prompt design.
Examines deductive, inductive, abductive, and causal reasoning in LLMs, including how transformers support inference chains and where reasoning breaks down.
GSM8K tests grade-school mathematical reasoning with multi-step word problems. Covers dataset structure, answer scoring, and known benchmark limitations.
Build intelligent dialogue systems with LLMs, covering conversation management, slot filling, memory strategies, and chatbot evaluation techniques.
Apply model merging to combine task fine-tunes, blend styles, compose capabilities, and evaluate merged models using normalized scores and Pareto analysis.
HellaSwag evaluates physical commonsense reasoning with adversarial filtering designed to remove superficial cues and annotation artifacts.
Combine pre-trained models into one without retraining. Topics include weight averaging, task arithmetic, TIES sign-election merging, and DARE random pruning.
MMLU tests language models across 57 subjects from STEM to law. Topics include evaluation protocols, few-shot methods.
Explains how structured pruning removes entire attention heads, transformer layers, and feed-forward neurons to create smaller, faster models.
Explains how weight pruning reduces neural network size by removing redundant parameters.
Covers calibration in machine learning and how model confidence aligns with actual accuracy. Examines ECE, temperature scaling.
Explains how feature distillation, attention transfer, progressive distillation, and on-policy distillation extend knowledge distillation beyond soft labels.
Use Exact Match and token-level F1 to score NLP and question-answering systems, with normalization, partial credit, thresholds, and common scoring errors.
Build multi-layer guardrail systems for LLM applications, covering input validation, output safety, PII detection, topic enforcement.
Explains how differential privacy protects training data in language models, from the mathematical guarantee to DP-SGD, privacy budgets.
BERTScore evaluates text generation using contextual embeddings to measure semantic similarity.
Explains how knowledge distillation transfers a large teacher model's learned knowledge to a smaller student through soft label supervision.
Measure and communicate LLM confidence through calibration, verbalized uncertainty, semantic entropy, and trust-calibrated user interfaces.
Covers ROUGE metrics for summarization evaluation. Topics include ROUGE-N, ROUGE-L, and ROUGE-W, and how they differ from BLEU in assessing text quality.
Covers continual learning evaluation with forward transfer, backward transfer, forgetting metrics.
Examines architecture-based continual learning: progressive networks, PackNet, HAT masks, PathNet.
Explains how experience replay buffers, pseudo-rehearsal, and generative replay prevent catastrophic forgetting in continual learning systems.
Explains how transformer models power sentiment analysis, topic classification, and intent detection.
Explains how the BLEU score evaluates machine translation quality using modified n-gram precision and brevity penalties.
Explains how classification-based and rule-based content filters protect language model deployments.
Explains how Elastic Weight Consolidation and Synaptic Intelligence protect critical parameters to prevent catastrophic forgetting in continual learning.
Covers cross-entropy loss for language models. Topics include entropy, KL divergence, perplexity.
Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.
Free to join · Takes less than a minute
Why
I created this space so readers can learn together, ask questions, and make sense of difficult ideas.