Reward Hacking: Why AI Exploits Imperfect Reward Models
Examines reward hacking in RLHF where language models exploit proxy objectives. Topics include distribution shift, over-optimization, and mitigation strategies.
702 items · Page 7 of 15
Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.
Free to join · Takes less than a minute
Why
I created this space so readers can learn together, ask questions, and make sense of difficult ideas.

Examines reward hacking in RLHF where language models exploit proxy objectives. Topics include distribution shift, over-optimization, and mitigation strategies.
Explains Credit Valuation Adjustment for derivatives pricing, including exposure profiles, default probability modeling, and the broader XVA framework.
Collect and process human preference data for RLHF. Topics include pairwise comparisons, annotator guidelines, quality metrics, and interface design.
Examines the AI alignment problem and HHH framework. Explains why training language models to be helpful, harmless, and honest presents fundamental challenges.
Covers credit risk measurement through Probability of Default, Loss Given Default, and Exposure at Default. Topics include loan pricing and portfolio analysis.
Evaluate instruction-tuned LLMs using benchmarks like Alpaca Eval and MT-Bench, human evaluation protocols, and LLM-as-Judge automatic methods.
Covers VaR calculation using parametric, historical, and Monte Carlo methods. Examines Expected Shortfall and stress testing for market risk management.
Covers market, credit, liquidity, operational, and model risk. Topics include Basel III capital requirements and risk management governance structures.
Covers Black-Litterman models, robust optimization, practical constraints, and risk parity for institutional portfolio management.
Covers Brinson attribution for sector allocation and selection effects, plus factor-based methods to separate investment alpha from systematic beta exposures.
Covers practical techniques for creating instruction-tuning datasets. Topics include human annotation, template-based generation, seed expansion.
Covers Sharpe ratio, Sortino ratio, information ratio, and maximum drawdown metrics. Evaluate portfolios with Python implementations.
Covers Arbitrage Pricing Theory and multi-factor models. Topics include Fama-French factors, estimate factor loadings via regression.
Compare LoRA, QLoRA, Adapters, IA³, Prefix Tuning, and Prompt Tuning across efficiency, performance, and memory. Practical guide for choosing PEFT methods.
Covers Capital Asset Pricing Model: systematic risk, beta estimation, Security Market Line, and alpha. Essential foundations for asset pricing.
Covers model calibration techniques for quantitative finance. Topics include SABR, Heston, GARCH.
Covers PCA for extracting factors from yield curves and equity returns. Topics include dimension reduction, eigendecomposition.
Covers regression analysis for finance: estimate market beta, test alpha significance, diagnose heteroskedasticity.
Covers GARCH and ARCH models for time-varying volatility forecasting. Topics include estimation, persistence analysis, and dynamic VaR with Python examples.
Covers autoregressive and moving average models for financial time-series. Topics include stationarity, ACF/PACF diagnostics, ARIMA estimation, and forecasting.
Covers prompt tuning for efficient LLM adaptation. Prepend trainable soft prompts to inputs while keeping models frozen. Scales to match full fine-tuning.
Explains how IA3 adapts large language models by rescaling activations with minimal parameters. Compare IA3 vs LoRA for efficient fine-tuning strategies.
Covers Vasicek and CIR short-rate models for interest rate dynamics. Topics include mean reversion, bond pricing formulas, and derivative valuation techniques.
Covers exotic options pricing including Asian, barrier, lookback, and digital options. Topics include closed-form solutions and Monte Carlo simulation methods.
Covers Monte Carlo simulation for derivative pricing. Topics include risk-neutral valuation, path-dependent options like Asian and barrier options.
Covers LoRA's mathematical foundations including low-rank decomposition, gradient computation, rank selection.
Covers binomial tree option pricing with the Cox-Ross-Rubinstein model. Price American and European options using backward induction and risk-neutral valuation.
Explains why PEFT is essential for LLMs. Analyze storage costs, training memory requirements, and how adapter swapping enables efficient multi-task deployment.
Covers option Greeks: delta, gamma, theta, vega, and rho. Topics include sensitivity analysis, delta hedging, and portfolio risk management techniques.
Covers few-shot fine-tuning techniques for language models. Topics include PET, SetFit, and data augmentation to achieve strong results.
Derive the Black-Scholes-Merton PDE with Itô's lemma, delta hedging, and no-arbitrage assumptions, then connect the equation to option pricing.
Covers the no-arbitrage principle, replicating portfolios, and risk-neutral probabilities.
Explains how transfer learning enables pre-trained models to adapt to new NLP tasks. Topics include pre-training, fine-tuning, layer representations.
Examines heavy tails, volatility clustering, and the leverage effect in financial returns, with their implications for risk modeling.
Covers CDO mechanics, cash flow waterfalls, and correlation risk. Topics include tranche valuation, the Gaussian copula model, and lessons from the 2008 crisis.
Covers CDS pricing using hazard rates and survival probabilities. Topics include credit risk valuation, implied default probabilities, and spread calculations.
Explains how load balancing prevents expert collapse in Mixture of Experts models. Examines token fractions, load metrics.
Covers forward and futures pricing with cost-of-carry models. Topics include no-arbitrage strategies, basis risk, minimum variance hedge ratios.
Covers option strategies by combining basic building blocks. Construct spreads, straddles, and iron condors to visualize payoffs and manage risk.
Explains how sparse models decouple capacity from compute using conditional computation and mixture of experts to achieve efficient scaling.
Examines grokking: how neural networks suddenly generalize long after memorization. Topics include phase transitions, theories, and training implications.
Examines whether LLM emergent capabilities are genuine phase transitions or measurement artifacts.
Explains how chain-of-thought reasoning emerges in large language models. Topics include CoT prompting techniques, scaling behavior.
Covers financial data handling with pandas, NumPy, and Numba. Topics include time series operations, return calculations, and visualization for quant finance.
Explains how in-context learning emerges in large language models. Topics include scale thresholds, ICL vs fine-tuning, induction heads, and meta-learning.
Explains how LLMs suddenly acquire capabilities through emergence. Topics include phase transitions, scaling behaviors, and the ongoing metric artifact debate.
Covers bond pricing through present value calculations, yield to maturity analysis, and price-yield relationships. Topics include fixed income fundamentals.
Transform scaling laws into predictive tools for AI development. Topics include loss extrapolation, capability forecasting.
Create a free account to keep your reading organized, join thoughtful discussions, and get more from every chapter.
Free to join · Takes less than a minute
Why
I created this space so readers can learn together, ask questions, and make sense of difficult ideas.
