Efficient Inference and On-Device Models

Covers distillation objective, temperature in distillation, teacher selection, distillation for LLMs.

Covers feature distillation, attention transfer, progressive distillation, on-policy distillation.

Covers weight pruning, structured vs unstructured, pruning criteria, pruning schedule.

Covers head pruning, layer pruning, width pruning, structured pruning implementation.

Covers weight averaging, task arithmetic, TIES merging, DARE merging.

Covers multi-task merging, style merging, capability composition, merging evaluation.

Covers KV cache motivation, cache structure, cache memory requirements, cache management.

Covers cache size calculation, batch size effects, sequence length effects, memory bottleneck.

Covers memory fragmentation problem, page-based allocation, vLLM approach, paged attention benefits.

Covers cache eviction strategies, attention sink preservation, H2O algorithm, cache quantization.

Covers quantization fundamentals, per-tensor vs per-channel, symmetric vs asymmetric, calibration.

Covers INT8 range mapping, absmax quantization, smooth quantization, INT8 accuracy.

Covers 4-bit challenges, group-wise quantization, 4-bit accuracy trade-offs, 4-bit formats.

Covers GPTQ algorithm, layer-wise quantization, Hessian approximation, GPTQ implementation.

Covers salient weight preservation, AWQ algorithm, AWQ vs GPTQ, AWQ benefits.

Covers GGML/GGUF history, quantization types, GGUF file format, llama.cpp integration.

Covers speculative decoding concept, draft model selection, verification procedure, acceptance rate.

Covers acceptance criterion, expected speedup, draft quality effects, optimal draft length.

Covers static vs continuous batching, iteration-level scheduling, request completion handling, throughput gains.

Master LLM inference serving architecture, token-aware load balancing, and auto-scaling. Optimize time-to-first-token and throughput for production systems.

Capacity allocation, data quality, architecture choices, and capability boundaries below frontier scale.

Memory bandwidth, kernels, quantization formats, sparsity, and co-design for phones, browsers, and edge devices.

Private fine-tuning, adapters, local retrieval, user control, and update strategies under tight budgets.

Routing, partitioning, privacy boundaries, offline fallbacks, latency, and cost-aware orchestration.

Task fit, energy, latency, privacy, reliability, and comparing systems rather than parameter counts alone.