AI & ML
Explainers on how LLMs are trained and run: LoRA and QLoRA fine-tuning, quantization, the KV cache, speculative decoding, and reward methods such as RLVR. Paper reads link to the original papers.
8 articles
What Is Quantization? Running Big Models on Less VRAM
Quantization is what shrinks a 70B model from 140 GB to ~43 GB — enough for two consumer cards or a big-memory Mac, not one 24 GB GPU. Here's what it actually does, the memory math including the KV cache everyone forgets, how to read GGUF labels like Q4_K_M, and which quant level to pick for your VRAM.
VPRM: A Critical Read of Verifiable Process Reward Models
Seven places where the VPRM paper's abstract, prose, and citations do not line up with its own tables — including two coherence numbers that cannot both be right.
QLoRA Explained: Fine-Tuning Big LLMs on a Single GPU
QLoRA finetunes a frozen 4-bit base model into 16-bit LoRA adapters, dropping the memory to adapt a 65B model from over 780 GB to under 48 GB — one GPU instead of a cluster, with no reported quality loss. Here's how NF4, double quantization, and paged optimizers make 4-bit training work, and what the paper proves versus infers.
RLVR Explained: Verifiable Rewards and Where They Break Down
Most RLVR explainers stop at "it works." This one adds the three published results that complicate it — pass@k, spurious rewards, and what DeepSeek-R1 actually shipped.
Claude Sonnet 5: Near-Opus Agents at Sonnet Prices — With a Tokenizer Catch
Claude Sonnet 5 delivers near-Opus performance for coding and agents at $3/$15 — but a new tokenizer means the real cost math depends on which model you're coming from. Opus 4.8 users get a clean 40% saving; Sonnet 4.6 users on English-heavy workloads may effectively pay ~40% more after the intro pricing ends August 31.
KV Cache Explained: Why LLM Inference Slows Down
LLM generation slows down on long contexts because of one data structure: the KV cache. It grows linearly with every token and must be re-read in full on each decode step, making decode memory-bandwidth bound. Here is the formula, the real numbers, and how GQA, MLA, PagedAttention, and prefix caching fight back.
How LoRA Fine-Tuning Works (and When to Use It)
LoRA freezes a model's pretrained weights and trains a tiny low-rank update instead, cutting trainable parameters by ~10,000x and GPU memory by ~3x. This explainer covers how the mechanism works, what QLoRA and the variants add, and an honest, research-backed framework for when LoRA wins and when full fine-tuning still beats it.
How Speculative Decoding Speeds Up LLM Inference
Speculative decoding makes LLM inference 2-3x faster by letting a small draft model guess ahead and a large model verify the guesses in one parallel pass. A rejection-sampling step keeps the output mathematically identical to the slow path. Here's how it works, why it's lossless, and where it stops helping.