AI & ML
10 articles
What Is Quantization? Running Big Models on Less VRAM
Quantization is what shrinks a 70B model from 140 GB to ~43 GB — enough for two consumer cards or a big-memory Mac, not one 24 GB GPU. Here's what it actually does, the memory math including the KV cache everyone forgets, how to read GGUF labels like Q4_K_M, and which quant level to pick for your VRAM.
VPRM: A Critical Read of Verifiable Process Reward Models
Seven places where the VPRM paper's abstract, prose, and citations do not line up with its own tables — including two coherence numbers that cannot both be right.
QLoRA Explained: Fine-Tuning Big LLMs on a Single GPU
QLoRA finetunes a frozen 4-bit base model into 16-bit LoRA adapters, dropping the memory to adapt a 65B model from over 780 GB to under 48 GB — one GPU instead of a cluster, with no reported quality loss. Here's how NF4, double quantization, and paged optimizers make 4-bit training work, and what the paper proves versus infers.
Mixtral and Mixture-of-Experts, Explained
Mixtral 8x7B holds ~47B parameters but activates only ~13B per token, matching or beating Llama 2 70B and GPT-3.5 on most metrics at a fraction of the active compute. A walkthrough of how sparse mixture-of-experts works, what the benchmarks show, and the surprising finding that expert assignment appears to align more with syntax than with domain.
RLVR Explained: Verifiable Rewards and Where They Break Down
Most RLVR explainers stop at "it works." This one adds the three published results that complicate it — pass@k, spurious rewards, and what DeepSeek-R1 actually shipped.
LoRA Explained: Low-Rank Adaptation for Cheap Fine-Tuning
LoRA freezes a pretrained model's weights and trains a tiny pair of low-rank matrices to represent the update instead. On GPT-3 175B it trains ~10,000x fewer parameters and matches full fine-tuning quality, and because the update merges back into the weights it adds zero inference latency.
KV Cache Explained: Why LLM Inference Slows Down
LLM generation slows down on long contexts because of one data structure: the KV cache. It grows linearly with every token and must be re-read in full on each decode step, making decode memory-bandwidth bound. Here is the formula, the real numbers, and how GQA, MLA, PagedAttention, and prefix caching fight back.
How LoRA Fine-Tuning Works (and When to Use It)
LoRA freezes a model's pretrained weights and trains a tiny low-rank update instead, cutting trainable parameters by ~10,000x and GPU memory by ~3x. This explainer covers how the mechanism works, what QLoRA and the variants add, and an honest, research-backed framework for when LoRA wins and when full fine-tuning still beats it.
What Is RAG? Retrieval-Augmented Generation Explained
Retrieval-Augmented Generation (RAG) grounds an LLM's answers in information it pulls from an external knowledge source at query time, instead of relying on frozen training data. Here's what RAG is, how the indexing and retrieval pipelines actually work, and when to choose it over fine-tuning or long-context.
How Speculative Decoding Speeds Up LLM Inference
Speculative decoding makes LLM inference 2-3x faster by letting a small draft model guess ahead and a large model verify the guesses in one parallel pass. A rejection-sampling step keeps the output mathematically identical to the slow path. Here's how it works, why it's lossless, and where it stops helping.