[LLM Algorithms-4] The Evolution of FlashAttention: From IO to GPU Pipelines
A hardware-aware tour of FlashAttention v1, v2, and v3: how IO awareness, GPU parallelism, and Hopper pipelines accelerate exact attention without changing its mathematics.
/ writing
Writing
72 technical notes across AI infrastructure, LLM algorithms, CS336, key concepts, and engineering reviews.
[Infra-16] vLLM KV Connector: Interfaces, Lifecycle, and the LMCache / Mooncake Boundary
A lifecycle-oriented guide to vLLM V1 KV Connector, covering scheduler-worker coordination, lookup, block allocation, asynchronous load/save, and the boundaries between LMCache, NIXL, Mooncake Transfer Engine, and Distributed Store.
[Infra-15] From Inference Engine to Cloud Service: The Full Stack Behind an LLM API
A request-path walkthrough of the complete LLM cloud stack, covering inference engines, model serving, orchestration, gateways, observability, security, billing, and the boundaries between them.
[Kimi-2] Kimi Linear: Rewriting Long-Context Attention with KDA
A systems-oriented derivation of Kimi Delta Attention, covering channel-wise forgetting, the specialized DPLR transition, chunkwise kernels, the 3:1 KDA/MLA hybrid, controlled evaluations, and the quality-throughput trade-off at 1M context.
[Infra-14] vLLM-Omni: From Token Scheduling to a Heterogeneous Generative Runtime
A systems-oriented guide to how vLLM-Omni unifies autoregressive, diffusion, encoder, and audio-decoder workloads through stages, orchestration, transport, hierarchical scheduling, and disaggregated deployment.
[Infra-13] How vLLM Fixed Mamba Prefix Cache Poisoning
A deep dive into vLLM PR #51113: how MTP/EAGLE, concurrent chunked prefill, and prefix caching could advertise an incomplete Mamba recurrent state as a complete cache block, and how scheduler alignment restores correctness.
[Kimi-1] Kimi k1.5 Technical Report: Scaling Reinforcement Learning with 128K Long CoT
A systems-oriented reading of Kimi k1.5, covering verifiable RL prompts, Long-CoT SFT, online mirror-descent policy optimization, length penalties, partial rollouts, hybrid training and inference deployment, multimodal training, and long2short.
[DeepSeek-5] DeepSeek-V4 Technical Report: Hybrid Attention and Post-Training for Million-Token Contexts
A systems-oriented reading of the DeepSeek-V4 preview report, covering the 1.6T/284B MoE models, CSA+HCA hybrid attention, mHC, Muon, 32T-token pretraining, specialist+OPD post-training, FP4 QAT, KV-cache management, million-token evaluation, and limitations.
[DeepSeek-6] DSpark: From Draft Models to Serving-Aware Speculative Decoding
A systems-oriented deep dive into DSpark, covering semi-autoregressive drafting, Markov/RNN sequential heads, confidence heads, STS calibration, hardware-aware prefix scheduling, training objectives, DeepSpec implementation, and DeepSeek-V4 deployment tradeoffs.
[LLM Algorithms-3] Standard MHA: From Multi-Head Computation to KV Cache
A shape-by-shape guide to standard Multi-Head Attention, covering Q/K/V projections, per-head computation, output concatenation, prefill, decode, KV cache, and the differences between MHA, GQA, and MQA.
Notes: MLA, Linear Attention, and Long-Context Memory
A systems-oriented comparison of GQA, MLA, Linear Attention, and Sparse Attention, and why hybrid memory architectures are emerging as the practical direction.
[LLM Algorithms-2] Gated DeltaNet: Teaching Linear Attention to Overwrite and Forget
A derivation of Gated DeltaNet from linear attention, covering error-driven Delta Rule updates, global decay, parallel training, recurrent inference, memory capacity, and comparisons with Transformers and Mamba.
[Skills-08] Learning from Deprecated Skills: Four Workflows Absorbed Elsewhere
Distilling four deprecated skills and tracing how their useful ideas moved into broader, composable workflows.
[Skills-07] Encoding Repetitive Work: Git Guardrails, Scaffolds, and Personal Workflows
Distilling four miscellaneous and two personal skills for Git safety, hooks, migrations, exercise scaffolds, editing, and Obsidian.
[DeepSeek-4] DeepEP: The Token Traffic System for MoE Expert Parallelism
A systems-oriented deep dive into DeepEP V2, covering MoE dispatch and combine semantics, rank-level deduplication, ElasticBuffer, EPHandle, direct and hybrid communication, FP8 transport, SM/QP modeling, and communication-compute overlap.
[Skills-06] Working Across Contexts: Handoffs, Teaching, and Writing Pipelines
Distilling handoff, teach, and seven experimental skills for persistent sessions, interviews, teaching, and writing.
[Skills-05] Maintaining Agent-Friendly Codebases: Deep Modules, Architecture, and Triage
Distilling codebase-design, architecture improvement, research, triage, and TypeScript module boundaries.
[Skills-04] Welding Feedback Loops into Delivery: Prototype, TDD, Debug, and Review
A feedback-loop-centered synthesis of prototype, tdd, implement, diagnosing-bugs, code-review, and resolving-merge-conflicts.
[DeepSeek-3] FlashMLA: Turning MLA's KV-Cache Savings into GPU Throughput
A systems-oriented deep dive into FlashMLA, from weight absorption and its 576/512 layout to paged KV cache, Split-KV, online softmax, seesaw scheduling, sparse attention, FP8 KV cache, and crossover.
[Skills-03] From Conversation to an Executable Map: Specs, Tickets, and Wayfinding
Distilling to-spec, to-tickets, wayfinder, and to-questionnaire into durable planning artifacts.
[DeepSeek-2] DeepSeek-R1 Technical Report: Scaling Reasoning with RL, Verifiers, and Systems Engineering
A systems-oriented reading of DeepSeek-R1 and R1-Zero, covering GRPO, outcome rewards, multi-stage post-training, test-time compute, distillation, RL infrastructure, costs, failures, and limitations.
[Skills-02] Ask the Right Questions: Grilling and Domain Modeling
Distilling grilling, grill-me, grill-with-docs, and domain-modeling into a disciplined alignment workflow.
[DeepSeek-1] DeepSeek-V3 Technical Report: From a 671B MoE to Cost-Efficient Training
A systems-oriented reading of DeepSeek-V3, covering MLA, DeepSeekMoE, auxiliary-loss-free balancing, MTP, FP8, DualPipe, long-context training, deployment, evaluation, and cost accounting.
[Skills-01] Start with the Map: Composing Agent Engineering Workflows
A map of the 41 skills in mattpocock/skills: the main flow, on-ramps, routing, project setup, and principles for predictable skills.
What Is Beam Search?
A technical guide to Beam Search: sequence scoring, expansion and pruning, beam width, length normalization, stopping criteria, and inference costs.
[Infra-12] RL Infrastructure: The Distributed Factory Behind LLM Reinforcement Learning
A systems guide to RL infrastructure for LLM post-training, covering rollout, environments, rewards, trajectories, training, weight synchronization, deployment models, open-source frameworks, and the path from inference infrastructure to RL systems.
What Is a Compiler Pass?
An engineering-oriented introduction to compiler passes: how they analyze and rewrite graphs and IR, and how fusion, lowering, pattern rewriting, and kernels fit together.
[Infra-11] Stride: From Tensor Views to Kernels, KV Cache, and Communication Layouts
A systems-level guide to tensor strides: address calculation, zero-copy views, kernel layout contracts, memory coalescing, attention and paged KV cache layouts, quantization, communication packing, compilation, and debugging.
[Infra-10] All-to-All Optimization: From MoE Token Routing to Communication-Computation Overlap
A systems guide to All-to-All optimization in MoE, covering dispatch and combine, uneven routing, small messages, topology, permutation overhead, hierarchical communication, kernel fusion, overlap, expert placement, and profiling.
[Infra-9] FlexKV: Turning KV Cache into a Tiered, Transferable, Shared Resource
A systems view of FlexKV: prefix matching, RadixTree indexing, asynchronous transfer graphs, and distributed KV reuse with Redis and Mooncake.
[Infra-8] MTP: Multi-Token Prediction Modules
A serving-oriented guide to Multi-Token Prediction: its training objective, DeepSeek-V3 sequential MTP modules, speculative verification, KV cache commit, and scheduler integration.
[MLC-07] Computational Graph Optimization and Pattern Rewriting
Notes for lesson seven: Relax ASTs, visitors, operator pattern rewriting, Linear + ReLU fusion, and mapping graph optimization to TensorIR calls.
[MLC-06] GPU Acceleration and Tensorization
Notes for lesson six: thread blocks, local/shared-memory tiling, hardware specialization, and tensorization for mapping tensor programs to GPUs.
[MLC-05] Integrating with Machine Learning Frameworks
Notes for lesson five: TE, BlockBuilder, and PyTorch FX integration for importing framework models into IRModule.
[MLC-04] Automated Program Optimization and Meta-Schedule
Notes for lesson four: turning schedules into a search space, stochastic schedule transformations, candidate measurement, and the Meta-Schedule optimization loop.
[Infra-6] xLLM: From Inference Engine to Cluster Serving
Where xLLM fits in LLM infrastructure: not a model, but an inference engine and clustered serving system for diverse AI accelerators, with Service-Engine decoupling, PD/EPD disaggregation, global KV cache, dynamic scheduling, and MoE optimizations.
[Infra-7] Distributed Inference Parallelism: TP, PP, DP, CP, EP, and PD Disaggregation
A unified guide to DP, TP, PP, SP, CP, EP, and prefill-decode disaggregation in LLM inference: what each dimension partitions, how they compose, and where MoE and KV cache create practical coupling.
[LLM Algorithms-1] Understanding PPO, DPO, and GRPO Through Comparison Signals
A mathematical comparison of PPO, DPO, and GRPO through policy gradients, KL regularization, critics, rewards, old policies, reference policies, and practical tradeoffs.
[MLC-03] End-to-End Model Execution and Relax
Notes for lesson three: using Relax graphs, call_tir, dataflow blocks, and parameter binding to assemble TensorIR functions into an executable model.
[MLC-02] Tensor Program Abstraction and TensorIR
Notes for lesson two: primitive tensor functions, loops, buffers, block axes, and why TensorIR schedules make tensor programs optimizable.
[Infra-5] K8s and Infra
A practical walkthrough of where Kubernetes fits in AI infrastructure: deployment, scheduling, scaling, recovery, and its relationship with Docker, Ray, Slurm, and LLM serving engines.
[MLC-01] Machine Learning Compilation Overview
Notes for lesson one of Machine Learning Compilation: development and deployment forms, tensor functions, and the abstraction-implementation boundary.
What Is an N-Gram Model?
A practical explanation of N-Gram language models: chain rule, Markov assumption, count-based estimation, sparsity, smoothing, backoff, interpolation, and how they differ from Transformers.
Notes: Why Prefix Caching Should Be Disabled for Compute Benchmarks
A systems note on why prefix caching can turn hardware compute benchmarks into measurements of cache hit rate, serving policy, and workload repetition.
Talking About Reasoning
A narrative note on what reasoning changes and does not change, from training and reasoning tokens to vLLM serving, PD disaggregation, and production benchmarking.
What Are Checkpoints?
A practical explanation of checkpoints in LLMs: what they store, how they enable resume training, how they relate to model weights and .safetensors, and how serving systems load them.
[Infra-4] Prefill-Decode Disaggregation
A practical walkthrough of prefill-decode disaggregation: why prefill and decode conflict, what is actually disaggregated, how it affects TTFT and TPOT, and why KV cache transfer becomes the new systems bottleneck.
What Is CUDA Graph?
A practical explanation of CUDA Graph: how it captures stable CUDA workloads for replay, why it reduces launch overhead, and how it is used in PyTorch and LLM serving.
What Is Linear Attention?
A practical explanation of Linear Attention: how kernel feature maps and recurrent states reduce long-context cost, and why Gamma, Delta, Gate, and Conv variants matter.
What Is muP?
An intuitive guide to Maximal Update Parametrization, covering its motivation, relation to NTK and standard parametrization, Transformer-specific changes, and hyperparameter transfer.
[Infra-3] Distributed Training and Distributed Inference
A practical comparison of distributed training and distributed inference across goals, dataflow, memory, communication, performance metrics, scheduling, and vLLM deployment.
What Is a Skill?
A practical explanation of skills in AI agents: reusable workflow packages that combine instructions, metadata, optional scripts, and resources.
[Infra-2] NCCL
A first-person walkthrough of NCCL: its role in distributed GPU systems, common collectives, topology-aware communication, multi-node paths, and practical debugging.
[Infra-1] How to Compute Bus Bandwidth and Algorithm Bandwidth Across Topologies
A practical guide to link bandwidth, injection bandwidth, bisection bandwidth, algbw, and busbw across PCIe, ring, NVSwitch, and fat-tree topologies.
First Technical Blog: Vibe Coding a Personal Site with Codex
From outgrowing a resume to building a dynamic personal showcase with Next.js, MDX, Tailwind, and GitHub Pages.
[CS336-17] Policy Gradient and GRPO
Notes for Stanford CS336 Spring 2025 lecture 17: Policy Gradient, GRPO.
[CS336-16] RLVR
Notes for Stanford CS336 Spring 2025 lecture 16: RLVR, GRPO.
[CS336-15] RLHF and Alignment
Notes for Stanford CS336 Spring 2025 lecture 15: Alignment, RLHF, SFT.
[CS336-14] Data Filtering and Dedup
Notes for Stanford CS336 Spring 2025 lecture 14: Data Filtering, Dedup.
[CS336-13] Data Overview
Notes for Stanford CS336 Spring 2025 lecture 13: Data, Pretraining.
[CS336-12] Evaluation
Notes for Stanford CS336 Spring 2025 lecture 12: Evaluation, Benchmarks.
[CS336-11] Scaling Details
Notes for Stanford CS336 Spring 2025 lecture 11: Scaling, muP.
[CS336-10] Inference
Notes for Stanford CS336 Spring 2025 lecture 10: Inference, KV Cache.
[CS336-09] Scaling Laws Basics
Notes for Stanford CS336 Spring 2025 lecture 9: Scaling Laws, Chinchilla.
[CS336-08] Distributed Training in Code
Notes for Stanford CS336 Spring 2025 lecture 8: torch.distributed, NCCL.
[CS336-07] Parallelism Basics
Notes for Stanford CS336 Spring 2025 lecture 7: Parallelism, ZeRO.
[CS336-06] Kernels and Triton
Notes for Stanford CS336 Spring 2025 lecture 6: Triton, Kernels.
[CS336-05] GPUs
Notes for Stanford CS336 Spring 2025 lecture 5: GPU, CUDA, FlashAttention.
[CS336-04] Mixture of Experts
Notes for Stanford CS336 Spring 2025 lecture 4: MoE, Routing.
[CS336-03] Architecture and Hyperparameters
Notes for Stanford CS336 Spring 2025 lecture 3: Architecture, Transformer.
[CS336-02] PyTorch and Resource Accounting
Notes for Stanford CS336 Spring 2025 lecture 2: PyTorch, FLOPs, Memory.
[CS336-01] Overview and Tokenization
Notes for Stanford CS336 Spring 2025 lecture 1: Overview, Tokenization.