Hardware, Kernels, and LLMs #

Overview #

This tutorial is about teaching AI models to write kernels: the small, low-level programs that tell an AI chip exactly how to do a piece of math as fast as possible. This first section sets the stage, and you do not need any special background to follow it.

We build up two things. First, the target that kernels run on: what the specialized AI chips look like inside, how everyday Python code gets translated down into something a chip can run, and the languages people use to hand-write the fastest code. Second, the tool we will later teach: how a language model produces text, and the two main ways of training one to get good at a task, namely learning from worked examples and learning by trial and error.

A recurring theme is that automatic tools get you most of the way, but the last stretch of performance still comes from carefully hand-written code, which is exactly the gap we want an AI to help fill. The goal here is intuition and shared vocabulary, not depth; the later sections are where we dig in.

Contents of this section #

1.1 Overview of AI Hardware: GPUs, TPUs, and AWS Trainium #

A tour of the chips behind modern AI. They come from different makers, but all share the same three ingredients that any fast program has to work around: units that do the math, a small pool of ultra-fast on-chip memory, and wiring that lets many chips team up.

1.2 Overview of AI Software Stacks #

How a friendly Python program travels down through several layers to reach the chip. We look at where today’s compilers already do a great job, and where they leave off and a person (or an AI) still has to hand-write a kernel.

1.3 Overview of System Kernels: CUDA, Triton, and NKI #

A first look at the three languages for hand-writing fast code: CUDA for NVIDIA chips, Triton for more portable code, and NKI for Amazon’s Trainium chips. All three chase the same idea: keep data close to the math so the chip never sits idle waiting on slow memory.

1.4 LLM Decoding and Inference-Time Scaling #

How a language model writes text one word at a time, and why that is slow and expensive. Seeing this explains both why fast kernels are in such demand and why letting a model think a little longer can buy noticeably better answers.

1.5 Supervised Fine-Tuning (SFT) for Code Generation #

The most direct way to turn a general model into a coding helper: show it many tasks paired with correct answers and have it imitate them. It works well, but it can never do better than the examples it copies.

1.6 RL Post-Training: PPO and GRPO Variants #

A second way to train the model, called reinforcement learning: let it try its own answers, score each by whether it actually works, and shift toward the successes. This is how a model can eventually go beyond the examples it first saw.