Andrej Karpathy’s gist lays out a complete GPT training loop in under 300 lines of pure Python, no dependencies. It begins by fetching a simple text dataset—here, a list of names—and builds a character-level tokenizer that assigns each unique character an integer ID plus one special “beginning of sequence” token. From there he defines a tiny autodiff engine (class Value) that tracks computations and gradients through addition, multiplication, exponentiation, ReLU and so on. That engine underpins both the forward pass and back-propagation without relying on PyTorch or TensorFlow. Next comes the model itself: a one-layer transformer with 16-dim embeddings, 4 attention heads, 16-token context window, and roughly a few thousand parameters (embeddings, query/key/value/projection matrices, feed-forward weights). Position embeddings and token embeddings get summed, then flow through an RMS-norm, a multi-head self-attention block with dot-product scaled by √d, plus residual connections, followed by a two-layer feed-forward network with ReLU. Logits over the next character are produced by a final linear “lm_head” layer. Training uses plain stochastic gradient descent via the Adam update rule (learning rate 0.01, β1=0.85, β2=0.99, ε=1e-8). At each step, a single name sequence (padded with BOS tokens) is passed through the model, the cross-entropy loss is averaged over up to 16 positions, then back-propagated. Parameter updates happen immediately using the moment estimates stored in two Python lists. A thousand training steps wind up teaching the network how to predict the next character in simple name strings. Everything outside of the algorithm—batching, fast GPU kernels, optimized matrix math—is stripped away. What remains is a didactic, end-to-end blueprint of how a GPT-style transformer works under the hood: from raw token IDs through self-attention, MLP layers, loss computation and gradient descent, all expressed in straightforward Python.