When you send a prompt to an LLM, it first turns your text into token IDs via Byte Pair Encoding. That splits words into common chunks—“the” is one token, “unhappiness” becomes “un + happi + ness.” Each ID then pulls a 4,096-dimensional vector from a giant embedding table where similar words sit close together. Position data joins in here too, using schemes like RoPE to tell the model which token came first. Next comes the transformer stack—often 32 layers deep. Each layer builds three new vectors (queries, keys, values) per token, then mixes information across tokens by matching queries with keys and blending in the corresponding values. After that self-attention step, each token passes through a small feed-forward network that does most of the “thinking.” At the final layer, the model projects the last token’s vector back to a 50,000-way vocabulary and picks the highest-probability next token. Generating text splits into two distinct phases. Prefill tackles your entire prompt in parallel, flooding the GPU with matrix-matrix multiplies. It’s compute-bound, and its speed shows up as Time to First Token (TTFT). Decode then takes over one token at a time, reusing cached key and value matrices and doing tiny vector-matrix multiplies. Now the GPU stalls waiting on memory, and Inter-Token Latency (ITL) becomes your measure of speed. That cache of keys and values is both the savior and the headache. Without it, each new token would cost quadratic time. With it, a 13B-parameter model uses about 1 MB of VRAM per token, so a 4,000-token context eats 4 GB just for cache. Fixes range from quantizing cache entries to INT8/INT4, to dropping old tokens, or paging cache data like an OS. Some labs go further: DeepSeek’s V4 combines sparse and dense compressed attention to slash cache size by 90% at a million-token context.