Michael Nielsen’s tip—to lock in new knowledge by building something demanding—frames much of this week’s deep dives. First up: distilling large models. If it costs around $25 million to feed 1 trillion tokens from a frontier model at $25 per million tokens, open-source efforts can catch up fast. Hiding chain-of-thought won’t help much—you can prompt models to skip writing thoughts or reconstruct them as an RL target. And real leverage comes when models manage tool use on local machines: writing code, running bash, updating files. Cloud-only workflows are a tough sell. A slick workaround: coding products can record user edits as the “gold diff” and train their own model to mimic only those end-state outputs, rewarding convergence and punishing dead ends. Then there’s why pretraining runs crash or underperform. One culprit is breaking causality in expert routing. Training setups sometimes balance token load across experts at the cost of feeding a token information about future routing decisions—information unavailable at inference time. Rumor has it that this explains Llama 4’s disappointing results. Token dropping—which skips low-priority tokens to save compute—also injects bias, because later tokens can influence whether earlier ones get processed. Numerical bias is another silent killer. Early GPT-4 training reportedly stalled due to FP16 collectives rounding sums of many small gradients by orders of magnitude once accumulators exceeded 1 024. That bug took weeks to pinpoint. A veteran engineer argues that no single fix will smooth out at every scale. As labs push into bigger models, new failure modes will emerge—whether in numerics, routing, or supply chains of GPUs. He’s skeptical that AI will write high-performance kernels on its own anytime soon; Nvidia’s top experts needed months to tune Blackwell. Even small drift between training and inference engines in RL can skew policy gradients. That makes disciplined processes for stacking compute multipliers essential to catch subtle biases before they cascade.