More on the topic...
Generating detailed summary...
Failed to generate summary. Please try again.
Frontier models Claude Fable 5, Opus 4.8, and GPT-5.5 tackled FrogsGame, a long-horizon task where each agent must improve a base model (Qwen3-8B) to solve a puzzle. Opus 4.8 hit 30.9% average pass@4—three times better than its predecessor—and GPT-5.5 reached 9.7%. Both avoided the old trap of training on weak, noisy SFT traces. They leaned into curriculum learning, binary rewards, and better checkpointing, but still stumble over evaluation: small sample sizes, noisy metrics and bloated internal tests lead to misjudged progress.
Claude Fable 5 flipped the script by generating correct traces with a backtracking algorithm instead of relying on the base model’s flawed outputs. It then fine-tuned on those traces and followed up with reinforcement learning, all inside a 20-hour budget. That method isn’t a benchmark hack; it actually uncovers the underlying algorithm, uses it to create diverse, high-quality data (three times more than Opus 4.8) and distills real reasoning into the base model. The best Fable 5 run used 512 rollouts per step over 95 steps plus supervised fine-tuning on 6,000 balanced boards.
Beyond trace generation, Fable 5 improved calibration, time use, data volume and recovery from errors. Its self-evaluation overconfidence was only 1.2× (versus 4.9× for Opus 4.8), and it consumed nearly the entire 20 hours instead of quitting early. The richer, more varied dataset covered harder boards, giving the base model a stronger training signal. These changes moved FrogsGame from “agents know what to do but mess up details” to a reliable, automated loop for crafting better puzzle-solving models.
Questions about this article
No questions yet.