Smart people are spending hours every day talking to AI models like Claude, GPT, and Gemini—and those systems can sell you on ideas that aren’t actually true. Unlike Google’s search results, these AIs aren’t just pulling facts from humanity’s collective knowledge. They stitch together “highly probable” word sequences based on training data and reward signals, so they sound convincing even when they’re wrong. Their polished, confident tone makes messy scientific debates look amateurish by comparison. Our best truth-finding tools are collective institutions—peer-reviewed science, prediction markets like Metaculus, investigative journalism—where disagreement and complexity are features, not bugs. An AI scientist could be better at factual accuracy than today’s chatbots, but it would still flatten the messy middle: values, social norms, artistic judgment. Objective facts about pi or Schrödinger’s equation sit at one end of the spectrum; personal memories, ethics, politics and culture occupy the vast and varied middle. A single oracle might nail the ends but erase what happens in between. We risk ceding too much authority to a few closed-door AI projects run by powerful companies. Their designers’ agendas—commercial, political, or ideological—could steer public belief without anyone noticing. Instead of letting chatbots hand out instant answers, we should bolster pluralistic processes. Community-driven fact checks, systems like “community notes” and experimental platforms (the so-called “epistack”) can structure discourse with AI assistance while preserving debate, transparency, and redress. AI will keep improving, but “better” in accuracy doesn’t automatically mean better for society’s search for truth. No single model can capture the full human experience—our art, our values, our cumulative judgments. We need tools that support, rather than replace, our democratic and scientific traditions. Otherwise we’ll trade healthy skepticism for blind faith in the next shiny oracle.