The article examines why translating natural language into SQL for live data warehouses remains a tough problem. It traces the rise of ad-hoc queries in 1990s‐era historical data stores and shows how schema complexity and performance needs outpace current text-to-SQL systems.

• Benchmarks like Spider (max 200 tables, single database per task) don't reflect real enterprise warehouses with hundreds of tables and thousands of columns spanning dozens of schemas
• Production metadata is messy: cryptic/inconsistent column naming, typos, abbreviations, and mixed-language query logs undermine model performance
• Vendor-specific SQL extensions (Oracle's LISTAGG, PostgreSQL's JSONB operators) and user-defined types routinely break academic text-to-SQL models
• The authors call for shared benchmarks built from anonymized real industry schemas/workloads and argue LLMs need integration with metadata services and human feedback loops before replacing analysts in critical settings

This article reruns a 2023 benchmark with the latest LLMs, comparing direct SQL generation against querying through a structured dbt Semantic Layer. It finds that while text-to-SQL accuracy has jumped, a modeled Semantic Layer still delivers near-perfect, deterministic results for covered queries, making it ideal for complex or critical use cases.

• Top 2026 LLMs (Opus 4.6, Sonnet 4.6, GPT-5.3 Codex, GPT-5.2) now hit 80%+ accuracy on raw text-to-SQL, up sharply from the GPT-4 era's ~60%
• A fully modeled dbt Semantic Layer still reaches near-100% accuracy on in-scope queries, versus ~95% for a minimal semantic layer and ~60-80% for plain text-to-SQL
• Data modeling matters more than model choice: basic modeling boosted accuracy 15-20 points, while swapping between top-tier LLMs or reasoning settings only shifted results by single digits
• Semantic layers pay off for complex, messy, enterprise-scale data, while raw LLM text-to-SQL remains a fine low-setup option for one-off or small-dataset analysis

The article discusses the shortcomings of achieving high accuracy in Text-to-SQL systems, emphasizing that 90% accuracy is insufficient for enterprise applications. It highlights the need for rigorous evaluation frameworks, like Spider 2.0, to ensure reliability and trust in AI-driven analytics.

• 90% accuracy still means 1 in 10 SQL queries is wrong, which is enough to destroy user trust and stall enterprise adoption.
• Standard benchmarks like Spider 1.0 don't reflect enterprise reality; Spider 2.0 tests against schemas with 3,000+ columns to simulate real-world messiness.
• Simple accuracy scores are misleading — metrics like Execution Accuracy (EX) and Soft-F1 are needed to actually capture whether generated SQL is reliable.