This newsletter covers major developments across data and AI, including DoorDash’s new Write-Ahead Intent Log for scalable CDC, OpenAI’s Kepler AI analyst for massive data insights, and Lyft’s Metric Semantic Layer for unified business metrics. It also reviews ClickHouse ingestion optimizations, DuckDB’s latest release and agent use cases, Databricks’ Lakehouse//RT and LTAP pushes, plus tools for self-healing pipelines and data quality monitoring.

Data lakehouses combine low-cost, flexible storage with warehouse-style governance to power enterprise AI. Companies like DocuSign and Lemongrass use them to feed and train AI agents, but impose strict security reviews, access controls and audit trails. Vendors are adding vector indexing, MCP connectivity and semantic layers to ensure agents grasp business context and operate safely.

Rill lets you define BI workflows—models, metrics, dashboards and access policies—in YAML and SQL, and runs queries on engines like ClickHouse or DuckDB. It adds conversational and agent-driven interfaces for natural-language querying and automated dashboard generation, with real-time performance, embeddable dashboards, and Git-backed deployments.

Rill’s Metrics SQL lets you define business metrics once and query them using plain SQL across dashboards, notebooks, and AI agents. It compiles metric views into optimized OLAP queries, handling grouping, filters, time functions, and security automatically.

This article reruns a 2023 benchmark with the latest LLMs, comparing direct SQL generation against querying through a structured dbt Semantic Layer. It finds that while text-to-SQL accuracy has jumped, a modeled Semantic Layer still delivers near-perfect, deterministic results for covered queries, making it ideal for complex or critical use cases.