This newsletter covers major developments across data and AI, including DoorDash’s new Write-Ahead Intent Log for scalable CDC, OpenAI’s Kepler AI analyst for massive data insights, and Lyft’s Metric Semantic Layer for unified business metrics. It also reviews ClickHouse ingestion optimizations, DuckDB’s latest release and agent use cases, Databricks’ Lakehouse//RT and LTAP pushes, plus tools for self-healing pipelines and data quality monitoring.

• DoorDash replaced Debezium with a Write-Ahead Intent Log (WAIL) that writes mutation intents to Kafka and the DB, enabling faster recovery and much higher CDC throughput at Cassandra scale
• MotherDuck claims a $2.40/hour DuckDB instance beats a $64/hour Snowflake 2XL cluster by ~5x on ClickBench, with median query latencies of 3ms, positioning DuckDB for fleets of lightweight agent workloads
• Lyft's Metric Semantic Layer assigns dual business/operational ownership to "Golden Metrics" defined in versioned JSON, auto-propagating changes across dashboards, APIs, and AI agents to prevent divergent calculations
• OpenAI's Kepler analyst now operates over 600 PB of data using Codex-based crawlers and AST-normalized LLM grading to infer lineage, freshness, and hidden semantics automatically

Rill lets you define BI workflows—models, metrics, dashboards and access policies—in YAML and SQL, and runs queries on engines like ClickHouse or DuckDB. It adds conversational and agent-driven interfaces for natural-language querying and automated dashboard generation, with real-time performance, embeddable dashboards, and Git-backed deployments.

• Full BI stack (models, metrics, dashboards, agent config) spins up from three prompts via `rill init`, then `rill start` gives an instant local UI.
• Metrics-as-code approach: SQL models plus YAML-defined dimensions/measures let Rill generate queries on the fly, returning sub-second results even at billions of rows.
• Same semantic layer powers both natural-language conversational BI and traditional SQL-based analyst workflows.
• Deployment is Git/CI-CD native via `rill deploy`, with dashboards embeddable and metrics exposed through a REST API.