More on the topic...
Generating detailed summary...
Failed to generate summary. Please try again.
At GitHub, the analytics team built Qubot, an internal Copilot-powered agent that turns plain-language questions into SQL and returns data in seconds. Users ask a question in Slack, VS Code or the Copilot CLI, and Qubot spins up on GitHub’s cloud agent. Results land as a Slack reply and a markdown report in a pull request, which teams can tweak or plug into dashboards. It isn’t meant to replace dashboards; it’s for quick exploratory queries like “Which user cohort shows highest retention?” or “What drove this metric last week?” With zero-maintenance and no setup, even employees who’ve never touched the data warehouse can dive in immediately.
Under the hood, Qubot has three layers: the user interface, a federated context layer, and a query engine. The context layer pulls metadata from three tiers of data—raw events (bronze), conformed facts (silver) and curated business datasets (gold). Product teams feed telemetry schemas for bronze data; the analytics team maintains query examples and mandatory filters for silver; and business owners contribute metric definitions for gold. This context gets loaded at runtime via GitHub’s MCP server and is continually enriched through ETL pipelines and a context-agent that standardizes new contributions via pull requests.
Every update to either context or the agent itself goes through an offline evaluation framework. A suite of test cases—prompts with known SQL answers—runs multiple parallel trials via the GitHub CLI. The framework measures completion rate, accuracy and latency, catching regressions before they reach users. Query execution happens over two engines: Kusto for fast, recent-data exploration and Trino for complex joins and long-term history. Qubot defaults to Kusto, then switches to Trino when needed, so users don’t have to choose.
Since launch, hundreds of GitHub employees have run thousands of queries through Qubot. Questions in the central analytics Slack channels have plummeted, freeing up data analysts for tougher tasks. The team found that well-structured context not only boosted Qubot’s accuracy but also cut answer-time by two-thirds. By centralizing dataset rules and metadata, Qubot turns distributed knowledge into a single source of truth—and gives every product and business team a stake in keeping that knowledge current.
Questions about this article
No questions yet.