A blank beats a guess: assembling context for analytics agents

An analytics agent answers whatever you ask it, from whatever it can see. Whether the answer is right depends on context: what the tables mean, how they join, which definitions the company actually uses, where the traps are. How that context should be organized once you have it is its own subject; this guide is about where its content comes from. The hard question is where to start. On every warehouse we have looked at, much of that context already existed, scattered across the schema, the dbt project, the dashboards and a glossary someone once wrote, and nothing was reading it. The first version is less a writing project than a sorting one. Recover what the stack already says, keep the evidence attached, and turn the rest into questions.

This guide is the order we collect it in, measured on several production warehouses and one project we can name in full: GitLab’s analytics repo, 2,921 dbt models. The steps below are packaged as an open-source kit you can run on your own project.

One expectation to set first. Nobody has documented their warehouse completely, or perfectly. Read everything an organization ever wrote, the code, the docs, the Slack threads, the meeting transcripts, and the perfect context still is not in there. The target is the most accurate and reliable first version you can assemble, a base to enrich, expand and fix when needed.

The order of operations, short version:

1. Scope a first slice on usage, not on the warehouse.
2. Extract everything the code proves, deterministically. No LLM involved.
3. Let an LLM verify and draft against that proof, every sentence traced to evidence. A blank beats a guess.
4. Free-form docs enter under a binding: a human key, a question, or a SQL check.
5. Everything still open ships as a question to a person, with the assumption attached.

The output of the five is a reviewable package: the scoped tables with their grain, columns with provenance-tagged descriptions, joins and metrics with the evidence behind them, domain notes, and the list of named assumptions. What follows is each step, with its numbers.

What we’re assuming

Everything below follows from four things we believe today.

• Most of the context an agent needs already exists in the stack, written for other readers. → The first version is a sorting job, not a writing one.
• A wrong answer costs more than a missing one, because nobody can see it. → A blank beats a guess, at every step.
• The reader is an analytics agent picking tables and writing SQL, not the next engineer reading the pipeline. → The output is organized around the questions people ask, not mirrored from the dbt project.
• Bootstrapping moves your source of truth: the context becomes what agents read, and regenerating it from dbt docs on every change is not realistic. → It has to carry the evidence its sources gave it, and be maintained from there.

Start smaller than the warehouse

Eventually you want the agent to cover as much of the warehouse as people ask questions about. You still start small. A first slice keeps the review load humane, shows you what works and what does not before you scale the method, and produces something usable in days rather than years. The honest signal for picking it is usage: query logs name the tables people actually read.

On your own warehouse the recipe is short:

• Pull about three months of query history (access_history on Snowflake, INFORMATION_SCHEMA.JOBS on BigQuery) and drop the scheduled jobs first, or the logs measure your pipelines rather than your people. Mind the BI service account: dropping it naively also drops every read your BI tool proxies, so pair the warehouse logs with the BI tool’s own view counts.
• Rank tables by how many distinct people read them. The slice to start with is usually sitting at the top already clustered: the tables one team actually reads, joined to each other inside the same queries.
• Take that cluster plus the tables the logs show it joining to, and leave out anything nobody queried in the window, whatever its name suggests. That list is your first slice.

One caveat on the signal: usage has a chicken-and-egg side. A table nobody queries may be dead, unknown because nothing ever documented it, or simply seasonal, the quarter-end tables sleep eleven weeks out of twelve. And usage tells you where trust must be built first, not where every future question will land. If you do not know where to start, it is a start, and later slices are where the neglected tables get their turn.

We could not run this on GitLab: a public repo has no query history. So we scoped on what it offers instead: 13 declared dbt exposures, dependency centrality across the consumption models, then the closure around the cluster that topped both, revenue. (We read the project through the public fork yogeshkrishnan/analytics at a58c84f6; the original is no longer publicly readable.) The GitLab project resolves to 2,183 tables; the closure gave 50 tables and 2,641 columns, and the slice is not timid: those 50 carry the ARR reporting surface of a decade-old production project. The in/out call stays human: the closure also pulled in dim_marketing_contact, and a marketing contact is not a revenue entity, so it was cut. The next slice repeats the same steps.

Take what the code proves before an LLM runs

Every source in a data stack either proves something or claims something. Code proves: a dbt model’s SQL declares its joins, its tests encode the grain it intends, its macros are the computation itself. Prose claims: descriptions, glossaries and wikis carry the richest meaning and no guarantee it is current.

By proof, I mean something narrow: what the code declares at the commit you read, not that the business logic is right, and not that the repo matches production. This run never touched the warehouse, so every number below is about what the project declares. On your own stack, make the deployed schema the first cross-check. Extract that proof first, because none of it needs an LLM.

We have tested every extraction path in this table on a real project. The query-log recommendation follows from what query history records directly: observed reads and joins. The dashboard row works the same way when you have a BI tool: every chart carries the SQL it runs, so an export of the definitions (Looker models, Metabase cards) hands you observed join pairs and the formulas your metrics are actually computed with, plus a popularity signal for scoping. The titles are claims; the SQL underneath is the proof. What a dbt project’s own metadata can and cannot establish on its own is measured separately, across 13 public projects and 5,284 models.

On the GitLab scope, the deterministic pass took about two minutes and no LLM calls. It produced:

• A 39,271-column universe from the repo alone, 22,004 of them never declared in any yml, only recoverable by parsing the SQL. (A project that parses hands you this in manifest.json; this one wants dbt deps and its Snowflake environment variables even to parse, so we read the SQL raw.)

• 876 descriptions carried into the context tree we were assembling, GitLab’s own words in GitLab’s own places, from the project’s own yml and expanded doc blocks. 65 more were rejected for restating the column name.

  What that looks like on one column of mart_arr:

  IS_ARPU:
    description: Flag to identify what counts towards Annual Revenue Per User (ARPU).
      It works by excluding `Storage` product_tier and `EDU` product_rate_plan.
    description_source: repo_text

  The decision at this stage is binary: keep the owner’s words with their provenance, unless they restate the column name (AMENDMENT_NAME: “The name of the amendment in Zuora” is the kind the gate rejects).

• The other two thirds of the scope left for anything smarter than a script.

Two client numbers say why this stage comes first. On a 97-table production warehouse, a name-keyed glossary the team handed over prefilled 1,052 definitions, covering 76% of the documented columns. On another project, 1,491 doc blocks were shipping as zero descriptions because nothing had ever expanded the {{ doc() }} references: the documentation existed and was invisible. Both recoveries are file parsing. Neither needs judgment.

This order exists because we paid for the alternative: an earlier version of the pipeline ran an agent stage over the prose first, and it spent 1.16 million tokens re-deriving what the deterministic pass extracts for free. That stage no longer exists. Measure what the scripts can reach before any agent runs: it costs seconds and separates delivery problems from judgment problems.

Let the LLM verify and draft, never invent

What is left for an LLM is narrower than “point AI at your metadata” suggests, and it runs under one rule: every sentence it writes must trace to something on disk. Three jobs qualify:

• Proposing how the tables group into domains.
• Checking prose claims against the table’s own SQL.
• Drafting descriptions where the SQL supports them, leaving the column blank where it does not.

Drafting here means turning evidence already present in the code into prose. It does not make the LLM a source.

Neither a blank nor an unverifiable claim vanishes: the run files each one as an open question, written with the assumption it made in the meantime and what breaks if that assumption is wrong. The last two steps of this guide run on that list.

The rule exists because an LLM arrives already knowing what ARR usually means, and it will happily draft fluent, plausible meaning from a column name alone. Treat that knowledge as the main failure mode, not as a source. On this run it expanded is_arpu as average revenue per user; GitLab’s own documentation says Annual Revenue Per User, excluding the Storage tier and the EDU rate plan.

A judge pass caught it, along with three more contradictions across the 38 verdicts it returned, checking each claim against the table’s SQL and the project’s own text. The claims judged were not a random sample: the audit targeted drafted claims that asserted something checkable, an exclusion, a derivation, a formula. Two of the four contradictions traced to a rule we ourselves had written into the domain grouping: the judge does not care who wrote the claim. Verification is the half that makes drafting usable.

Drafting from proof looks like this. The Delta ARR movement categories are computed by a 17-line macro:

CASE
  WHEN {{ row_number }} = 1 THEN 'New'
  WHEN {{ arr }} = 0 AND {{ previous_arr }} > 0 THEN 'Churn'
  WHEN {{ arr }} < {{ previous_arr }} AND {{ arr }} > 0 THEN 'Contraction'
  WHEN {{ arr }} > {{ previous_arr }} AND {{ row_number }} > 1 THEN 'Expansion'
  WHEN {{ arr }} = {{ previous_arr }} THEN 'No Impact'
  ELSE NULL
END

What the LLM drafted from the body, stamped authored:

Category of this parent-account month's ARR movement, computed by the
type_of_arr_change macro. Exactly one of 'New' (the first period for this
grain, whatever the ARR comparison says), 'Churn' (ARR is 0 and the prior
period's was above 0), [...] 'No Impact' (ARR equal to the prior period's);
NULL when none of those hold.

Every clause traces to a line of the CASE, down to the capitalized literals, which is what an agent needs: filtering on 'new' returns an empty set.

The books, for anyone auditing: the tree ended at 2,573 of 2,641 columns covered, 871 in GitLab’s own words, 1,702 drafted, 68 deliberately blank. The prefill stage had carried 876 of the owner’s texts; the delta is two malformed column names dropped from the scope, one of them prefilled, and four owner texts corrected against the SQL and re-stamped as drafted.

Provenance survives into the output: the owner’s own words stay stamped as the owner’s, drafted text stays stamped as drafted, and nothing the LLM wrote is allowed to corroborate a metric.

Checking prose against SQL also produces a byproduct you did not ask for: defects in the documentation you already had. Two contradictions in 200 checked claims on one client project, eight on another. A metric that summed a column absent from the deployed warehouse. In the other direction, documentation that saves an agent from the data: GitLab documents one opportunity mart’s ARR column as a WIP Salesforce formula field that returns 0, which is exactly what stops an agent from summing zeros and calling it revenue.

Bind free-form docs before you read them

Free-form documentation comes last, under the strictest rule: prose enters the context through one of three bindings.

• A person keyed it to columns.
• A question is asking for it.
• SQL can check it.

What an LLM must never do is invent the binding. And a binding decides where text may attach, not whether it is true: a handed-over glossary can be stale or wrong, so its text gets checked against the SQL like any other prose.

One corpus, used both ways, shows why. The tempting move is to aim it at the columns: extract definitions, match them to names, fill the blanks. We tried exactly that with GitLab’s public handbook, 4,736 pages and 46.9 million characters of it. Extraction produced 647 definition-shaped entries. Nine matched a scoped column name. Eight prefilled a description. Zero were correct.

The Delta ARR mart’s DELIVERY column, which distinguishes SaaS from self-managed revenue, received prose from an architecture document about package download modes, and the pipeline stamped it as the warehouse owner’s own words. That stamp was the deeper defect: the pipeline recorded where the text came from and trusted a classifier for what it was about.

Fixed since: a guessed binding is stamped as a guess and can corroborate nothing; only a dictionary a person handed over keeps owner status. The client glossary from earlier worked because a person made the name-to-column binding; the handbook’s classifier guessed it. The text was not wrong in itself: it had meaning, for its own referent, bound to the wrong one.

The same corpus earned its place when we pointed it at the run’s open questions instead of its columns: retrieval reduced 6.8 million tokens of handbook to about 5,500 tokens of candidate passages in 2.3 seconds, then one blind agent per question judged the candidates. One question came back answered:

What timezone are timestamps stored in?
Answered, from handbook/enterprise-data/platform/_index.md:873:
"All timestamp data in the warehouse should be stored in UTC",
with one named exception, the pacific-time _pt_id date keys.

A citation for a person to confirm, never an auto-apply. Four refusals, all correct. The strongest refusal found a CRM field carrying the categories New / Growth / Contraction / Churn and rejected it as an upstream classification rather than the mart’s computation, the exact mistake the column path could not avoid. Zero false answers. One caveat: the retrieval half was tuned against these labeled questions; the judging agents were blind.

The unknowns ship as questions, not guesses

Whatever the first four steps did not settle, the LLM does not settle either. Each remaining gap ships as an explicit question to the data owner, carrying the assumption already made and the error if that assumption is wrong. This list, not the coverage number, is the deliverable to read first: our run ended at 97.4% coverage while every dollar metric in scope hung on an unanswered currency question. Here is that question’s tail, as the run shipped it:

Assumed instead: shipped every dollar metric as a plain SUM, on the working
assumption (unverified) that these marts report in one currency.
If that is wrong: every SUM-based dollar metric in this run is silently wrong
by however much non-reporting-currency revenue exists, with no way to detect
it from the data alone.

The run filed 45 questions. Six are blockers, gating 30 tables, columns and metrics between them; two are rulings where an owner picks between two defensible definitions; the other 37 triage into defects found rather than asked, rules not bound to every table they change, and the standard questions every run must answer. Where the blockers’ answers lived:

• One was public and unread. The timezone of every warehouse timestamp sat in a single sentence of the handbook until a question pulled it out.
• One looked public and was not. GitLab’s reporting currency is stated publicly, and the blind judge rightly rejected the page: what a company reports in says nothing about whether a mart’s rows are already converted. The question survived, more precise than before.
• One was in the repo all along. The movement categories of the macro above. The authoring inputs carried the macro call 175 times and its body zero times, so the categories could not be verified until the bodies shipped with the tables that call them.
• Three could not be answered from any corpus that exists outside the company: a mart whose declared columns exclude the ARR its own SQL computes, five reporting tables with no recoverable column list, and the one table that exists because snapshot and live disagree, which carries no documentation at all.

The last group contains the decisions only your team can make, and it is small. Order the list riskiest first, each question carrying its assumption, so the cheap ones are one-word confirmations and the expensive ones get the afternoon they deserve.

The first version is a snapshot

We packaged these steps into a kit: the deterministic stages as scripts, the judgment stages as agent prompts with the verification gates attached, and the decisions as checkpoints that stop the run until a person answers. Your team’s side is the four checkpoints: agree the scope, approve the domain grouping, review the metrics, answer the questions. Review sittings, not implementation time.

The GitLab pass was an afternoon: about two minutes of scripts, under two hours of agent wall clock, just under three million tokens, most of them on a mid-tier model. It runs from a dbt project export, without warehouse access. We have not measured agent answer accuracy before and after on this project, because there is no warehouse to run questions against. What the run demonstrates is the reviewable base and its named unknowns, not downstream answer quality.

Everything above produces an assembled view of the evidence, not a freestanding body of AI-written documentation. It is a snapshot, and it starts aging the day it lands. A new column arrives undescribed. A filter moves two models upstream and a mart quietly changes which companies it counts. The description that was verified against the SQL in August describes the SQL of August.

Bootstrapping also moves your source of truth. What agents read is now this ontology, not the dbt docs and the glossary it was assembled from, and regenerating it from those on every change is not realistic once a hundred people are relying on the answers.

The questions age fastest of all, and they are also where change lands first: the assumption you confirmed at the checkpoint is the one the next schema migration invalidates. Assembling what already exists compresses into an afternoon. Keeping the answers true while the warehouse moves underneath them is a different problem, and nothing in this guide solves it. We ended the run with a reviewed context, two questions settled from material that had been on disk the whole time, and the rest carried as named assumptions.