What breaks when you point an analytics agent at your dbt project

Ask an analytics agent a question and it will give you an answer. Whether it is right depends on context the raw warehouse does not carry: what each column means, which table to trust, how a number is defined. Most teams start with the context they already have: the dbt project’s descriptions, tests, and models. How much of it can an agent rely on?

To get concrete, I ran an agent-readiness audit I built, an open-source tool, on 13 public dbt projects covering 5,284 models. I was looking for places where an agent could not establish an answer from the repository alone.

They range from a public mirror of GitLab’s analytics project (a 2024 snapshot, not their current repo), to Mattermost’s data warehouse, to the live one California’s Department of Transportation runs for its public transit data, plus packages many teams run in production like Stripe, GA4, and Snowplow. The full list is at the end. Several are mature, extensively documented projects built by experienced teams. These patterns are not confined to neglected repositories.

The audit reads a project’s dbt source and docs: the model SQL, the schema files, the doc blocks behind {{ doc() }}, and the READMEs an agent would read alongside them. It flags mismatches between names, descriptions, tests, joins, and grain. For these public projects I ran it on source alone: no dbt compile, no compiled manifest, no warehouse access. Those limits matter, and I come back to them at the end. Each example below links to a pinned commit you can open yourself.

This is a source-level case study, not an agent benchmark or prevalence estimate. It identifies assumptions the repository leaves unresolved. It does not prove that every agent will make them.

The problems were not exotic. They were ordinary metadata gaps that become dangerous when an agent treats them as settled facts.

The same name, two different concepts

This is the cleanest example of concept ambiguity. One column name means two different things, and the metadata says so out loud in two different models.

Mattermost’s warehouse tracks per-server product usage, and count_registered_users is one of its headline metrics. It appears in seven models across the intermediate, mart, and report layers. Two of them are fact tables an agent would reach for first, and their descriptions are one word apart.

fct_active_users (_product__models.yml):

- name: count_registered_users
  description: Total number of users, including deleted users. Reported by mattermost server.

fct_board_activity (_product__board__models.yml):

- name: count_registered_users
  description: Total number of users, excluding deleted users. Reported by mattermost server.

Including, excluding. Same column name, same sentence, opposite populations. The descriptions expose the difference. They do not establish which table owns the concept or whether the measures are comparable. An agent that retrieves one description, or trusts the shared name, can compare different populations.

It gets worse one layer down. int_server_active_days_spined, the model that feeds fct_active_users, is documented as including deleted users, and its SQL coalesces to a third field when both registered-user sources are empty (int_server_active_days_spined.sql):

coalesce(activity.count_registered_users, legacy_activity.count_registered_users, d.count_users,  0) as count_registered_users

count_users is a different field, and nothing here establishes that it preserves the same deleted-user semantics. The fallback’s meaning is undocumented.

The join whose cardinality you cannot prove

A one-to-many join is not automatically wrong. It becomes dangerous when the model metadata declares one grain, the SQL joins at another, and nothing states whether the expansion is intentional.

Cal-ITP, California’s public-transit data platform, has this shape in the models that track how often transit agencies change their published routes. The upstream model is explicit about its grain: int_gtfs_quality__organization_dataset_map is tested unique on the tuple (date, organization_key, gtfs_dataset_key) (_int_gtfs_quality.yml):

data_tests:
  - dbt_utils.unique_combination_of_columns:
      arguments:
        combination_of_columns:
          - date
          - organization_key
          - gtfs_dataset_key

Downstream it is joined on a different, coarser key: the dataset key is dropped, and organization_key is replaced by organization_source_record_id (fct_monthly_route_id_changes.sql):

LEFT JOIN organization_dataset_map AS orgs ON (reports_index.date_start = orgs.date)
    AND (reports_index.organization_source_record_id = orgs.organization_source_record_id)

The test guarantees uniqueness for (date, organization_key, gtfs_dataset_key). It says nothing directly about uniqueness for (date, organization_source_record_id), the key used by the downstream join. Multiple datasets can therefore match one report row. That is enough to flag the join, but not to prove an overcount: the next step expands each dataset into its routes, which may be intentional. Proving a bug needs duplicated logical routes in the warehouse or an explicit grain contract that the output violates. The source-level finding is narrower and still actionable: no uniqueness test covers the key the join actually uses.

The description points at the wrong column

The last two examples leave meaning or cardinality unresolved. This one is a direct metadata error. An agent often picks columns by their descriptions, and when two columns carry the same description, nothing holds them apart.

GitLab’s headcount report counts separations two ways, voluntary and involuntary, the difference between someone quitting and being let go. Both columns carry the same description (schema.yml):

- name: rolling_12_month_voluntary_separations
  description: Provides the total number of the employees separated voluntarily for the current month and previous 11 months.
- name: rolling_12_month_involuntary_separations
  description: Provides the total number of the employees separated voluntarily for the current month and previous 11 months.

The description is wrong. It defines involuntary separation as voluntary. Ask how many people were let go last quarter, and the metadata points the agent at the opposite concept.

Cal-ITP does the same in its transit metrics. One model exposes both a count, n_tu_trips, and a ratio, pct_tu_trips, of trips carrying a real-time feed (tu is trip updates), and documents both with the count’s description (_mart_gtfs_fcts.yml):

- name: n_tu_trips
  description: '{{ doc("column_n_tu_trips") }}'
- name: pct_tu_trips
  description: '{{ doc("column_n_tu_trips") }}'

The ratio points to the count’s definition. Ask what share of trips had a real-time feed, a figure that goes into public service-quality reporting, and the metadata can return a raw number where the question requires a percentage.

The fix is one line per column. Finding them is the problem. Both projects can look documented at a glance, the descriptions are right there, and nothing flags that they point at the wrong column. Manual review does not scale across thousands of models, and the agent cannot tell which column is right from the descriptions alone.

Documented is not the same as grounded

Being documented hides where the gaps are. In Mattermost, the entire intermediate/sales/hightouch directory, four models that compute ARR and seat counts (all four), has no schema file at all. Revenue-related logic is undocumented, while the project can still look well documented from a distance. Cal-ITP looks well documented too, until you find one description applied verbatim to nineteen quality flags in a single model, dim_annual_service_mode_time_periods, through a YAML anchor (_mart_ntd.yml):

- &questionable_data
  name: questionable_data
  description: '{{ doc("ntd_questionable_data") }}'
...
- <<: *questionable_data
  name: mode_voms_questionable
- <<: *questionable_data
  name: vehicle_miles_questionable
- <<: *questionable_data
  name: deadhead_miles_questionable

Nineteen columns, all counted as documented, all sharing one sentence.

A semantic layer addresses part of this problem. MetricFlow defines metrics, entities, and dimensions, with entities as the join keys between semantic models. It still governs only what has been modeled. Two of these projects define their metrics while leaving ordinary dbt columns undocumented: zero descriptions across 63 columns in one, zero across 137 in the other. In one, a community marketing-analytics project, a model spells out its grain trap, then gives its columns tests and nothing else (schema.yml):

- name: fct_channel_performance
  description: >
    Aggregated spend, revenue, orders, ROAS, and CAC per (date × channel).
    Grain is daily × channel after the time-dimension refactor — channel
    alone is not unique.
  columns:
    - name: date
      tests:
        - not_null
    - name: channel
      tests:
        - not_null
    - name: total_spend
      tests:
        - not_null

The semantic layer can tell the agent what revenue is and how governed dimensions join to it. The moment an agent writes SQL against columns outside that modeled surface, it is back to whatever context the underlying models provide. A semantic layer reduces ambiguity inside its boundary. It does not make arbitrary warehouse SQL safe by itself.

What a source read cannot catch

Two things this kind of audit cannot see, both worth saying plainly.

First, the loud failures. Broken references and columns that no model produces look like the easy wins, because they error. Read from source alone, they were by far the least reliable findings, and the reason is structural: a parser reading raw SQL sees Jinja, not the columns Jinja will produce. On GitLab, thirty-five of them, every one I checked, were false positives. A date part read as a column inside DATEADD. Columns generated by macros and Jinja loops. Package models that resolve once you run dbt deps. In the Stripe package, every flagged column traced to one macro that builds its column list at compile time (stg_stripe__balance_transaction.sql):

{{
    fivetran_utils.fill_staging_columns(
        source_columns=adapter.get_columns_in_relation(ref('stg_stripe__balance_transaction_tmp')),
        staging_columns=get_balance_transaction_columns()
    )
}}

The tool now suppresses those classes by construction rather than guessing at them: it recognizes date-part keywords as units, treats the Fivetran column-generating macros as unresolvable and skips the model, and holds back unresolved refs when a project declares packages it has not installed. That took GitLab’s thirty-five flags to zero and Stripe’s two hundred and eighteen to zero for these known classes. The tradeoff is explicit: skipped models are blind spots, not clean bills of health.

Reading columns created by warehouse-aware macros requires compiled SQL. A parsed manifest.json describes the project without including compiled SQL for every node. dbt compile needs a data platform connection, because generating compiled SQL for many models means running introspective queries against it. In practice, an audit can consume compiled artifacts produced by CI instead of connecting to the warehouse itself. Source-only analysis cannot recover what those artifacts never expose.

Second, the data. A source read sees the code, not the rows. The messiest project I ran, a RevOps warehouse its author had deliberately seeded with duplicate domains and duplicate emails, came back nearly clean, because the mess was in the data and the code that handles it was correct.

Everything earlier is what a source read can inspect or flag: descriptions, declared grain, and join keys. It can identify an assumption that lacks support. It cannot always decide whether the data makes that assumption true.

So what is good enough

Across these projects the risks rhyme. The meaning an agent needs is missing, copied from a column where it does not belong, or split across definitions with no declared owner. Documentation coverage does not resolve those problems.

For an analytics agent, good enough means the critical path has two contracts:

• Semantic: what each critical column means, what grain each model has, which table owns each business concept, and which joins are valid.
• Verification: tests on the keys actually used, compiled artifacts where source is dynamic, warehouse checks for data-dependent claims, and a way for the agent to abstain when evidence conflicts.

If those facts live in one correct, reachable place, the agent has something to stand on. If they are missing, duplicated, stale, or contradicted by the SQL, documentation volume is not enough. That layer of grounded, reachable meaning is what many dbt projects, including strong ones, have not finished building.

The audit is an open-source tool that runs on a dbt project’s source, no warehouse or credentials needed: github.com/GetCassis/dbt-agent-readiness. If you run it on yours, I would like to compare notes on what it finds.

Appendix: the projects

Every example above is quoted verbatim from a public repository, under its open-source license, linked to a pinned commit.

Model counts are from the audit’s own inventory pass. The jaffle_corp repo holds several dbt projects; the count is its platform project.

The GitLab project is a public mirror of GitLab’s former internal analytics repo, MIT-licensed and last active in 2024, not GitLab’s current repository. The actively maintained Cal-ITP project shows the same patterns, which is why it appears alongside it. Public projects tend to be tidier than the private warehouses they mirror, because the scrutiny is external.