What breaks when you point an analytics agent at your dbt project

I built the dbt agent-readiness audit to find out what an analytics agent could learn from a dbt project, and where it would have to guess. The tool started as a Claude Code skill that I shared with practitioners. I then ran it on 13 public projects covering 5,284 models and manually checked every finding used in this article against a pinned commit.

The audit reads model SQL, schema files, the doc blocks behind {{ doc() }}, and repository READMEs. It compares what the models do with what their names, descriptions, tests, and declared grain say they do. For these runs I supplied no warehouse access and no compiled SQL.

The projects include a 2024 public mirror of GitLab’s former analytics repository, Mattermost’s data warehouse, Cal-ITP’s live public transit project, and packages such as Stripe, GA4, and Snowplow. Several are mature and extensively documented. I could inspect them only because their maintainers chose to work in public.

I checked the source for incorrect metadata and assumptions it could not settle. I did not test how often agents make these mistakes, and these 13 projects cannot tell us how common the problems are.

One name, two populations

Mattermost’s warehouse tracks per-server product usage, and count_registered_users appears in seven models across the intermediate, mart, and report layers. Two fact tables define it one word apart.

fct_active_users (_product__models.yml):

- name: count_registered_users
  description: Total number of users, including deleted users. Reported by mattermost server.

fct_board_activity (_product__board__models.yml):

- name: count_registered_users
  description: Total number of users, excluding deleted users. Reported by mattermost server.

Including, excluding. Both descriptions may be correct for their tables. But if you ask for registered users without specifying whether to count deleted users, which table should the agent choose? The repository does not name a default or explain whether the two counts are comparable.

The model feeding fct_active_users, int_server_active_days_spined, adds another unresolved case. It is documented as including deleted users, and its SQL falls back to a third field when both registered-user sources are empty (int_server_active_days_spined.sql):

coalesce(activity.count_registered_users, legacy_activity.count_registered_users, d.count_users,  0) as count_registered_users

count_users is a different field. The repository does not say whether that fallback includes deleted users.

A uniqueness test on a different key

Cal-ITP, California’s public transit data platform, tests int_gtfs_quality__organization_dataset_map as unique on (date, organization_key, gtfs_dataset_key) (_int_gtfs_quality.yml):

data_tests:
  - dbt_utils.unique_combination_of_columns:
      arguments:
        combination_of_columns:
          - date
          - organization_key
          - gtfs_dataset_key

The downstream route-change model joins it on (date, organization_source_record_id) instead (fct_monthly_route_id_changes.sql):

LEFT JOIN organization_dataset_map AS orgs ON (reports_index.date_start = orgs.date)
    AND (reports_index.organization_source_record_id = orgs.organization_source_record_id)

Even if this test passes, multiple datasets could still match one report row: the join uses a different key. The next step expands datasets into routes, and that may be intentional. Proving an overcount would require duplicated logical routes in the warehouse or an explicit output-grain contract that this model violates. What I could establish is that no declared uniqueness guarantee covers the key used in this join.

The description points at the wrong column

GitLab’s headcount report counts separations two ways, voluntary and involuntary, the difference between someone quitting and being let go. Both columns carry the same description (schema.yml):

- name: rolling_12_month_voluntary_separations
  description: Provides the total number of the employees separated voluntarily for the current month and previous 11 months.
- name: rolling_12_month_involuntary_separations
  description: Provides the total number of the employees separated voluntarily for the current month and previous 11 months.

The second description is wrong. It defines involuntary separation as voluntary, so an agent looking for the column about people who were let go gets conflicting clues from its name and description.

Cal-ITP does the same in its transit metrics. One model exposes both a count, n_tu_trips, and a ratio, pct_tu_trips, of trips carrying a real-time feed (tu is trip updates), and documents both with the count’s description (_mart_gtfs_fcts.yml):

- name: n_tu_trips
  description: '{{ doc("column_n_tu_trips") }}'
- name: pct_tu_trips
  description: '{{ doc("column_n_tu_trips") }}'

An agent looking for the share of trips with a real-time feed gets a count’s definition on the ratio column.

Both errors are one-line fixes. They are hard to spot because the descriptions exist and look complete until two neighboring fields are compared.

Coverage hides where meaning is missing

Mattermost’s intermediate/sales/hightouch directory contains four models that compute ARR and seat counts (all four). The directory has no schema file, so that revenue-related logic carries no model or column descriptions.

Cal-ITP shows a different coverage problem. In dim_annual_service_mode_time_periods, nineteen quality flags inherit one description through a YAML anchor (_mart_ntd.yml):

- &questionable_data
  name: questionable_data
  description: '{{ doc("ntd_questionable_data") }}'
...
- <<: *questionable_data
  name: mode_voms_questionable
- <<: *questionable_data
  name: vehicle_miles_questionable
- <<: *questionable_data
  name: deadhead_miles_questionable

All nineteen columns count as documented. They still share one sentence.

Some projects also define a semantic layer. MetricFlow defines metrics, entities, and dimensions, with entities as the join keys between semantic models. Two projects in the audit define that layer while leaving ordinary dbt columns undocumented: zero descriptions across 63 columns in full-funnel and zero across 137 in jaffle_corp. In full-funnel, the model description states its grain, then the columns carry tests and no descriptions (schema.yml):

- name: fct_channel_performance
  description: >
    Aggregated spend, revenue, orders, ROAS, and CAC per (date × channel).
    Grain is daily × channel after the time-dimension refactor — channel
    alone is not unique.
  columns:
    - name: date
      tests:
        - not_null
    - name: channel
      tests:
        - not_null
    - name: total_spend
      tests:
        - not_null

MetricFlow gives the agent defined measures, dimensions, and entity joins. If the question needs a column outside those definitions, though, the agent is back to reading the underlying dbt models. Those columns still need descriptions of their own.

What source alone cannot establish

My audit got its most obvious findings wrong. I manually checked 35 GitLab flags for broken references or undefined columns. All 35 were false positives.

The parser had treated a date unit inside DATEADD as a column. It could not see columns generated by macros and Jinja loops. It flagged package models whose references resolve after dbt deps. In the Stripe package, 218 flags traced to one macro that builds its column list at compile time (stg_stripe__balance_transaction.sql):

{{
    fivetran_utils.fill_staging_columns(
        source_columns=adapter.get_columns_in_relation(ref('stg_stripe__balance_transaction_tmp')),
        staging_columns=get_balance_transaction_columns()
    )
}}

I changed the tool after those checks. It now recognizes date-part keywords as units, treats Fivetran’s column-generating macros as unresolvable, skips those models, and holds back unresolved references when declared packages are absent. Those 35 GitLab flags disappeared. Stripe fell from 218 to zero. A skipped model remains a blind spot, not evidence that the project is clean.

Warehouse-aware macros require compiled SQL. A parsed manifest.json describes the project, but it does not include compiled SQL for every node. dbt compile can require a data platform connection when macros run introspective queries. An audit can instead consume compiled artifacts produced by CI. If neither is available, raw source cannot expose the generated columns.

Source also sees code, not rows. The messiest project in the set was a RevOps warehouse whose author had deliberately seeded duplicate domains and duplicate emails. It came back nearly clean because the mess lived in the data, while the code handling it was correct.

I could check descriptions against SQL and compare join keys with declared tests. To find out whether an untested join actually duplicates rows, I would need the warehouse.

What the agent still needs

Start with the models the agent will query. Define what the critical columns mean, state the grain, and identify which table to use for each business concept. Check the joins on the keys they actually use. Where macros generate SQL, make the compiled artifacts available; where correctness depends on the rows, run the warehouse checks.

Not every model needs the same attention. But when the agent reaches conflicting definitions or an assumption the repository cannot settle, it needs to ask rather than choose for itself.

The audit is open source and runs on a dbt project’s source without warehouse credentials: github.com/GetCassis/dbt-agent-readiness. If you run it on yours, I’d like to compare notes on what it finds.

Appendix: the projects

Every example above is quoted verbatim from a public repository, under its open-source license, linked to a pinned commit.

Model counts are from the audit’s own inventory pass. The jaffle_corp repo holds several dbt projects; the count is its platform project.

The GitLab project is a public mirror of GitLab’s former internal analytics repo, MIT-licensed and last active in 2024, not GitLab’s current repository. Cal-ITP is actively maintained, which is why it appears alongside the older mirror.