I built a benchmark to answer that properly: one frozen ecommerce dataset with five planted data problems, ground truth computed independently, and an LLM agent answering the same 20 business questions twice, once against raw BigQuery tables and once through a governed Cube semantic layer. This first part runs entirely on Google Cloud.
58% vs 43%
of answers passed through the governed layer vs raw tables, same Gemini agent, 60 runs each
14/15 vs 9/15
straightforward lookups answered correctly, the questions leaders assume are safe
22/22
layer correctness checks within 0.5% of ground truth, run without any LLM involved
$10
to run the full 60-answer evaluation on Gemini through Cube ($7 on raw tables)
My role: solo build, covering benchmark design, star schema, semantic layer, agent and evaluation harness.
The question
Teams are connecting LLMs to their warehouses and asking things like "what was revenue last quarter?". The model will always produce a number. The risk is that it produces one just as confidently when "revenue" is ambiguous, when test accounts sit in the orders table, or when a product category was renamed halfway through the year.
The usual advice is to put a semantic layer in front of the model. That is partly right: a semantic layer makes answers consistent, because every question goes through the same definitions. It does not make them correct by itself. The layer is only as correct as the judgment encoded into it, and I wanted to measure that gap instead of asserting it.
How the benchmark works
The planted problems
Four are measurement problems: the business did not change, the data did. One is a real business signal hidden in the noise. A good analyst flags the first kind and explains the second.
| ID | What is wrong in the data | How the governed layer handles it |
|---|---|---|
| P1 | "Revenue" is ambiguous. Cancelled and returned items make gross and net revenue differ substantially. | Deliberately left open: both metrics exist with explicit definitions, to test whether the agent notices the split or quietly picks one. |
| P2 | A category is renamed mid-window. Sweaters becomes Knitwear in October 2025; old sales seem to collapse and new ones appear from nowhere. | A hand-maintained mapping assigns both names to one category family, so the rename is invisible downstream. |
| P3 | Consent tracking loss. From one date, a share of new users and sessions lose their traffic source, so every channel appears to decline. | Missing values become an explicit "Unattributed" bucket. The true source is unrecoverable; the layer can only make the gap visible. |
| P4 | Internal and test accounts. 1,000 test users with many tiny orders inflate order counts and repeat-purchase rate. | A rule-based internal flag, excluded from every governed metric by default. The raw backend structurally cannot exclude them. |
| P5 | A high-return acquisition cohort. Users from one channel who signed up in a three-month period return far more items. Real, but invisible at channel level. | Nothing to fix; it is real. A cohort mart makes it findable at the right grain. The agent still has to look there. |
Results
Share of individual answers that passed every check, by question type (each bar is 3 to 5 questions times 3 phrasings). Under the strict rule, where a question only counts if all three phrasings pass and agree, the scores are 40% on raw tables, 45% through Cube with Gemini, and 30% through Cube with Claude.
Three answers side by side
Q01 · Internal accounts
Truth: 4,633
The agent noticed a real anomaly, a thin last day of the month, and blamed the gap on that. It had no way to see 1,000 test accounts. Confidently wrong.
Not exact either, but its stated definition says test and internal accounts are excluded, because the exclusion is built into the governed view. An error you can explain.
Q18 · Margin by channel
Truth: all five channels within 0.3 points (51.7% to 52.0%)
Reported Organic at about 30% margin against 46 to 47% elsewhere, then explained the gap with a product-mix story. The gap does not exist.
Said the channels are virtually identical on margin rate and separated that from absolute dollars, where Search leads on volume alone.
Q10 · Category rename
Trap: Knitwear was called Sweaters until October 2025
Three phrasings of the same question produced different conclusions. A leader asking in their own words would get whichever one they happened to trigger.
The category family mapping carries the rename, so the trend is continuous no matter how the question is worded.
Where the layer did not help
Engineering findings
Order-item metrics and order-level metrics both needed one "order date". In v1.7.46 that forced an unsupported join on every query. I routed the single catalog name to whichever physical member each metric needs, so the agent still sees one clean name.
Telling the model today was 31 August made it read "last month" as July. Naming the completed month explicitly fixed it. Date semantics belong in the prompt and the contract, written down once.
The test-account generator piled fake orders into the latest months (683 extra in one month). Unit tests checked averages and passed; a real question about last month exposed it. Evals on business questions are a test layer, not only a scoreboard.
BigQuery DATE vs TIMESTAMP in Cube time dimensions, and unqualified columns inside expressions, both break only when a particular join appears. A fixed set of 22 layer checks with no LLM involved is what made them cheap to find.
| Run | Tokens in | Tokens out | Cost | Median latency |
|---|---|---|---|---|
| Raw tables · Gemini | 8.19M | 88K | $7.08 | 55 s |
| Cube · Gemini | 11.84M | 78K | $10.03 | 55 s |
| Cube · Claude (control) | 2.00M | 199K | $14.97 | 41 s |
The agent receives schema metadata and aggregated query results only, never raw rows. Gemini runs on Vertex AI in an EU region inside the project's own Google Cloud boundary. The data is Google's public, synthetic theLook dataset; no real customer data is involved at any point.
02: the same benchmark on dbt Core and MetricFlow, with Claude as the agent. 03: Power BI with the model defined in TMDL. Same data, same questions, same judge, so the stacks can be compared fairly.
