Your AI agent just answered a revenue question wrong — again. It pulled gross revenue when finance wanted churn-adjusted net revenue. It joined orders to customers on the wrong key because two tables both have a customer_id column with different semantics. It ignored the grace-period exclusion logic that every analyst on the team knows by heart but nobody has ever written down.
The agent has no idea what your data actually means.
A context layer for AI agents bridges raw warehouse tables and business-meaningful definitions, mapping columns, metrics, and exception logic to entities an agent can reference without guessing schema intent. Without it, agents hallucinate joins, misapply metric definitions, and produce numbers your CFO won’t trust.
Raw warehouse tables contain columns, data types, and foreign keys. They don’t contain business definitions, exception logic, or the tribal knowledge that separates a number your CFO will trust from one that will get you called into a meeting. Three realistic approaches exist for building that layer:
- Manual configuration using YAML, LookML, or Snowflake Semantic Views
- Hybrid configuration layering dbt’s MetricFlow definitions on top of existing dbt models
- Autonomous construction that mines query logs, runs NER over unstructured document corpora, and clusters semantic concepts without requiring a single hand-authored file
Each approach makes a different trade-off across four dimensions that matter when running agents in production:
- Setup time: how long before the agent has a usable context layer
- Maintenance burden: what it costs to keep that layer accurate as schemas evolve
- Coverage depth: what fraction of the entities users actually query the agent about are defined
- Agent reasoning quality: whether the agent generates correct SQL on the first pass, applies the right metric definition across departments, and handles exception logic without user correction
The right approach depends on where your team is right now and what it can realistically sustain.
Approach 1: Manual Configuration (YAML, LookML, Snowflake Semantic Views)
Manual context definition is the most direct path: you author a definition, the agent uses it. Snowflake Cortex Analyst consumes semantic models written in YAML. LookML models in Looker serve the same function. The workflow is the same regardless of tooling: an analytics engineer identifies a metric, writes the definition, encodes the business logic, and commits it.
A hand-authored definition for net_revenue in a Snowflake Cortex Analyst semantic model looks like this:
# Snowflake Cortex Analyst — semantic model name: revenue_model tables: - name: orders description: "Subscription order records, one row per billing event" measures: - name: net_revenue expr: SUM(gross_amount - refund_amount) description: > Subscription revenue after refunds. Excludes accounts where churned_status = 'grace_period' (30-day post-churn window). data_type: NUMBER filters: - name: exclude_grace_period expr: "churned_status != 'grace_period'" dimensions: - name: customer_id expr: customer_id description: "Maps to accounts.account_id — not crm_contacts.contact_id" data_type: VARCHAR
That disambiguation in the customer_id dimension description (“maps to accounts.account_id, not crm_contacts.contact_id“) is the kind of exception logic that prevents a hallucination. Without it, the agent joins whichever customer_id it finds first.
However, writing that line requires someone to know it needs to be written. That’s where the manual approach hits its ceiling.
The biggest cost isn’t writing the YAML — it’s achieving enough coverage to be useful. Plan for days to weeks per entity cluster upfront, depending on complexity. A single net_revenue metric with three exception rules and two disambiguation notes can take a full day to define correctly. Multiply that across a schema with hundreds of business entities and you’re looking at months of engineering time before the agent can handle the full range of user queries.
Maintenance adds another layer of friction. Every upstream schema change — a column rename, a new enum value in churned_status, a table migration — creates a drift risk. Those definitions don’t update themselves. You typically discover the gap when the agent starts producing wrong answers, not before.
Honest scoring: Setup time is high. Maintenance burden is high and grows linearly with entity count. Coverage depth equals exactly what someone has authored — a hard ceiling with a cliff edge. Agent reasoning quality is excellent for covered entities and absent for everything else.
Manual configuration works well when the scope is narrow: a specific domain (finance reporting, for example), a stable schema, and a dedicated engineer who owns definitions. It stops working when agents are expected to answer questions across the full data surface.
Approach 2: Hybrid Configuration (dbt Semantic Layer + Catalog Metadata)
The dbt Semantic Layer with MetricFlow is the strongest hybrid option available. If you’re already running dbt, MetricFlow lets you define metrics directly in version-controlled .yml files on top of existing dbt models. The metric definitions live in Git, follow normal PR cadence, and integrate with dbt’s lineage graph — a meaningful improvement over free-floating YAML files with no ownership signals.
A MetricFlow net_revenue definition:
metrics: - name: net_revenue label: Net Revenue description: > Subscription revenue after refunds and churn adjustments. Excludes accounts with churned_status = 'grace_period'. type: derived type_params: expr: gross_revenue - refunds - churn_adjusted_deductions metrics: - name: gross_revenue - name: refunds - name: churn_adjusted_deductions
Adding a catalog tool like Atlan on top of MetricFlow definitions brings in descriptions, ownership signals, and lineage metadata. An agent consuming this layer gets the metric definition, the dbt model it derives from, the owner, and the lineage back to raw source tables.
The main constraint is dbt’s own coverage. The hybrid approach can only reach what dbt models already include. Any metric that isn’t modeled in dbt is invisible to the agent. Most data warehouses have a long tail of tables, columns, and business logic that never made it into a dbt model — because there wasn’t time, because the analyst who knew the definition left, or because the logic lives in a Slack thread from two years ago.
Tribal knowledge is the specific failure mode. MetricFlow can encode that net_revenue excludes grace-period churn, but only if someone wrote that down first. An analyst who adds a new Jinja filter in a dbt model and explains it in the PR description produces context that never reaches the semantic layer. A sales ops manager who knows that EMEA accounts follow a different fiscal calendar carries knowledge that lives outside any .yml file entirely.
Honest scoring: Setup time is meaningfully lower than manual if dbt models already exist. Maintenance is more manageable because metric changes follow the standard dbt PR workflow. Coverage depth remains human-bounded, capped at whatever the dbt model surface area covers. Agent reasoning quality improves over manual for in-model entities, but the same cliff edge exists at the dbt model boundary.
The hybrid approach is the right call when your organization already runs dbt with 50+ models and wants metric coverage to track with model development cadence. When users expect the agent to answer questions across the full data surface, it falls short — because the full surface almost always extends well beyond what the dbt model graph covers.
Approach 3: Autonomous Construction (Query Log Mining, NER, BERTopic, Co-occurrence Analysis)
Autonomous context layer construction inverts the workflow entirely. Instead of asking engineers to write definitions, it mines evidence of how your data is actually used and builds the context layer from that signal. The pipeline runs in four sequential stages.
Stage 1: Query log mining
Historical SQL from SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY, BigQuery’s INFORMATION_SCHEMA.JOBS, or Databricks system tables gets processed using FP-Growth and Apriori algorithms to surface frequently co-queried entities, join patterns, and filter conditions. At meaningful scale — roughly 3 million queries — this produces around 300 inferred entity relations and 600 candidate metrics. The mining identifies which column combinations consistently appear together, which filters recur across query authors, and which join conditions are stable versus inconsistent.
Stage 2: NER over unstructured corpora
Named Entity Recognition runs over internal wikis, Notion pages, Slack threads, and PDF documentation to extract business terms and metric names that exist in everyday language but not in the schema. This is where tribal knowledge enters the pipeline. If your data team’s Slack channel contains 40 messages referencing “bookings-adjusted ARR” with varying filter conditions, NER surfaces that term and its surrounding context as candidate material for a Semantic Atom.
Stage 3: BERTopic clustering
Topic modeling groups recurring semantic concepts across the document corpus. BERTopic identifies that “churn-adjusted revenue,” “net revenue (ex-churn),” and “recognized revenue post-grace” all refer to the same underlying business concept, even when they appear in different documents using different language. Manual methods can only encode terms someone thought to define. Semantic patterns spread across thousands of documents stay hidden without this step.
Stage 4: Co-occurrence analysis and entity resolution
Co-occurrence patterns across query logs and documents resolve disambiguation conflicts. When two tables both have a customer_id column mapping to different business entities, co-occurrence analysis identifies which customer_id appears alongside ARR metrics versus support ticket counts, and encodes that distinction directly into the entity definition.
The output of this pipeline is a set of Semantic Atoms that fuse into Semantic Entities. Each Semantic Entity carries three components: a contextual definition in natural language, a SQL implementation the agent can execute, and provenance signals identifying which source contributed each element (query log, document, or schema). The provenance layer matters for the review pass.
Jedify’s Semantic Fusion™ runs this pipeline in production. The Connect + Generate pass (connecting a warehouse and running the full pipeline) completes in under an hour. The output is a Context Graph: a structured representation of business entities, their relationships, and the definitions an agent needs to reason correctly.
The human-in-the-loop requirement. Automated generation requires a review pass to correct misclassified entity groupings, add exception logic not present in any document, and encode tribal knowledge that has no written form. Jedify’s Refine step (a Semantic Statements interface for correcting and extending entity definitions) feeds each human correction back into the model via an RLHF loop. Over successive “Refine” passes, the pipeline improves: the same correction that fixes one entity signals how to handle similar entities in subsequent generation cycles. Maintenance shifts from authoring from scratch to reviewing and correcting auto-generated definitions.
Honest scoring: Coverage depth is the highest of the three approaches, surfacing entities and relations that were never explicitly authored — including business logic encoded only in Slack threads and tribal knowledge carried only in analyst memory. Setup time is low for the initial Generate pass. The caveat is first-pass accuracy: sparse query logs (a new product line with three months of history) or a thin document corpus reduces the signal available to NER and BERTopic, lowering the quality of auto-generated definitions before the Refine pass. The approach works best when the warehouse has meaningful query history and the organization has internal documentation to mine.
Evaluation Matrix
| Setup Time | Maintenance Burden | Coverage Depth | Agent Reasoning Quality | |
|---|---|---|---|---|
| Manual (YAML / Snowflake Semantic Views) | 🔴 High — days to weeks per entity cluster | 🔴 High — linear with entity count; spikes on schema changes | 🔴 Low — bounded by team bandwidth; nothing beyond what’s been authored | ✅ Excellent for covered entities / Zero for uncovered |
| Hybrid (dbt MetricFlow + Catalog) | 🟡 Medium — lower if dbt models exist | 🟡 Medium — changes follow dbt PR cadence; still engineer-bottlenecked | 🟡 Medium — bounded by dbt model surface; tribal knowledge falls through | 🟡 Good for in-model entities / Same cliff at dbt boundary |
| Autonomous (Semantic Fusion™) | ✅ Low — Connect + Generate in under 1 hour | ✅ Low — shifts from authoring to reviewing; RLHF improves over cycles | ✅ High — surfaces unauthored entities; ingests unstructured sources | ✅ Highest — contextual grounding reduces hallucinations; degrades on sparse logs |
Where each approach breaks
- Manual breaks at team turnover. Definitions live in YAML files with no provenance trail. When the engineer who wrote them leaves, nobody knows which exception rules were intentional versus forgotten edge cases. A column rename doesn’t throw an error in your semantic layer — it produces wrong answers, silently.
- Hybrid breaks when schema changes outpace dbt model updates. A new billing table that hasn’t been modeled in dbt yet is invisible to MetricFlow. In fast-moving product orgs, that delta is continuous rather than occasional.
- Autonomous breaks when query logs are sparse (a new product, a recently migrated data model, or a warehouse with less than a few months of history) or when the document corpus is too thin for NER to extract meaningful signals. First-pass accuracy drops proportionally.
Manual coverage is viable below roughly 50 defined entities on a stable schema. The hybrid approach extends that to the full dbt-modeled surface area. The autonomous approach becomes necessary when agents are expected to reason over the full scope of what users actually ask — covering the portion the team has never pre-modeled.
Selecting the Right Approach
The three approaches aren’t mutually exclusive, but each has a clear primary use case.
Choose manual when you have fewer than 50 target entities, a stable schema, a dedicated analytics engineer who owns definitions, and agents scoped to a narrow, well-documented domain. A finance reporting agent covering a dozen well-defined metrics is a good manual candidate. A generalist data agent expected to answer ad hoc questions across the full warehouse needs a different approach.
Choose hybrid when you’re already running dbt with 50+ models and want metric coverage to track with model development cadence. It’s the right call if your team values version-controlled definitions, if PRs are already the mechanism for schema governance, and if you can accept that coverage is bounded by what the dbt graph covers. It’s the strongest engineering-process answer to the context layer problem.
Choose autonomous when your schema surface area is large, meaningful business logic lives in Slack threads and Notion docs rather than any YAML file, you expect agents to answer the full range of user questions rather than a pre-defined subset, and you can’t staff manual authoring at the entity count your query surface requires. This is the right answer for most mid-to-large data organizations trying to ship a general-purpose AI agent against a real data warehouse.
One clarification worth making explicit: autonomous pipelines can ingest existing YAML and LookML definitions as a bootstrapping input. The Semantic Fusion™ Context Graph becomes a superset of your existing semantic layer. If you’ve already authored 40 MetricFlow metrics, those definitions become source material for the autonomous pipeline, which inherits them and extends beyond them.
Measure Your Coverage Delta Before Deciding
Before committing to any approach, run this query against your warehouse to surface the 20 most-referenced table/column pairs that have no semantic definition. The results make the coverage delta concrete before any tooling decision.
Snowflake:
SELECT
obj.value:objectName::STRING AS table_name,
col.value:columnName::STRING AS column_name,
COUNT(*) AS query_frequency
FROM snowflake.account_usage.access_history,
LATERAL FLATTEN(base_objects_accessed) obj,
LATERAL FLATTEN(obj.value:columns, outer=>TRUE) col
WHERE query_start_time >= DATEADD('day', -90, CURRENT_TIMESTAMP())
AND col.value:columnName IS NOT NULL
GROUP BY 1, 2
ORDER BY query_frequency DESC
LIMIT 20;
BigQuery:
SELECT referenced_table.project_id || '.' || referenced_table.dataset_id || '.' || referenced_table.table_id AS table_reference, COUNT(*) AS query_frequency FROM `region-us`.INFORMATION_SCHEMA.JOBS, UNNEST(referenced_tables) AS referenced_table WHERE creation_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY) AND state = 'DONE' GROUP BY 1 ORDER BY query_frequency DESC LIMIT 20;
Cross-reference the results against your existing semantic definitions, whatever form they take. The delta is your coverage gap. If the top-10 results are all tables you’ve defined, the manual or hybrid approach may be sufficient. If they include tables your analytics engineers haven’t touched in years, that’s the autonomous case.
Connect a warehouse (Snowflake, BigQuery, Databricks, or Redshift) to Jedify’s self-serve setup flow to see what autonomous construction surfaces from your actual query logs. Semantic Fusion™ runs a full Generate pass (query log mining, NER, BERTopic clustering, co-occurrence resolution) and produces a Context Graph in under an hour. Comparing the resulting Semantic Entities against your existing YAML definitions is the fastest way to quantify what autonomous construction surfaces that manual authoring missed.
Once the Context Graph is ready, the Jedify Contextual MCP Server exposes the finished graph to any MCP-compatible agent client (Claude Desktop, Cursor, custom apps) through a single endpoint, with no fragmented prompt engineering required.
Connect your warehouse and see what Semantic Fusion™ surfaces from your actual query logs — self-serve setup completes in under an hour. Get started for free.
Frequently Asked Questions
What is a context layer for AI agents, and why does it matter?
A context layer is a structured mapping between raw warehouse tables and the business definitions, metric logic, and exception rules an AI agent needs to produce correct answers. Without one, agents resolve ambiguous column names by guessing, apply the wrong metric definitions across departments, and miss exception logic that analysts treat as common knowledge. The context layer is what separates a number your CFO trusts from one that triggers a meeting.
How long does it take to build a semantic layer manually in YAML or LookML?
A single metric with exception rules and disambiguation notes typically takes a full day to author correctly. An entity cluster (a related group of metrics and dimensions) takes days to weeks. A schema with hundreds of meaningful business entities can require months of engineering time before an agent has sufficient coverage to be useful across the full query surface.
What is the dbt Semantic Layer, and how does MetricFlow differ from manual YAML authoring?
The dbt Semantic Layer with MetricFlow lets teams define metrics in version-controlled .yml files on top of existing dbt models, following the same PR cadence used for model changes. Unlike standalone YAML files, MetricFlow definitions carry lineage back to source tables and integrate with dbt’s ownership signals. The constraint is coverage: MetricFlow can only define what dbt models already cover, leaving any un-modeled table or tribal knowledge outside the agent’s reach.
What makes autonomous context layer construction different from manual or hybrid approaches?
Autonomous construction mines existing evidence — SQL query logs, internal wikis, Slack threads, Notion pages — rather than requiring engineers to author definitions from scratch. Query log mining surfaces join patterns and filter conditions from millions of historical queries. NER extracts business terms from unstructured documents. BERTopic clusters semantic variants of the same concept. The result is coverage of entities that were never explicitly defined, including business logic that exists only in analyst memory or informal documentation.
When does autonomous context generation produce low-quality results?
First-pass accuracy degrades when query logs are sparse — typically fewer than a few months of warehouse history, or a recently migrated data model with limited historical queries. A thin document corpus similarly reduces NER signal quality. The Refine step (human review of auto-generated definitions with RLHF feedback) corrects misclassifications and adds exception logic that no document captures; the pipeline improves over successive review cycles.
Can autonomous pipelines coexist with existing YAML or LookML definitions?
Yes. Autonomous pipelines can ingest existing YAML, LookML, and MetricFlow definitions as bootstrapping input. The resulting Context Graph becomes a superset of the existing semantic layer, inheriting hand-authored definitions and extending coverage beyond them. Teams with 40 MetricFlow metrics already authored can use those definitions as source material without discarding prior work.
How do I measure my current semantic coverage gap before choosing an approach?
Run a query against SNOWFLAKE.ACCOUNT_USAGE.ACCESS_HISTORY (Snowflake) or INFORMATION_SCHEMA.JOBS (BigQuery) to surface the most-frequently-referenced table/column pairs over the last 90 days. Cross-reference those results against your existing semantic definitions. The unmatched rows are your coverage gap — the entities your agents are resolving by guessing rather than by definition.
