New benchmark data shows how a smarter approach to context can cut AI costs and improve accuracy
The natural instinct for most teams building AI analytics tools is to make sure the AI understands your entire database, including every table, column and relationship. But what if I told you that instinct could cost your organization millions of dollars while also making your AI less accurate?
Here’s the problem with the “more context is better” mentality:
Every question a business user asks an AI tool triggers a new call to an LLM. If that call includes your full database schema, you’ll pay for those tokens every time, whether answering the questions requires three tables or 30.
Gartner has previously reported that agentic AI workloads consume anywhere from five to 30 times more tokens per task than a standard chatbot. One healthcare company we’ve seen referenced spent $6 million in unplanned annual token costs, all because its model’s context kept growing unchecked.
And cost isn’t even the biggest problem; accuracy is. Research on LLMs consistently shows what has become known as the “lost in the middle” effect, where a model needs to search through a long, cluttered prompt for relevant details and gets continually worse at finding them. Yes, it costs more to feed an AI more and more schema, but at a certain point it also makes the AI’s answers less reliable.
Context Graphs Support a Different Approach
There are plenty of perfectly capable models on the market, so the fix isn’t to simply use a smarter model. What teams need to get smarter about is deciding what the model sees in the first place.
That’s the idea behind a context graph. Rather than exposing AI to your entire database schema on every query, a context graph pre-encodes your business logic, including definitions, calculations, relationships and rules, into a structured knowledge base. That context is continuously maintained and autonomously expanded as the business evolves, allowing the graph to grow alongside new data, relationships and business logic. When a question is asked, the system pulls only the handful of entities relevant to that specific question so it can deliver a more compact but more precise package to the model.
Jedify’s Semantic Fusion™ architecture was built around this idea, and we wanted to find out how it holds up under real-world conditions. To do so, we ran a production benchmark with 100 real business questions across three difficulty tiers, and tested it twice for a total of 200 data points measured against a live enterprise data warehouse.
Spoiler alert: the results held up.
The Data Behind the Claim
Jedify’s context graph architecture used an average of 25,036 tokens per SQL generation call. Published research on traditional schema-injection approaches reports between 50,000 and 150,000 tokens for the same type of task, and some multi-agent systems run past 339,000 tokens per request.
Complexity wasn’t a needle-mover; the hardest questions in our test set (cohort analysis and multi-entity funnels) used only 6% more tokens on average than the simplest single-metric questions. In a traditional system, tougher questions usually mean the model needs to reason across more data, which drives up token usage. Our architecture doesn’t have that problem; the amount of context it retrieves scales with the entities a question touches, not with how difficult the question is.
When it comes to accuracy, the context graph answered 87% of graded test questions correctly. Published research on traditional schema-injection approaches reports 60% to 70% accuracy at enterprise scale, though it should be noted that those numbers come from different studies on different schemas, so they’re only reference points and not head-to-head comparisons.
Token ROI is Arguably the Metric That Matters Most
You need more than token count to determine if an AI system is worth what it costs. It’s not a bargain if it was cheap to produce a wrong answer. Costs from inaccuracies can build quickly because you’re essentially paying twice (or more): once for the original query, and again when someone catches and needs to fix the mistake.
Token ROI tells you how much business value a system delivers per token consumed. By that measure, our context graph architecture delivered roughly 4X the ROI of enterprise DDL-based approaches, and 18X the ROI of naive multi-agent pipelines. It’s a win-win because you’re spending fewer tokens, and those tokens produce more correct answers.
There’s a bonus: cheaper models become more viable. When context is properly scoped, the model doesn’t need to be the smartest one available. It needs to classify a question against a small, well-defined set of entities, a smaller task that cheaper models handle well.
Our analysis suggests that roughly 85% of enterprise analytics queries can run on lower-cost, open-source models instead of expensive frontier models without sacrificing much in terms of accuracy. Combined with the token savings, that gives you a much more attractive cost structure.
Here’s Where We’re Being Careful
Let’s be transparent about what this benchmark doesn’t prove. The baseline comparisons come from published studies on other schemas and question sets, not from re-running them on our own warehouse.
All 100 test questions were within our context graph’s existing coverage, so the benchmark measures accuracy on covered questions, not the share of a live enterprise question stream that a context graph would cover on day one. The results also reflect one production warehouse over two runs.
That said, the story holds up. This post covers some of the top-line findings, but the full report goes deeper, including the complete methodology, tier-by-tier token breakdowns, the accuracy grading process, and our view on where the study should be extended next.
Download the full report here.
