Your AI agents are only as smart as your business data

Why structured data needs context

Business data rarely explains itself. A help center article usually contains both the information and its meaning. If it says "customers have 30 days to return an item," an AI agent can use that information directly.

Structured data works differently. A dashboard might report an average resolution time of 71 minutes, but the number alone says very little. Is it good or bad? Is it typical for Mondays? Has the definition of resolution time changed? Does it apply to every support channel?

People naturally bring this context with them. They understand how metrics are defined, which comparisons matter, and what the business considers normal. AI needs that same understanding made explicit.

Like humans, LLMs also struggle to retain precise knowledge over time. During pretraining, exact facts are compressed across trillions of tokens rather than stored for direct retrieval. As researchers behind DeepSeek's recent architecture paper observed, "transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation."

This is why context engineering has become so important. Techniques like retrieval-augmented generation (RAG) provide AI with the specific business context it needs at the moment a decision is made, reducing hallucinations by grounding responses in reliable information rather than relying solely on model memory.

The same principle applies to structured business data. AI needs access to a database along with the business context that explains how data was generated, how metrics are defined, what relationships exist between datasets, and how the organization interprets performance.

Recent research from Anthropic illustrates just how important this context is. Simply giving an AI agent access to structured business data resulted in just 21% accuracy on the company's internal analytics evaluations. Anthropic found the biggest improvements came from adding business context through governed semantic layers, curated skills, and maintained documentation, lifting accuracy to more than 95%. Anthropic also found that this knowledge needs continual maintenance. As businesses evolve, metric definitions change, new data sources are introduced, and workflows adapt. Without keeping that knowledge current, accuracy declined significantly over time.

OpenAI reached a similar conclusion when developing its in-house data agent. Success depended on curated business context created by domain experts—capturing intent, semantics, business meaning, and known caveats that couldn't be inferred from the data alone.

The implementation will differ across organizations. It might take the form of skills, semantic layers, context engineering, or self-improving memories. The underlying principle is the same: AI needs more than access to business data. It needs the meaning behind that data.