Preventing Context Window Waste in AI Workflows

What’s the point of a massive context window if using over 5% of it causes the model to melt down? Bigger windows are great for demos. They crumble in production. When we stuff prompts with pages of maybe-relevant text and hope for the best, we pay in three ways: 1️⃣ Quality: attention gets diluted, and the model hedges, contradicts, or hallucinates. 2️⃣ Latency & cost: every extra token slows you down, and costs rise rapidly. 3️⃣ Governance: no provenance, no trust, no way to debug and resolve issues. A better approach is a knowledge graph + GraphRAG pipeline that feeds the model the most relevant data with context instead of all the things it might need with no top-level organization. ✅ How it works at a high level: Model your world: extract entities (people, products, accounts, APIs) and typed relationships (owns, depends on, complies with) from docs, code, tickets, CRM, and wikis. GraphRAG retrieval: traverse the graph to pull a minimal subgraph with facts, paths, and citations, directly tied to the question. Compact context, rich signal: summarize those nodes and edges with provenance, then prompt. The model reasons over structure instead of slogging through sludge. Closed loop: capture new facts from interactions and update the graph so the system gets sharper over time. ✅ A 30-day path to validate it for your use cases: Week 1: define a lightweight ontology for 10–15 core entities/relations built around a high-value workflow. Week 2: build extractors (rules + LLMs) and load into a graph store. Week 3: wire GraphRAG (graph traversal → summarization → prompt). Week 4: run head-to-head tasks against your current RAG; compare accuracy, tokens, latency, and provenance coverage. Large context windows drive cool headlines and demos. Knowledge graphs + GraphRAG work in production, even for customer-facing use cases.

One of the biggest challenges I see with scaling LLM agents isn’t the model itself. It’s context. Agents break down not because they “can’t think” but because they lose track of what’s happened, what’s been decided, and why. Here’s the pattern I notice: 👉 For short tasks, things work fine. The agent remembers the conversation so far, does its subtasks, and pulls everything together reliably. 👉 But the moment the task gets longer, the context window fills up, and the agent starts forgetting key decisions. That’s when results become inconsistent, and trust breaks down. That’s where Context Engineering comes in. 🔑 Principle 1: Share Full Context, Not Just Results Reliability starts with transparency. If an agent only shares the final outputs of subtasks, the decision-making trail is lost. That makes it impossible to debug or reproduce. You need the full trace, not just the answer. 🔑 Principle 2: Every Action Is an Implicit Decision Every step in a workflow isn’t just “doing the work”, it’s making a decision. And if those decisions conflict because context was lost along the way, you end up with unreliable results. ✨ The Solution to this is "Engineer Smarter Context" It’s not about dumping more history into the next step. It’s about carrying forward the right pieces of context: → Summarize the messy details into something digestible. → Keep the key decisions and turning points visible. → Drop the noise that doesn’t matter. When you do this well, agents can finally handle longer, more complex workflows without falling apart. Reliability doesn’t come from bigger context windows. It comes from smarter context windows. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

Your AI agent is forgetting things. Not because the model is bad, but because you're treating memory like storage instead of an active system. Without memory, an LLM is just a powerful but stateless text processor - it responds to one query at a time with no sense of history. Memory is what transforms these models into something that feels way more dynamic and capable of holding onto context, learning from the past, and adapting to new inputs. Andrej Karpathy gave a really good analogy: think of an LLM's context window as a computer's RAM and the model itself as the CPU. The context window is the agent's active consciousness, where all its "working thoughts" are held. But just like a laptop with too many browser tabs open, this RAM can fill up fast. So how do we build robust agent memory? We need to think in layers, blending different types of memory: 1️⃣ 𝗦𝗵𝗼𝗿𝘁-𝗧𝗲𝗿𝗺 𝗠𝗲𝗺𝗼𝗿𝘆: The immediate context window This is your agent's active reasoning space - the current conversation, task state, and immediate thoughts. It's fast but limited by token constraints. Think of it as the agent's "right now" awareness. 2️⃣ 𝗟𝗼𝗻𝗴-𝗧𝗲𝗿𝗺 𝗠𝗲𝗺𝗼𝗿𝘆: Persistent external storage This moves past the context window, storing information externally (often in vector databases) for quick retrieval when needed. It can hold different types of info: • Episodic memory: specific past events and interactions • Semantic memory: general knowledge and domain facts • Procedural memory: learned routines and successful workflows This is commonly powered by RAG, where the agent queries an external knowledge base to pull in relevant information. 3️⃣ 𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝗠𝗲𝗺𝗼𝗿𝘆: A temporary task-specific scratchpad This is the in-between layer - a temporary holding area for multi-step tasks. For example, if an agent is booking a flight to Tokyo, its working memory might hold the destination, dates, budget, and intermediate results (like "found 12 flights, top candidates are JAL005 and ANA106") until the task is complete, without cluttering the main context window. Most systems I've seen use a hybrid approach, using short-term memory for speed with long-term memory for depth, plus working memory for complex tasks. Effective memory is less about how much you can store and more about 𝗵𝗼𝘄 𝘄𝗲𝗹𝗹 𝘆𝗼𝘂 𝗰𝗮𝗻 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗲 𝘁𝗵𝗲 𝗿𝗶𝗴𝗵𝘁 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗮𝘁 𝘁𝗵𝗲 𝗿𝗶𝗴𝗵𝘁 𝘁𝗶𝗺𝗲. The architecture you choose depends entirely on your use case. A customer service bot needs strong episodic memory to recall user history, while an agent analyzing financial reports needs robust semantic memory filled with domain knowledge. Learn more in our context engineering ebook: https://lnkd.in/e6JAq62j

BREAKING: Anthropic launches Claude Opus 4.5 Several of the upgrades are squarely at people like me, developers building agents. I'm particularly impressed with how elegantly they handle one of the biggest issues developers are dealing with: tool calling. MCP is great (as a protocol) but has a major issue with how it front-loads the context window and consuming a lot of tokens. It's easy to have a run-away set of MCP servers/tools that clutter the context window and degrade performance. So, here's what we have now (that I'm playing with for the next 12 hours): 1. Tool Search Instead of shoving every tool definition into the model up front (like packing your entire house for a weekend trip), Claude can now fetch tool definitions only when it needs them. Fewer tokens. Faster responses. Less clutter. More joy. 2. Programmatic Tool Calling Claude can now write code to orchestrate multiple tools without dragging every intermediate result back into the model’s context. Think of it as moving from “chatty assistant” to “competent developer who actually reads the docs.” 3. Tool Use Examples You can now give example calls to show how a tool should be used—not just what fields it has. This dramatically reduces the “I see your schema and choose chaos anyway” problem. Why this matters: If you’re building agent workflows with lots of tools, these upgrades cut token usage, reduce latency, improve reliability, and generally make your agent behave more like a well-trained teammate and less like an overeager intern. My take: As agent architectures get more complex, the bottleneck isn’t the model—it's the orchestration. These features move us closer to agents that can reason, retrieve, call tools, and coordinate real work at scale. In other words: better plumbing, better agents. If you're building anything agent-heavy, it’s worth a look. p.s. The image was created with my new ImageGen .ai agent (single prompt consisting of the text of the post).

Context engineering is quickly becoming one of the most critical skills in applied AI. Not prompt tweaking. Not model fine-tuning. But knowing what information a model needs, and when to give it. That is the real unlock behind AI agents that actually work. At its core, context engineering is about delivering the right information to the model, at the right time, in the right format; so it can reason effectively. It pushes developers to think more intentionally about how they shape a model’s inputs: 🔸What does the model need to know for this task? 🔸Where should that information come from? 🔸How do we fit it within the limits of the context window? 🔸And how do we prevent irrelevant or conflicting signals from getting in the way? Why does this matter so much? In practice, most agent failures are not due to weak models. They happen because the model did not have the context it needed. It missed a key fact, relied on stale data, or was overloaded with noise. Context engineering addresses this directly. It forces you to design the flow of information step by step, not just what the model sees, but how and when it sees it. This context can come from many places: 🔹Long- and short-term memory (such as prior conversations or user history) 🔹Retrieved data from APIs, vector stores, or internal systems 🔹Tool definitions and their recent outputs 🔹Structured formats or schemas that define how information is used 🔹Global state shared across multi-step workflows Frameworks like LlamaIndex, LangGraph AI, LangChain, are evolving to support this shift, giving developers the tools to manage context with much more precision. And there are now better resources than ever to help teams write, select, compress, and organize context with real control. Image from Langchain blog. #contextengineering #llms #generativeai #artificialintelligence #technology 

Context engineering is the art of giving AI systems the right information at the right time. Most engineers focus on initial retrieval, that is, finding relevant documents from a large corpus. But retrieval is just the first step. The real challenge is prioritization. A reranker sits between the first step of retrieval and generation, making critical decisions about which pieces of context deserve the model's attention. This is essential regardless of context window length. Enterprise documents can contain conflicting information across sources over time. Recent work on context rot demonstrates that LLMs don’t process long input contexts uniformly, and best practices still require precise optimization of context at each step. You might retrieve 50 relevant documents with similar content, but your model can only meaningfully process 10. The reranker decides which 10 matter most. Sometimes, with simple problems, you can use metadata filtering instead of reranking. However, metadata filters quickly hit limitations when dealing with semantic understanding, relative importance, or multi-dimensional relevance criteria that require actual comprehension of content. But instruction-following rerankers go further: they can be configured at the agent/application level for consistent behavior, or dynamically at the query level for specific tasks. These rerankers excel at complex, nuanced requirements that metadata alone can't capture: "Prioritize documents from the last 6 months and internal sources. However, if dealing with foundational concepts, include older authoritative external sources." They handle semantic nuance and soft filters. Good context engineering recognizes that relevance isn't binary, it's instructable and contextually complex. Rerankers are a critical control point that transforms raw retrieval results into precisely curated context for optimal model performance.

Everyone claims "context is king," but Dropbox admits the messy reality: adding more tools to your agent actually makes it stupider. Most engineers think "Agentic AI" means giving an LLM access to every API in the stack. Dropbox found the opposite: giving agents more tools caused "analysis paralysis." To fix this "context rot," they implemented three architectural shifts. First, they collapsed dozens of granular retrieval APIs (Jira, Slack, GDocs) into a single "Universal Search" tool, drastically reducing the prompt's schema load. Second, they shifted relevance computation upstream, using a pre-built Knowledge Graph to rank and prune content before it ever touches the context window. Finally, they decoupled logic by offloading complex query construction to a specialized "Search Agent," leaving the main "Planner Agent" free to focus on orchestration. This proves that the bottleneck for production agents isn't model capability; it's the signal-to-noise ratio of the input. We don't need larger context windows; we need better context engineering. More information in the blog post: 🔗https://lnkd.in/egy4jTF4

Managing context windows for AI agents is surprisingly complicated. I’m working on a video about context engineering, and a paper on Agentic Context Engineering (ACE) ( 🔗 link in comments) tackled some common issues I'm seeing: •If you over summarize or truncate, you risk losing the important details. Continual and repeated summarizing removes the specifics the agent needs to do the job and so over time the results can degrade. This is called context collapse. •If you over-rely on packing massive 1 million token context windows without being intentional, you may be adding noise that buries the important details (and you pay for a lot of tokens that aren't helping much) The paper lays out a pattern to attempt to address these common issues. • Treat your context like a living playbook. Instead of rewriting or compressing everything, make small, structured updates based on what the agent learned. • Approach changes to context incrementally rather than wholesale rewriting. • Represent context as a collection of structured bullets, rather than a single monolithic prompt. The ACE pattern uses three separate agents working together: 🤖 Generator agent Takes the current context and produces a result, as well as an explanation of how it got there. 🪞 Reflector agent Reviews the generator’s output and identifies what worked and what didn’t. 🎨 Curator Determines how to update the context with any new insights from the reflector. It decides what to keep, what to discard, and what to add. I'm going to test this pattern out myself soon, and I'll report back the results in the video + following blog that comes from this. How are you handling or thinking about context engineering? Have you tried or heard of ACE? I'd love to hear your thoughts! Let me know in the comments. #AI #AIAgents #ContextEngineering #LLM #MachineLearning