Evaluating Long-Term Performance of LLM Chatbots

Most people evaluate LLMs by just benchmarks. But in production, the real question is- how well do they perform? When you’re running inference at scale, these are the 3 performance metrics that matter most: 1️⃣ Latency How fast does the model respond after receiving a prompt? There are two kinds to care about: → First-token latency: Time to start generating a response → End-to-end latency: Time to generate the full response Latency directly impacts UX for chat, speed for agentic workflows, and runtime cost for batch jobs. Even small delays add up fast at scale. 2️⃣ Context Window How much information can the model remember- both from the prompt and prior turns? This affects long-form summarization, RAG, and agent memory. Models range from: → GPT-3.5 / LLaMA 2: 4k–8k tokens → GPT-4 / Claude 2: 32k–200k tokens → GPT-OSS-120B: 131k tokens Larger context enables richer workflows but comes with tradeoffs: slower inference and higher compute cost. Use compression techniques like attention sink or sliding windows to get more out of your context window. 3️⃣ Throughput How many tokens or requests can the model handle per second? This is key when you’re serving thousands of requests or processing large document batches. Higher throughput = faster completion and lower cost. How to optimize based on your use case: → Real-time chat or tool use → prioritize low latency → Long documents or RAG → prioritize large context window → Agentic workflows → find a balance between latency and context → Async or high-volume processing → prioritize high throughput My 2 cents 🤌 → Choose in-region, lightweight models for lower latency → Use 32k+ context models only when necessary → Mix long-context models with fast first-token latency for agents → Optimize batch size and decoding strategy to maximize throughput Don’t just pick a model based on benchmarks. Pick the right tradeoffs for your workload. 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

LLMs are optimized for next turn response. This results in poor Human-AI collaboration, as it doesn't help users achieve their goals or clarify intent. A new model CollabLLM is optimized for long-term collaboration. The paper "CollabLLM: From Passive Responders to Active Collaborators" by Stanford University and Microsoft researchers tests this approach to improving outcomes from LLM interaction. (link in comments) 💡 CollabLLM transforms AI from passive responders to active collaborators. Traditional LLMs focus on single-turn responses, often missing user intent and leading to inefficient conversations. CollabLLM introduces a :"Multiturn-aware reward" system, apply reinforcement fine-tuning on these rewards. This enables AI to engage in deeper, more interactive exchanges by actively uncovering user intent and guiding users toward their goals. 🔄 Multiturn-aware rewards optimize long-term collaboration. Unlike standard reinforcement learning that prioritizes immediate responses, CollabLLM uses forward sampling - simulating potential conversations - to estimate the long-term value of interactions. This approach improves interactivity by 46.3% and enhances task performance by 18.5%, making conversations more productive and user-centered. 📊 CollabLLM outperforms traditional models in complex tasks. In document editing, coding assistance, and math problem-solving, CollabLLM increases user satisfaction by 17.6% and reduces time spent by 10.4%. It ensures that AI-generated content aligns with user expectations through dynamic feedback loops. 🤝 Proactive intent discovery leads to better responses. Unlike standard LLMs that assume user needs, CollabLLM asks clarifying questions before responding, leading to more accurate and relevant answers. This results in higher-quality output and a smoother user experience. 🚀 CollabLLM generalizes well across different domains. Tested on the Abg-CoQA conversational QA benchmark, CollabLLM proactively asked clarifying questions 52.8% of the time, compared to just 15.4% for GPT-4o. This demonstrates its ability to handle ambiguous queries effectively, making it more adaptable to real-world scenarios. 🔬 Real-world studies confirm efficiency and engagement gains. A 201-person user study showed that CollabLLM-generated documents received higher quality ratings (8.50/10) and sustained higher engagement over multiple turns, unlike baseline models, which saw declining satisfaction in longer conversations. It is time to move beyond the single-step LLM responses that we have been used to, to interactions that lead to where we want to go. This is a useful advance to better human-AI collaboration. It's a critical topic, I'll be sharing a lot more on how we can get there.

In the rapidly evolving world of conversational AI, Large Language Model (LLM) based chatbots have become indispensable across industries, powering everything from customer support to virtual assistants. However, evaluating their effectiveness is no simple task, as human language is inherently complex, ambiguous, and context-dependent. In a recent blog post, Microsoft's Data Science team outlined key performance metrics designed to assess chatbot performance comprehensively. Chatbot evaluation can be broadly categorized into two key areas: search performance and LLM-specific metrics. On the search front, one critical factor is retrieval stability, which ensures that slight variations in user input do not drastically change the chatbot's search results. Another vital aspect is search relevance, which can be measured through multiple approaches, such as comparing chatbot responses against a ground truth dataset or conducting A/B tests to evaluate how well the retrieved information aligns with user intent. Beyond search performance, chatbot evaluation must also account for LLM-specific metrics, which focus on how well the model generates responses. These include: - Task Completion: Measures the chatbot's ability to accurately interpret and fulfill user requests. A high-performing chatbot should successfully execute tasks, such as setting reminders or providing step-by-step instructions. - Intelligence: Assesses coherence, contextual awareness, and the depth of responses. A chatbot should go beyond surface-level answers and demonstrate reasoning and adaptability. - Relevance: Evaluate whether the chatbot’s responses are appropriate, clear, and aligned with user expectations in terms of tone, clarity, and courtesy. - Hallucination: Ensures that the chatbot’s responses are factually accurate and grounded in reliable data, minimizing misinformation and misleading statements. Effectively evaluating LLM-based chatbots requires a holistic, multi-dimensional approach that integrates search performance and LLM-generated response quality. By considering these diverse metrics, developers can refine chatbot behavior, enhance user interactions, and build AI-driven conversational systems that are not only intelligent but also reliable and trustworthy. #DataScience #MachineLearning #LLM #Evaluation #Metrics #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gAC8eXmy

LLMs Get Lost in Multi-turn Conversation The cat is out of the bag. Pay attention, devs. This is one of the most common issues when building with LLMs today. Glad there is now paper to share insights. Here are my notes: The paper investigates how LLMs perform in realistic, multi-turn conversational settings where user instructions are often underspecified and clarified over several turns. I keep telling devs to spend time preparing those initial instructions. Prompt engineering is important. The authors conduct large-scale simulations across 15 top LLMs (including GPT-4.1, Gemini 2.5 Pro, Claude 3.7 Sonnet, DeepSeek-R1, and others) over six generation tasks (code, math, SQL, API calls, data-to-text, and document summarization). Severe Performance Drop in Multi-Turn Settings All tested LLMs show significantly worse performance in multi-turn, underspecified conversations compared to single-turn, fully-specified instructions. The average performance drop is 39% across six tasks, even for SoTA models. For example, models with >90% accuracy in single-turn settings often drop to ~60% in multi-turn settings. Degradation Is Due to Unreliability, Not Just Aptitude The performance loss decomposes into a modest decrease in best-case capability (aptitude, -15%) and a dramatic increase in unreliability (+112%). In multi-turn settings, the gap between the best and worst response widens substantially, meaning LLMs become much less consistent and predictable. High-performing models in single-turn settings are just as unreliable as smaller models in multi-turn dialogues. Don't ignore testing and evaluating in multi-turn settings. Main reasons LLMs get "lost" - Make premature and often incorrect assumptions early in the conversation. - Attempt full solutions before having all necessary information, leading to “bloated” or off-target answers. - Over-rely on their previous (possibly incorrect) answers, compounding errors as the conversation progresses. - Produce overly verbose outputs, which can further muddle context and confuse subsequent turns. - Pay disproportionate attention to the first and last turns, neglecting information revealed in the middle turns (“loss-in-the-middle” effect). Practical Recommendations: - Users are better off consolidating all requirements into a single prompt rather than clarifying over multiple turns. - If a conversation goes off-track, starting a new session with a consolidated summary leads to better outcomes. - System builders and model developers are urged to prioritize reliability in multi-turn contexts, not just raw capability. This is especially true if you are building complex agentic systems where the impact of these issues is more prevalent. - LLMs are really weird. And all this weirdness is creeping up into the latest models too but it more subtle ways. Be careful out there, devs.

🧠 LLMs still get lost in conversation. You should pay attention to this, specially when building AI Agents! A new paper just dropped, and it uncovers something many of us suspected: LLMs perform way worse when instructions are revealed gradually in multi-turn conversations. 💬 While LLMs excel when you give them everything up front (single-turn), performance drops by an average of 39% when the same task is spread across several conversational turns. Even GPT-4 and Gemini 2.5 stumble. Why? Because in multi-turn chats, models: ❌ Make premature assumptions ❌ Try to “wrap up” too soon ❌ Get stuck on their own past mistakes ❌ Struggle to recover when they go off-track The authors call this the “𝗟𝗼𝘀𝘁 𝗶𝗻 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻” effect, and it explains why LLMs sometimes seem great in demos, but frustrating in real-world use. 🔍 If you’re building agentic AI products, this is a wake-up call. Most evaluation benchmarks don’t reflect how users actually interact with messy, evolving, often underspecified prompts. 📄 Paper link in comments.

LLMs Get Lost in Multi-Turn Conversations: New Research Reveals Major Reliability Gap Just read a fascinating new paper from Microsoft and Salesforce Research revealing a critical flaw in today's LLMs: they dramatically underperform in multi-turn conversations compared to single-turn interactions. 📊 Key findings: 🔗 LLMs suffer an average 39% performance drop in multi-turn settings across six generation tasks 🔗 This occurs even in conversations with as few as two turns 🔗 The problem affects ALL tested models, including the most advanced ones (Claude 3.7, GPT-4.1, Gemini 2.5) 🔍 The researchers call this the "lost in conversation" phenomenon - when LLMs take a wrong turn in conversation, they get lost and don't recover. This is caused by: 🔗 Making assumptions too early 🔗 Prematurely generating final solutions 🔗 Relying too heavily on previous (incorrect) answers 🔗 Producing overly verbose responses 💬 Why conversation-level evaluation matters: Traditional LLM benchmarks focus on single-turn performance, creating a dangerous blind spot. Real-world AI interactions are conversational by nature, and this research shows that even the most capable models struggle with maintaining context and adapting to new information over multiple turns. Without robust conversation-level evaluation, we risk deploying systems that perform brilliantly in lab tests but frustrate users in practice. 🔎 At Coval, this is exactly what we focus on: evaluating LLMs in realistic conversational scenarios rather than isolated prompts. By measuring how models handle the natural flow of information across turns, we can identify reliability issues before they impact users and guide development toward truly conversational AI. This research highlights a critical gap between how we evaluate LLMs (single-turn) versus how we use them in practice (multi-turn). As we build AI assistants and agents, addressing this reliability issue becomes essential.

"When LLMs take a wrong turn in a conversation, they get lost and do not recover." Microsoft Research. New research shows top models drop 39% in performance during multi-turn conversations. Not temporarily confused. Lost. Unable to find their way back. Microsoft tested this with a simple math problem. When given all at once: "Jay makes 20 snowballs/hour but 2 melt every 15 minutes. How long until he has 60?" Models solve it fine. But break it into conversation: "How long before Jay's ready for the snowball fight?" "He's preparing for a fight with his sister" "He can make 20 snowballs per hour" "He's trying to get to 60 total" "The problem is that 2 melt every 15 minutes" Same information. Delivered conversationally. Performance drops 39%. This is exactly how real customer conversations unfold. Information comes in pieces. Context builds gradually. Requirements change mid-stream. Companies deploy these chatbots thinking they've solved customer service. The research tested GPT-4, Claude, Gemini, all the "frontier" models. All failed the same way. Everyone's racing to build AI agents for complex workflows. But the foundation can't handle a five-part conversation about snowballs. We're facing architectural limitations, not resource constraints. Maybe the future is admitting what models can't do, and building systems that solve these problems instead of ignoring them. Curious about different approaches to this challenge. What's working in your experience? Failures are equally valuable to share.