Automating Model Evaluation Using LLMs

Evaluating LLMs is not like testing traditional software. Traditional systems are deterministic → pass/fail. LLMs are probabilistic → same input, different outputs, shifting behaviors over time. That makes model selection and monitoring one of the hardest engineering problems today. This is where Eval Protocol (EP) developed by Fireworks AI is so powerful. It’s an open-source framework for building an internal model leaderboard, where you can define, run, and track evals that actually reflect your business needs. → Simulated Users – generate synthetic but realistic user interactions to stress-test models under lifelike conditions. → evaluation_test – pytest-compatible evals (pointwise, groupwise, all) so you can treat model behavior like unit tests in CI/CD. → MCP Extensions – evaluate agents that use tools, multi-step reasoning, or multi-turn dialogue via Model Context Protocol. → UI Review – a dashboard to visualize eval results, compare across models, and catch regressions before they ship. Instead of relying on generic benchmarks, EP lets you encode your own success criteria and continuously measure models against them. If you’re serious about scaling LLMs in production, this is worth a look: evalprotocol.io

Evaluating Retrieval-Augmented Generation (RAG) systems has long been a challenge, given the complexity and subjectivity of long-form responses. A recent collaborative research paper from institutions including the University of Waterloo, Microsoft, and Snowflake presents a promising solution: the AutoNuggetizer framework. This innovative approach leverages Large Language Models (LLMs) to automate the "nugget evaluation methodology," initially proposed by TREC in 2003 for assessing responses to complex questions. Here's a technical breakdown of how it works under the hood: 1. Nugget Creation:   - Initially, LLMs automatically extract "nuggets," or atomic pieces of essential information, from a set of related documents.   - Nuggets are classified as "vital" (must-have) or "okay" (nice-to-have) based on their importance in a comprehensive response.   - An iterative prompt-based approach using GPT-4o ensures the nuggets are diverse and cover different informational facets. 2. Nugget Assignment:   - LLMs then automatically evaluate each system-generated response, assigning nuggets as "support," "partial support," or "no support."   - This semantic evaluation allows the model to recognize supported facts even without direct lexical matching. 3. Evaluation and Correlation:   - Automated evaluation scores strongly correlated with manual evaluations, particularly at the system-run level, suggesting this methodology could scale efficiently for broad usage.   - Interestingly, the automation of nugget assignment alone significantly increased alignment with manual evaluations, highlighting its potential as a cost-effective evaluation approach. Through rigorous validation against human annotations, the AutoNuggetizer framework demonstrates a practical balance between automation and evaluation quality, providing a scalable, accurate method to advance RAG system evaluation. The research underscores not just the potential of automating complex evaluations, but also opens avenues for future improvements in RAG systems.

Evaluating LLMs accurately/reliably is difficult, but we can usually automate the evaluation process with another (more powerful) LLM... Automatic metrics: Previously, generative text models were most commonly evaluated using automatic metrics like ROUGE and BLEU, which simply compare how well a model’s output matches a human-written target resopnse. In particular, BLEU score was commonly used to evaluatate machine translation models, while ROUGE was most often used for evaluating summarization models. Serious limitations: With modern LLMs, researchers began to notice that automatic metrics did a poor job of comprehensively capturing the quality of an LLM’s generations. Oftentimes, ROUGE scores were poorly correlated with human preferences—higher scores don’t seem to indicate a better generation/summary [1]. This problem is largely due to the open-ended nature of most tasks solved with LLMs. There can be many good responses to a prompt. LLM-as-a-judge [2] leverages a powerful LLM (e.g., GPT-4) to evaluate the quality of an LLM’s output. To evaluate an LLM with another LLM, there are three basic structures or strategies that we can employ: (1) Pairwise comparison: The LLM is shown a question with two responses and asked to choose the better response (or declare a tie). This approach was heavily utilized by models like Alpaca/Vicuna to evaluate model performance relative to proprietary LLMS like ChatGPT. (2) Single-answer grading: The LLM is shown a response with a single answer and asked to provide a score for the answer. This strategy is less reliable than pairwise comparison due to the need to assign an absolute score to the response. However, authors in [2] observe that GPT-4 can nonetheless assign relatively reliable/meaningful scores to responses. (3) Reference-guided grading: The LLM is provided a reference answer to the problem when being asked to grade a response. This strategy is useful for complex problems (e.g., reasoning or math) in which even GPT-4 may struggle with generating a correct answer. In these cases, having direct access to a correct response may aid the grading process. “LLM-as-a-judge offers two key benefits: scalability and explainability. It reduces the need for human involvement, enabling scalable benchmarks and fast iterations.” - from [2] Using MT-bench, authors in [2] evaluate the level of agreement between LLM-as-a-judge and humans (58 expert human annotators), where we see that there is a high level of agreement between these strategies. Such a finding caused this evaluation strategy to become incredibly popular for LLMs—it is currently the most widely-used and effective alternative to human evaluation. However, LLM-as-a-judge does suffer from notable limititations (e.g., position bias, verbosity bias, self-enhancement bias, etc.) that should be considered when interpretting data.

“A Survey on LLM-as-a-Judge” outlines what could become a foundational shift in how we evaluate AI systems, and the paper is very insightful. The idea is simple, but profound: use LLMs not just to generate content, but to judge it across tasks like summarization, reasoning, classification, and beyond. Why does this matter? Because traditional evaluation methods no longer scale: - Human reviews are expensive, inconsistent, and hard to reproduce. - Automatic metrics like BLEU and ROUGE fail to capture meaning, nuance, or utility. LLM-as-a-Judge offers a compelling alternative: scalable, nuanced, and surprisingly aligned with expert judgment when done right. What makes this paper stand out is the depth and structure it brings to a chaotic space. It: 1. Defines a clear taxonomy of evaluation methods (scoring, pairwise, yes/no, multi-choice) 2. Details the full pipeline from prompt design to model selection to post-processing 3. Surfaces real risks (biases, hallucinations, format brittleness) and proposes mitigation strategies 4. Introduces benchmarks and best practices for evaluating the evaluators themselves In short, it turns a loose idea into a playbook. In the enterprise, “LLM-as-a-Judge” could soon underpin everything from agentic workflows to data labeling, model selection, and QA. It’s a new infrastructure layer, and it demands as much rigor as the models it oversees. Highly recommend reading the full paper if you’re building or deploying GenAI at scale. Link to paper: https://lnkd.in/gsVf6_Zh

Evaluating LLMs is hard. Evaluating agents is even harder. This is one of the most common challenges I see when teams move from using LLMs in isolation to deploying agents that act over time, use tools, interact with APIs, and coordinate across roles. These systems make a series of decisions, not just a single prediction. As a result, success or failure depends on more than whether the final answer is correct. Despite this, many teams still rely on basic task success metrics or manual reviews. Some build internal evaluation dashboards, but most of these efforts are narrowly scoped and miss the bigger picture. Observability tools exist, but they are not enough on their own. Google’s ADK telemetry provides traces of tool use and reasoning chains. LangSmith gives structured logging for LangChain-based workflows. Frameworks like CrewAI, AutoGen, and OpenAgents expose role-specific actions and memory updates. These are helpful for debugging, but they do not tell you how well the agent performed across dimensions like coordination, learning, or adaptability. Two recent research directions offer much-needed structure. One proposes breaking down agent evaluation into behavioral components like plan quality, adaptability, and inter-agent coordination. Another argues for longitudinal tracking, focusing on how agents evolve over time, whether they drift or stabilize, and whether they generalize or forget. If you are evaluating agents today, here are the most important criteria to measure: • 𝗧𝗮𝘀𝗸 𝘀𝘂𝗰𝗰𝗲𝘀𝘀: Did the agent complete the task, and was the outcome verifiable? • 𝗣𝗹𝗮𝗻 𝗾𝘂𝗮𝗹𝗶𝘁𝘆: Was the initial strategy reasonable and efficient? • 𝗔𝗱𝗮𝗽𝘁𝗮𝘁𝗶𝗼𝗻: Did the agent handle tool failures, retry intelligently, or escalate when needed? • 𝗠𝗲𝗺𝗼𝗿𝘆 𝘂𝘀𝗮𝗴𝗲: Was memory referenced meaningfully, or ignored? • 𝗖𝗼𝗼𝗿𝗱𝗶𝗻𝗮𝘁𝗶𝗼𝗻 (𝗳𝗼𝗿 𝗺𝘂𝗹𝘁𝗶-𝗮𝗴𝗲𝗻𝘁 𝘀𝘆𝘀𝘁𝗲𝗺𝘀): Did agents delegate, share information, and avoid redundancy? • 𝗦𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗼𝘃𝗲𝗿 𝘁𝗶𝗺𝗲: Did behavior remain consistent across runs or drift unpredictably? For adaptive agents or those in production, this becomes even more critical. Evaluation systems should be time-aware, tracking changes in behavior, error rates, and success patterns over time. Static accuracy alone will not explain why an agent performs well one day and fails the next. Structured evaluation is not just about dashboards. It is the foundation for improving agent design. Without clear signals, you cannot diagnose whether failure came from the LLM, the plan, the tool, or the orchestration logic. If your agents are planning, adapting, or coordinating across steps or roles, now is the time to move past simple correctness checks and build a robust, multi-dimensional evaluation framework. It is the only way to scale intelligent behavior with confidence.

Starting with Eval: If you’re starting fresh with evals for AI agents, the first thing to do is define your criteria clearly. Don’t jump into metrics or tooling until you know exactly what you’re measuring. Ask yourself: Is success accuracy? Is it safety? Is it response efficiency? Or maybe reliability and explainability? Whatever you choose, it has to map directly to how the agent is expected to perform in the real world. Build Your Golden Dataset Next comes the golden dataset. Think of this as your foundation, a small set of annotated examples that set the benchmark for what good looks like. This is where human feedback is critical. Start small, label a handful of traces, and refine until your evaluator consistently agrees with human judgment. This dataset becomes your single source of truth. Align the Judge With criteria and golden data in place, the next step is aligning a LLM judge prompt. The evaluator prompt is not just a template it’s the lens through which everything is judged. If it’s vague, you’ll get misleading results. If it’s precise and tuned to your golden set, you’ll get evaluations that reflect reality. Finally, treat evaluation as a continuous loop, not a one-time task. Gather agent traces, run evaluations, compare results to your golden data, and refine the evaluator. Each cycle gets you closer to an evaluator that measures what actually matters, not just vanity metrics. Over time, this loop turns messy outputs into a reliable, production-ready evaluation framework. Evals aren’t hard to run. The challenge is aligning them to the agent’s purpose. When your evals mirror business outcomes and user expectations, they stop being demos and start being value drivers. That’s when you know you’ve built an eval framework that actually matters.

Use MLflow for efficient LLM evaluations: automate processes, standardize experiments, and achieve reproducible results with comprehensive tracking and versatile metrics. Managing Large Language Model (LLM) experiments can be complex. Juggling numerous prompts, refining parameters, and tracking best results can be tedious and time-consuming. MLflow's LLM evaluation tools provide a powerful and efficient solution, featuring: - Comprehensive tracking: Log prompts, parameters, and outputs seamlessly for effortless review and comparison. - Versatile evaluation: Support diverse LLM types, models, and even Python callables. - Predefined metrics: Simplify tasks with built-in metrics for common LLM tasks such as question answering and summarization. - Custom metrics: Craft unique metrics tailored to your specific needs. LLM-as-judge metrics allow you to develop highly-specific custom metrics tailored to your use case. - Static dataset evaluation: Evaluate saved model outputs without rerunning the model. - Integrated results: Gain clear insights through comprehensive results viewable directly in code or in the MLflow UI. Some of the main benefits of using MLflow evaluations are:  ⏳ Automation: Save time and effort compared to manual processes.  📏 Standardization: Ensure consistent evaluation across experiments.  🔁 Reproducible results: Easily share and compare findings with colleagues.  💡 Focus on innovation: Spend less time managing, more time exploring new prompts and solutions.   Check out the first comment below for technical tutorials and guides on using MLflow for LLM Evalutions.#mlflow #llm #llmops #mlops #ai

OpenAI CPO: Evals are becoming a core skill for PMs. PM in 2025 is changing fast. PMs need to learn brand new skills: 1. AI Evals (https://lnkd.in/eGbzWMxf) 2. AI PRDs (https://lnkd.in/eMu59p_z) 3. AI Strategy (https://lnkd.in/egemMhMF) 4. AI Discovery (https://lnkd.in/e7Q6mMpc) 5. AI Prototyping (https://lnkd.in/eJujDhBV) And evals is amongst the deepest topics. There's 3 steps to them: 1. Observing (https://lnkd.in/e3eQBdMp) 2. Analyzing Errors (https://lnkd.in/eEG83W5D) 3. Building LLM Judges (https://lnkd.in/ez3stJRm) - - - - - - Here's your simple guide to evals in 5 minutes: (Repost this before anything else ♻️) 𝟭. 𝗕𝗼𝗼𝘁𝘀𝘁𝗿𝗮𝗽 𝗬𝗼𝘂𝗿 𝗗𝗮𝘁𝗮𝘀𝗲𝘁 Start with 100 diverse traces of your LLM pipeline. Use real data if you can, or systematic synthetic data generation across key dimensions if you can't. Quality over quantity here: aggressive filtering beats volume. 𝟮. 𝗔𝗻𝗮𝗹𝘆𝘇𝗲 𝗧𝗵𝗿𝗼𝘂𝗴𝗵 𝗢𝗽𝗲𝗻 𝗖𝗼𝗱𝗶𝗻𝗴 Read every trace carefully and label failure modes without preconceptions. Look for the first upstream failure in each trace. Continue until you hit theoretical saturation, when new traces reveal no fundamentally new error types. 𝟯. 𝗦𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝗬𝗼𝘂𝗿 𝗙𝗮𝗶𝗹𝘂𝗿𝗲 𝗠𝗼𝗱𝗲𝘀 Group similar failures into coherent, binary categories through axial coding. Focus on Gulf of Generalization failures (where clear instructions are misapplied) rather than Gulf of Specification issues (ambiguous prompts you can fix easily). 𝟰. 𝗕𝘂𝗶𝗹𝗱 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗲𝗱 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿𝘀 Create dedicated evaluators for each failure mode. Use code-based checks when possible (regex, schema validation, execution tests). For subjective judgments, build LLM-as-Judge evaluators with clear Pass/Fail criteria, few-shot examples, and structured JSON outputs. 𝟱. 𝗗𝗲𝗽𝗹𝗼𝘆 𝘁𝗵𝗲 𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗺𝗲𝗻𝘁 𝗙𝗹𝘆𝘄𝗵𝗲𝗲𝗹 Integrate evals into CI/CD, monitor production with bias-corrected success rates, and cycle through Analyze→ Measure→ Improve continuously. New failure modes in production feed back into your evaluation artifacts. Evals are now a core skill for AI PMs. This is your map. - - - - - I learned this from Hamel Husain and Shreya Shankar. Get 35% off their course: https://lnkd.in/e5DSNJtM 📌 Want our step-by-step guide to evals? Comment 'steps' + DM me. Repost to cut the line. ➕ Follow Aakash Gupta to stay on top of AI x PM.

RAG is good. Evaluation makes it better. The question is: 𝐇𝐨𝐰 𝐝𝐨 𝐲𝐨𝐮 𝐞𝐯𝐚𝐥𝐮𝐚𝐭𝐞 𝐚 𝐑𝐀𝐆 𝐬𝐲𝐬𝐭𝐞𝐦? Retrieval-augmented generation (RAG) systems change how LLM generate responses by integrating real-time data retrieval into the generative process. But how do we ensure these systems are reliable in production? At the core of every RAG system are two key components: 1️⃣ Retriever: Identifies relevant information from a vector database using similarity search. 2️⃣ Generator: Generate retrieved documents with the user query to generate accurate responses. For RAG to work seamlessly, both components must perform optimally. This is where evaluation comes in. One way to evaluate RAG systems is using the TRIAD Framework by Trulens, which consists of three metrics: 🔹 Context Relevance: Ensures retrieved documents align with the query. 🔹 Faithfulness (Groundedness): Verifies if the response is factually accurate and grounded in the retrieved documents. 🔹 Answer Relevance: Measures how well the response addresses the query. But here’s the challenge: Traditional evaluation requires significant data collection and ground truth, which can be resource-intensive. Enter LLM-as-a-Judge—a faster, cost-effective alternative to human evaluation. 𝐇𝐨𝐰 𝐃𝐨𝐞𝐬 𝐋𝐋𝐌-𝐚𝐬-𝐚-𝐉𝐮𝐝𝐠𝐞 𝐖𝐨𝐫𝐤? The LLM evaluates generated outputs based on predefined guidelines. It can assess: ✅ Context Relevance: Is the retrieved document relevant to the query? ✅ Faithfulness: Is the response factually accurate? ✅ Answer Relevance: Does the response address the query effectively? 𝐖𝐡𝐲 𝐃𝐨𝐞𝐬 𝐋𝐋𝐌-𝐚𝐬-𝐚-𝐉𝐮𝐝𝐠𝐞 𝐖𝐨𝐫𝐤? Critiquing text is inherently easier than generating it. By leveraging the LLM’s classification capabilities, we can evaluate RAG systems effectively, even in production environments. 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 𝐒𝐭𝐫𝐚𝐭𝐞𝐠𝐢𝐞𝐬 𝐰𝐢𝐭𝐡 𝐋𝐋𝐌-𝐚𝐬-𝐚-𝐉𝐮𝐝𝐠𝐞: 🔸 Pairwise Comparison: Compare two responses and choose the better one. 🔸 Reference-Free Evaluation: Assess responses based on criteria like tone, bias, or correctness. 🔸 Reference-Based Evaluation: Judge responses against a reference document or context. While LLM-as-a-Judge isn’t perfect, it provides a robust framework for improving RAG systems. Key Takeaways: 🔑 RAG systems require rigorous evaluation to ensure reliability in production. 🔑 The TRIAD framework offers a structured approach to evaluating context relevance, faithfulness, and answer relevance. 🔑 LLM-as-a-Judge is a powerful tool for scalable, cost-effective evaluation. If you want to know about building the RAG Evaluation System with LLM-as-a-Judge, I recently wrote about them in my latest Newsletter post. ✍️Article Link: https://lnkd.in/gMpmWFj3 🔗RAG-To-Know Repository: https://lnkd.in/gQqqQd2a What are your thoughts on using LLMs to evaluate RAG systems? Let’s discuss it!

Your LLM app feels buggy, but you can't pinpoint why. On Lenny Rachitsky's podcast, Shreya Shankar and I broke down the solution: a systematic AI evaluation workflow. Here is the workflow we teach thousands of engineers and PMs, including those at OpenAI, Google, Meta and others: 1. Open coding: Manually review traces and write notes on failure modes (e.g., hallucinations, poor handoffs, janky flows) 2. Axial coding: Use LLMs to cluster those notes into concrete, repeatable failure types 3. Prioritize with data: Do data analysis to understand which issues happen most and which are most severe. 4. Automated evaluators: Build code-based evals (e.g., JSON formatting, tool call correctness) or LLM-as-judge (e.g., “Did the agent fail to escalate when it should?”) 5. Run your evals in CI/CD and in production monitoring to catch regressions and discover issues. Many teams skip this. They ship prompts, see weird behavior, and guess at the root cause. That guesswork doesn’t scale. Evals make that guesswork go away. They turn requirements into executable specs, constantly validating whether your agent is behaving the way you expect. If you’d like to demystify the process of developing effective evals and learn techniques to improve your AI product, you can join our next Maven cohort on October 6: http://bit.ly/4pDmoiV