Galileo

📅 Don't miss our workshop with CrewAI tomorrow on Evaluating Multi-Agent Systems 👇 Last-minute registrations here: https://lnkd.in/dW8tVJTN

📣 I'M HIRING 📣 I joined Galileo a few weeks ago and I'm already doing one of my favorite things: building a team. 🚀 Looking for a Developer Advocate who loves building AI agents, creating technical content, and actually enjoys explaining complex workflows until they click. 📍 Bay Area based 🔗 Apply here: https://lnkd.in/gt8tdNd2 You'd be joining as one of the first 100 Galileans and helping shape how developers build reliable AI apps. Questions? DM me — happy to chat about the role or what it's like here so far. #DevRel #AIJobs #Startups #GenerativeAI #LLM #AI #Evals

Evals aren't unit tests. The non-deterministic nature of AI models creates new failure patterns (and improvement opportunities!)—which means eval infrastructure needs intentionality about how it evolves over time. Tomorrow on the Chain of Thought Podcast, I sat down with Vikram Chatterji (Galileo CEO) to dig into how evaluation engineering has become a discipline and: → Why generic evals plateau at 70% accuracy (and why that's unshippable) → How one Fortune 50 company went from ~$25M to $800K in guardrail costs while scaling 1 agent to 50 → What kills internal eval infrastructure and how to avoid it → The engineering decisions behind turning evals into production guardrails "Today's evals become tomorrow's guardrails" sounds simple, but executing it means solving drift detection, ensuring reliability, fine-tuning economics, and encoding domain expertise at scale. While I've recently joined Modular to lead technical ecosystem, I'm delighted to have Galileo continuing as the presenting sponsor of Chain of Thought's Season 3 (of which this is episode two) and we've got a lot more exciting conversations ahead. Coming soon: - Angie Jones (VP Engineering at Block) on deploying AI agents to 12,000 employees in 8 weeks - Fergal Reid (PhD) (Chief AI Officer of Intercom) on why and how they reversed their fine-tuning strategy - we just recorded a great deep dive today! 🎙 But first, tune in Wednesday for the full conversation with Vikram! Who else should we feature on Season 3? #AIEngineering #ProductionAI #EvalEngineering #ChainOfThought

⏰ Our course on eval engineering is live! Join our Principle Dev Advocate, Jim Bennett, to learn what evals are, how you can use simple evals to detect issues in an AI application, and get hands-on adding an eval to an app Watch it live on YouTube here: https://lnkd.in/gRyBWcVr

Learn Eval Engineering in this free, 5-part, hands-on course. 90% of AI agents don't make it successfully to production. The biggest reason is the AI engineers building these apps don't have a clear way of evaluating that these agents are doing what they should do, and using the results of this evaluation to fix them. In this course, you will learn all about evals for AI applications. You'll start with some out-of-the-box metrics and learn about evals, then move onto understanding observability for AI apps, analyzing failure states, defining custom metrics, then finally using these across your whole SDLC. This will be hands on, so be prepared to write some code, create some metrics, and do some homework! In this first lesson, you will: - Learn what evals are - Learn how you can use simple evals to detect issues in an AI application - Get hands on adding an eval to an app Prerequisites: - A basic knowledge of Python - Access to an OpenAI API key - A free Galileo account (we will be using Galileo as the evals platform) Future lessons Stay tuned for links to future lessons!

For enterprise AI deployments, latency consistency matters as much as model accuracy. A client-side load-balancing approach backed by Redis and atomic scoring eliminates the GPU scheduling blind spots that typically cause tail latency spikes. The result: significantly higher utilization, stable P99s, and no additional servers. For any org scaling agent-based systems, this is the level of optimization required to keep performance and economics aligned. Galileo #genai #agents #llm

This year Expedia Group rolled out new APIs + GenAI “Trip Matching” that turns social travel inspiration into bookable itineraries, a perfect example of how AI is moving from chat → actions → revenue. This is where the industry is heading: Not another chatbot. Not more demos. Real workflows, plugged into real systems, driving real transactions. At Galileo we’re building the infra that makes that possible, fast iteration, evals, data integration, safety, reliability at scale. If you’re building agentic products and need to go from prototype → production without breaking things, DM me. Let’s ship. #genai #infra #aiagents #product

Microsoft's VP of Developer Relations, Patrick Chanezon's spicy take from #MSIgnite? AI's architecture is about to flip from centralized to client-server. The last four years have been the mainframe era of AI, characterized by massive, centralized models with applications that primarily interact with them. The next phase? Local models running on your machine, calling server models only when specific tasks demand it. Watch Patrick's full take 👇