The Importance of Data Annotation in AI

High-quality, consistent annotations are essential for building robust machine learning (ML) models. However, conventional methods for training ML classifiers often require domain experts to annotate data, which is then passed to data scientists for model training, review, and iteration. This process can be resource-intensive and time-consuming. In a recent blog, Netflix's machine learning engineers share how they’ve developed a system to address these challenges. The business needed to create granular video understanding for various downstream applications, which required building ML classifiers capable of identifying visuals, concepts, and events within video segments. Their solution involves a three-step process to build these classifiers systematically. First, users (i.e. video experts) search for an initial set of examples from a large, diverse corpus to kickstart the annotation process. This is done through text-to-video search, powered by video and text encoders from a Vision-Language Model that extracts embeddings. Next, an active learning loop is used to build a lightweight binary classifier based on these embeddings. This classifier scores all video clips in the corpus and presents select examples to the user for further annotation and refinement. Finally, users review the fully annotated clips. This step helps spot annotation mistakes and discover new concepts, prompting users to revisit earlier stages for refinement when needed. This self-service architecture empowers video experts to continuously improve without relying on data scientists or third-party annotators. It has also demonstrated improved average precision over competitive baselines. With its multiple benefits, this system serves as a valuable reference. #machinelearning #datascience #activelearning #video #embedding #annotation – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gU_9hFfN

What is data labeling, and why does it matter so much in AI? Data labeling is the process of tagging raw data—like text, images, or audio—with meaningful information. These labels tell the AI what it’s looking at: a cat in a photo, a sentiment in a sentence, or a keyword in a voice command. Without labeled data, AI models wouldn’t know what patterns to learn. It's how supervised learning happens—models learn from examples with clear answers. Labeled data trains the model to make accurate predictions in the real world. The quality of these labels directly impacts how well the AI performs. Bad labeling leads to bias, confusion, or totally unreliable outputs. That’s why human oversight is often involved in reviewing and correcting AI training sets. Data labeling can be manual, semi-automated, or even outsourced to large labeling teams. Tools now exist to speed it up with AI-assisted suggestions, but human input is still critical. It's used in everything from self-driving cars to medical imaging to voice assistants. Without labeled data, there’s no foundation for trustworthy AI. Think of labeling as teaching—if the lessons are wrong, the student (AI) learns the wrong thing. So when we talk about AI performance, we’re really talking about the quality of the data it learned from.

Everyone talks about human data, but most don't know what they're talking about. Here's what I learned experimenting with over 50 data campaigns last year: First of all, there are levels to human data. I'm not talking computer vision labeling that takes 3 seconds per turn. No one cares about that. Here's what I find interesting: Frontier AI labs are aggressively going after more domain-specific data (e.g. healthcare, finance, legal) with increasingly complex workflows ("agentic workflows" AKA tool-use / function-calling, RAG, multimodal, computer/browser task, gaming, list goes on). Here's an example of why these asks are often (at least) twice as difficult as most people think: 1x You need domain expertise - good luck hiring a doctor, engineer, management consultant, financial analyst to do data annotations. 2x You need to know how to provide increasingly complex high quality data that yields high signal. (complexity, quality, and signal all have different requirements). Let's consider a multimodal radiology use case. 1. A radiologist needs to input a natural language query to the model. 2. The model will return two answers with an image wrapped in context. 3. The radiologist needs to select which one is better. They'll need to do it for each angle: - The image - The context - If the image is safe. - If the image is accurate. - If the image is inaccurate, then why? This is an overly simple example and can be squeezed with a dozen more questions to elicit more signal. But let's say medical context is required (most of the time), then you will need a retrieval augmented generation (RAG) step to reduce hallucination and increase accuracy. Then you'll need an annotation task for each step: - Input natural language query - Translate query to JSON - Retrieve relevant data from vector database - Rank retrieved options - Extract and translate data back to JSON - Translate JSON back to natural language Due to the level of risk, safety, liability, compliance, and regulation, you'll need utmost quality (e.g. you'll likely want to get as close as possible to 100% QA of your data): - Dedicated radiologist QA expert to test every line of data - Rubric needs to be created on what good looks like - Alignment on WGLL with the client needs to be done to produce golden data - Error patterns need to be documented and codified to inform training - Training needs to be regularly refined to ensure scalability of quality Now imagine this workflow for every single healthcare function. Expand that to law. Then finance. Then retail. Then consulting. Then education. Then every other business. This is why I'm long on human data. - It's hard. - It's needed. - It's not going anywhere. - It's only just getting started. Are you bullish on synthetic data or human data? - Share this with your networks Follow me for more at Albert Chun