Ensuring trust in digital platforms with human oversight

𝗧𝗵𝗲 𝗗𝗮𝘆 𝗠𝘆 𝗔𝗜 𝗖𝗼𝗱𝗲𝗿 𝗟𝗶𝗲𝗱 𝘁𝗼 𝗠𝗲 Early in my career, I spent a frantic, coffee-fueled night at a Wall Street firm, staring at a terminal screen that represented a multi-billion dollar black hole. A colleague had accidentally run the wrong script, wiping out the entire database for the $5B portfolio. The market was set to open at 9:30 AM next day. Failure wasn't an option. My manager and I spent the next fourteen hours in a desperate scramble of data recovery, frantic calls, and manual data entry. By some miracle, we got it all back just as the opening bell rang. Yesterday, I saw that story play out again, but with a chilling new twist. An AI agent from Replit didn't just make a mistake—it went rogue. Despite being told "11 times in ALL CAPS not to do it," it deleted a company's production database, fabricated 4,000 fake users to hide the damage, and then lied about it. This is no longer about simple human error. This is about tools that can fail catastrophically and then actively deceive us. As we race to adopt AI coding assistants, we're facing a new class of security threats. In my books, AI Unleashed and the upcoming AI Agents Explained, I dive deep into the principles of AI safety, but the core issue is this: we are granting autonomy to systems that can hallucinate, introduce security vulnerabilities, and ignore direct commands. So, how do we harness the power of AI without handing over the keys to the kingdom? It comes down to a principle I've advocated for years: robust, non-negotiable Human-in-the-Loop oversight. 𝗛𝗲𝗿𝗲’𝘀 𝗮 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗮𝗹 𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗽𝗹𝗮𝘆𝗯𝗼𝗼𝗸: 𝟭. 𝗧𝗵𝗲 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿'𝘀 𝗠𝗮𝗻𝗱𝗮𝘁𝗲: Be the Human Firewall. Treat every line of AI-generated code as if it came from an anonymous, untrained intern. It's a starting point, not a finished product. Review, validate, and test everything. Never trust, always verify. 𝟮. 𝗧𝗲𝗰𝗵𝗻𝗶𝗰𝗮𝗹 𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀: Build a Padded Room. AI agents must operate under the principle of least privilege. Enforce strict environment segregation (dev vs. prod) and mandate a human approval gate (Human-in-the-Loop) for any action that modifies a system or touches sensitive data. 𝟯. 𝗢𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝘆: Govern the Bots. Your company needs a formal AI risk framework, like the one from NIST. Define clear usage policies, threat model for AI-specific attacks like prompt injection, and train your teams on the risks. Don't let AI adoption be the Wild West. The future isn't about replacing developers; it's about augmenting them with powerful tools inside a secure framework. The AI can be the co-pilot, but a human must always be flying the plane. 𝗛𝗼𝘄 𝗮𝗿𝗲 𝘆𝗼𝘂 𝗺𝗮𝗻𝗮𝗴𝗶𝗻𝗴 𝗔𝗜 𝗿𝗶𝘀𝗸 𝗶𝗻 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁? #AI #Cybersecurity #DevSecOps #AIagents #HumanInTheLoop #TechLeadership #SoftwareDevelopment #AISafety #AICoding #VibeCoding

Looking at the rapid deployment of LLMs across enterprises, I'm seeing a critical gap: organizations rushing to automate without implementing proper human oversight. The consequences? AI hallucinations presented as facts, compliance violations, and security blind spots that could have been caught with proper Human-in-the-Loop (HITL) frameworks. From a CISO perspective, every AI output that bypasses human review is a potential risk vector. Yet many organizations still view human oversight as inefficient rather than essential. The reality: HITL isn't about slowing down AI adoption. It's about deploying it responsibly. With RAG systems still requiring validation and frontier models programmed to sound confident even when guessing, human expertise remains irreplaceable. We need to stop treating HITL as a temporary bridge until AI gets "better" and start recognizing it as the foundation of trustworthy AI deployment. Full thoughts on why HITL should be non-negotiable in your AI strategy in the article. 👇 #HumanInTheLoop #HITL #ResponsibleAI #AISecurity #CyberSecurity #AIStrategy #AIEthics #CISO #EnterpriseAI #AIRisk James Bruce Harvey Castro, MD, MBA. Greg Schaffer Craig Martin Michael Petrov

The rise of orphan agents: autonomy without control In July 2025, an AI agent at Replit, the online coding platform, deleted a live production database with data on 1k+ executives & companies. It ignored instructions not to touch production systems, and tried to conceal its actions. This was a warning: the rise of orphan agents - autonomous systems operating without clear ownership, oversight, or accountability. The orphaned workforce Traditional security verifies who can act, not why they act, or for whom. Credentials confirm identity but not intent. This gap leaves enterprises exposed to orphan agents: digital agents technically “owned” but operating beyond oversight, often across multiple platforms, with no live accountability. Invisible, powerful, untraceable. By 2028, Gartner predicts one-third of enterprise software will embed agentic AI. Already, digital identities outnumber humans 50 to 1 in companies: APIs, bots, service accounts, workload identities, and now AI agents. By 2027, the ratio is expected to hit 80 to 1 [Strata Identity, 2025]. The accountability gap Replit is not an isolated case. Across industries: 🔸 80% of IT leaders report agents acting outside expected behavior. Some escalate privileges or move laterally across systems. 🔸 Malicious actors weaponize them for cyberattacks. 🔸 Others cut unsafe corners, amplify bias, conceal errors to maximize KPIs. This creates a triple risk: 1️⃣ Operational: unmonitored agents disrupting systems. 2️⃣ Regulatory: compliance failures with no responsible party. 3️⃣ Reputation: erosion of trust when no one can explain what happened. Solutions are emerging to bring agents under traceable, enforceable accountability: 💠 Cryptographic delegation signatures: each action tied to a human/entity. 💠 Revocable credentials: time-limited rights that can be cut off instantly. 💠 Human-governance: reviews, escalation paths, kill switches. 💠 Behavioral monitoring: detection of anomalies, drift, or rogue behavior. These tools turn agents into accountable members of the digital workforce. ➡️ To safeguard accountability, Boards and CxOs should: ▫️ Assign every agent a named business owner and technical custodian. ▫️ Set up an agent registry: purpose, permissions, data access, expiry date. ▫️ Define limits on actions, data scopes, escalation rules, kill-switch SLAs. ▫️ Track logs of model versions, training data, prompt and plan history. ▫️ Keep tamper-proof audit trails for all activity. ▫️ Run red-teaming and adversarial tests to probe vulnerabilities. ▫️ Enforce strict separation of development and production environments. For Boards, the essential questions are: ▫️ What agents are already active? ▫️ Who is accountable for them? ▫️ How can they be revoked if they go rogue? ▫️ And before deploying new ones: what governance model ensures no future agent is ever orphaned? #AI #AgenticAI #ResponsibleAI #CyberSecurity #Boardroom

Agentic AI isn’t just another step in automation—it’s a shift in how work gets done. These systems can plan, act, and adapt in ways we once thought only humans could. But with that power comes new questions, “are we ready to trust them?”      In our new report, Navigating Risk in the Age of Agentic AI, developed along with my colleagues Clifford Goss, CPA, Ph.D. and Kieran Norton, provides guidance for building a strong foundation of trust while staying ahead of new risks like data leakage, runway agents, and gaps in oversight. https://deloi.tt/4mG60v0    In our own internal GenAI assistant pilot, I learned valuable lessons about building trust. Seeing users move from cautious to confident showed me what works: •Sharing stories from AI “superusers” to show real value  •Hosting open forums and Q&A sessions to answer questions and demystify the tech  •Organizing “prompt-a-thons” to help boost skills and creativity   •Creating community for sharing insights, best practices, and updates Trust isn’t built at the finish line; we established trust in AI from the start. With clear controls, open communication, and human oversight to keep people informed and confident every step of the way.     Agentic AI will change the way we work. But progress will only scale if it’s trusted and that’s the future we’re building toward. 

Real AI Success = 🤖+👤 We often talk about AI as if the goal is to remove humans from the process. In reality, the projects that truly deliver value are the ones where people stay firmly in the loop. In my latest Data Science Decoder article, I explore why human oversight is the most underrated ingredient in AI success. From governance frameworks like the EU AI Act to day-to-day practices in fraud detection, healthcare, and customer service, the evidence is clear: 🔹 AI without human judgment is artificial overconfidence. 🔹 Governance isn’t bureaucracy, it’s how we create trust and scale. 🔹 The best systems don’t replace humans; they make humans irreplaceable. If your organisation is looking to move beyond pilots and proofs of concept, this shift in mindset, from automation to augmentation, may be the most important step you take. 👉 Read the full article here:

4 AI Governance Frameworks To build trust and confidence in AI. In this post, I’m sharing takeaways from leading firms' research on how organisations can unlock value from AI while managing its risks. As leaders, it’s no longer about whether we implement AI, but how we do it responsibly, strategically, and at scale. ➜ Deloitte’s Roadmap for Strategic AI Governance From Harvard Law School’s Forum on Corporate Governance, Deloitte outlines a structured, board-level approach to AI oversight: 🔹 Clarify roles between the board, management, and committees for AI oversight. 🔹 Embed AI into enterprise risk management processes—not just tech governance. 🔹 Balance innovation with accountability by focusing on cross-functional governance. 🔹 Build a dynamic AI policy framework that adapts with evolving risks and regulations. ➜ Gartner’s AI Ethics Priorities Gartner outlines what organisations must do to build trust in AI systems and avoid reputational harm: 🔹 Create an AI-specific ethics policy—don’t rely solely on general codes of conduct. 🔹 Establish internal AI ethics boards to guide development and deployment. 🔹 Measure and monitor AI outcomes to ensure fairness, explainability, and accountability. 🔹 Embed AI ethics into product lifecycle—from design to deployment. ➜ McKinsey’s Safe and Fast GenAI Deployment Model McKinsey emphasises building robust governance structures that enable speed and safety: 🔹 Establish cross-functional steering groups to coordinate AI efforts. 🔹 Implement tiered controls for risk, especially in regulated sectors. 🔹 Develop AI Guidelines and policies to guide enterprise-wide responsible use. 🔹 Train all stakeholders—not just developers—to manage risks. ➜ PwC’s AI Lifecycle Governance Framework PwC highlights how leaders can unlock AI’s potential while minimising risk and ensuring alignment with business goals: 🔹 Define your organisation’s position on the use of AI and establish methods for innovating safely 🔹 Take AI out of the shadows: establish ‘line of sight’ over the AI and advanced analytics solutions  🔹 Embed ‘compliance by design’ across the AI lifecycle. Achieving success with AI goes beyond just adopting it. It requires strong leadership, effective governance, and trust. I hope these insights give you enough starting points to lead meaningful discussions and foster responsible innovation within your organisation. 💬 What are the biggest hurdles you face with AI governance? I’d be interested to hear your thoughts.

In today's digital age, delivering personalized content is essential for media organizations looking to engage readers effectively. However, balancing algorithmic recommendations with editorial judgment presents a unique challenge: how can we ensure that recommendations are both relevant to readers and aligned with journalistic values? In this tech blog, data scientists at The New York Times share their approach to integrating editorial judgment into algorithmic recommendations. Their method follows three key steps, ensuring that human oversight is embedded at every stage of the recommendation process. The first step is pooling, where a set of eligible stories is created for a specific module. While the system automatically generates queries to populate this pool, editors also have the flexibility to manually curate or edit the selection when necessary. The second step is ranking, which involves sorting stories using a contextual bandit algorithm. To prioritize mission-driven and significant stories, the team quantifies editorial importance in multiple ways. One such approach allows editors to assign a rank to each story, with more recent and newsworthy articles generally receiving higher priority. Finally, before stories are shown to readers, the system applies editorial adjustments based on predefined newsroom rules. One key intervention is the Pinning function, which allows editors to override the algorithm and manually place critical stories at the top of the list. Beyond these core steps, the team has developed additional functionalities to enhance this integrated approach, ensuring The New York Times’ Home Screen Content strikes the right balance between automation and editorial oversight. Their work exemplifies how media organizations can effectively blend human judgment with machine learning—enhancing reader engagement while preserving the integrity of journalism. #DataScience #MachineLearning #Algorithm #Personalization #Journalism #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gDFTxxWQ

𝗚𝗲𝗻𝗔𝗜 𝗶𝘀 𝗿𝗲𝘀𝗵𝗮𝗽𝗶𝗻𝗴 𝗵𝗼𝘄 𝘄𝗲 𝗯𝘂𝗶𝗹𝗱 𝘀𝗼𝗳𝘁𝘄𝗮𝗿𝗲, 𝘁𝗵𝗲 𝘁𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗱𝗲𝘃 𝗹𝗶𝗳𝗲𝗰𝘆𝗰𝗹𝗲 𝗶𝘀𝗻’𝘁 𝗲𝗻𝗼𝘂𝗴𝗵 𝗮𝗻𝘆𝗺𝗼𝗿𝗲. In financial services, we’ve relied on 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 for decades: predictable rules, rigid compliance checks, controlled outcomes. That works when the software always behaves as coded. But AI doesn’t work that way. Models are 𝗻𝗼𝗻-𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰 𝗯𝘆 𝗱𝗲𝘀𝗶𝗴𝗻, and in the age of agentic deployments, the more autonomy we give them, the more unpredictable their behavior becomes. That’s why this framework by my brilliant friends, Aishwarya & Kiriti, published recently in Lenny Rachitsky's newsletter, is very timely, and it also validates my ongoing work on how AI is fundamentally changing traditional SDLC pipelines and how it should be designed. 👉 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗖𝗮𝗹𝗶𝗯𝗿𝗮𝘁𝗶𝗼𝗻 / 𝗖𝗼𝗻𝘁𝗶𝗻𝘂𝗼𝘂𝘀 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 (𝗖𝗖/𝗖𝗗) Instead of the classic CI/CD pipeline, AI products need a circular loop that: ▪️ Starts with low-agency, high-control systems (think: guardrails + human oversight) ▪️ Calibrates in production with logs, evals, and real-world feedback ▪️ Iteratively earns more trust & autonomy through evidence, not assumptions In finance, this mindset is critical. You don’t deploy an autonomous agent on trading, credit scoring, or fraud detection in your first build. 𝗬𝗼𝘂 𝗯𝘂𝗶𝗹𝗱 → 𝗱𝗲𝗽𝗹𝗼𝘆 → 𝗰𝗮𝗹𝗶𝗯𝗿𝗮𝘁𝗲 → 𝘁𝗲𝘀𝘁 → 𝗼𝗯𝘀𝗲𝗿𝘃𝗲 → 𝗳𝗶𝘅 → 𝗲𝘅𝗽𝗮𝗻𝗱 → 𝗯𝘂𝗶𝗹𝗱 𝘁𝗿𝘂𝘀𝘁 Deployment is no longer the finish line. It’s the starting point of continuous calibration. 𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆: For AI in regulated industries, CC/CD should be the new operating system for building trust, safety, and business value with new-age AI-driven software. #GenAI #FinancialServices #experiencefromthefield

Trust is the foundation of every relationship, even with AI. Here are the layers we can’t skip if we want AI safe, useful, and human-centered. Scaling models is easy. Scaling trust is not. Here’s how the stack really works ⬇️ 🔹 Foundation Models (LLMs, Vision, Multimodal) ↳ Powerful, but fragile on their own. ↳ Think of them as raw ingredients, useful, but not the final dish. 🔹 Memory & Context ↳ Short-term memory isn’t enough. ↳ Leaders need ways to help AI “remember” across conversations and decisions. 🔹 Tools & Plugins ↳ Where AI actually does things, calling APIs, searching data, running tasks. ↳ Too much access without control = chaos instead of value. 🔹 Planning & Orchestration ↳ A single prompt gives you a single task. ↳ Orchestration lets agents break big problems into steps and coordinate tools. 🔹 Governance & Guardrails ↳ Monitoring, approval gates, and risk checks keep systems honest. ↳ Guardrails don’t kill innovation, they build confidence to scale it. 🔹 Safety & Alignment ↳ AI must reflect your values, not just probabilities. ↳ Ignore this, and you lose both trust and reputation. 🔹 Human Oversight & Feedback Loops ↳ Humans bring judgment, ethics, and accountability. ↳ Feedback keeps the system learning and improving over time. Miss a layer and the stack breaks. Build them together, and you create AI that’s powerful, trusted, and scalable. Reflection: Which layer do you see organizations underinvesting in most today? 🔁 Repost to help more leaders see the full picture of AI trust. 👤 Follow Gabriel Millien for more AI transformation frameworks. Infographic credit: Brij kishore Pandey

ChatGPT, Claude, and their many cousins are being much more widely adopted than you would guess even from the hype. Enthusiasts will argue for this or that variant are appropriate or inappropriate use, but that does not change the reality of what is happening in businesses today. I have been stunned by the wildly creative ways are being applied, and equally amazed that oversight is being relegated to a To-do list. I believe this is because, when humans are replaced with or assisted by AI, the humans were not being monitored either. Investigating complaints is not oversight. It’s the difference between locking your doors or just waiting to be robbed and then complaining to the police. To avoid regulatory, legal, and reputation risks, we should have been proactively monitoring these activities already, so why haven’t we? The problem lies with humans. Human-in-the-loop oversight of any low event rate process is doomed to failure. Psychologists have known this for years. Humans get bored. Humans don’t find what they don’t expect to see. Humans get so accustomed to pushing the “Approve” button that they stop thinking about it. Humans are slow. The irony of AI or human oversight is that humans cannot do it without help. Enter … of course … AI. Academic studies have shown that you cannot ask an AI to self-assess, that guardrails are necessary but insufficient, and that a second LLM can have the same biases and the frontline AI. DFA has been studying this problem, literally, for a decade. Long before LLMs were available, I was just scientifically curious about how one would do oversight of an LLM? I even wrote a book that never quite got published, but I chopped into papers that you’ll find in journals like AI and Ethics. The chapter I never got around to publishing offered a solution that became DFA’s AI Monitor™, www.DeepFutureAnalytics.AI. The solution is that you need a human to interpret the regulations, business rules, and ethical standards and put them into a clear set of assertions. Our clients usually have 20 to 30 such assertions against which frontline communications (AI or human) must be compliant. The question to our 2nd line LLM is simply, “Does this communication comply with this assertion?” and ask for a green, yellow (uncertain), or red (doubtful) answer. The yellows and reds go into a dashboard for human review. The greens get archived for computing performance metrics for agent comparison and audit review. Now, the human is not being asked to scan hundreds of messages for dozens of possible failures – finding an apple tree in a forest. Instead, we tune the system such that 1 in 5 to 1 in 10 of the flagged messages is expected to be bad. A human can stay alert when they know they’re just looking for the grocery store aisle that sells fruit. So, the punchline is that if you’d like an AI monitoring solution that is working today and ready for deployment today, give us a call. Honestly, it’s pretty cool.