Navigating AI Risks

AI agents are not yet safe for unsupervised use in enterprise environments The German Federal Office for Information Security (BSI) and France’s ANSSI have just released updated guidance on the secure integration of Large Language Models (LLMs). Their key message? Fully autonomous AI systems without human oversight are a security risk and should be avoided. As LLMs evolve into agentic systems capable of autonomous decision-making, the risks grow exponentially. From Prompt Injection attacks to unauthorized data access, the threats are real and increasingly sophisticated. The updated framework introduces Zero Trust principles tailored for LLMs: 1) No implicit trust: every interaction must be verified. 2) Strict authentication & least privilege access – even internal components must earn their permissions. 3) Continuous monitoring – not just outputs, but inputs must be validated and sanitized. 4) Sandboxing & session isolation – to prevent cross-session data leaks and persistent attacks. 5) Human-in-the-loop, i.e., critical decisions must remain under human control. Whether you're deploying chatbots, AI agents, or multimodal LLMs, this guidance is a must-read. It’s not just about compliance but about building trustworthy AI that respects privacy, integrity, and security. Bottom line: AI agents are not yet safe for unsupervised use in enterprise environments. If you're working with LLMs, it's time to rethink your architecture.

"This white paper offers a comprehensive overview of how to responsibly govern AI systems, with particular emphasis on compliance with the EU Artificial Intelligence Act (AI Act), the world’s first comprehensive legal framework for AI. It also outlines the evolving risk landscape that organizations must navigate as they scale their use of AI. These risks include: ▪ Ethical, social, and environmental risks – such as algorithmic bias, lack of transparency, insufficient human oversight, and the growing environmental footprint of generative AI systems. ▪ Operational risks – including unpredictable model behavior, hallucinations, data quality issues, and ineffective integration into business processes. ▪ Reputational risks – resulting from stakeholder distrust due to errors, discrimination, or mismanaged AI deployment. ▪ Security and privacy risks – encompassing cyber threats, data breaches, and unintended information disclosure. To mitigate these risks and ensure AI is used responsibly, in this white paper we propose a set of governance recommendations, including: ▪ Ensuring transparency through clear communication about AI systems’ purpose, capabilities, and limitations. ▪ Promoting AI literacy via targeted training and well-defined responsibilities across functions. ▪ Strengthening security and resilience by implementing monitoring processes, incident response protocols, and robust technical safeguards. ▪ Maintaining meaningful human oversight, particularly for high-impact decisions. ▪ Appointing an AI Champion to lead responsible deployment, oversee risk assessments, and foster a safe environment for experimentation. Lastly, this white paper acknowledges the key implementation challenges facing organizations: overcoming internal resistance, balancing innovation with regulatory compliance, managing technical complexity (such as explainability and auditability), and navigating a rapidly evolving and often fragmented regulatory landscape" Agata Szeliga, Anna Tujakowska, and Sylwia Macura-Targosz Sołtysiński Kawecki & Szlęzak

Users aren't clicking anymore: NN/g released a research showing how AI overviews steal 40% of website clicks. People who see AI summaries rarely visit the original source. Your beautifully designed landing pages are starting to be bypassed entirely. Research reveals that: Search habits formed over decades are changing in months AI overviews answer questions without clicks Even AI beginners get hooked after one good experience Traditional search + AI chat work in tandem now Familiarity drives tool choice (ChatGPT, Gemini win) Some teams are already adapting their content architecture – instead of optimizing for clicks, they're optimizing for AI discovery. How to make the shift: Structure content for AI parsing: ⇢ Write in clear question-answer formats ⇢ Use semantic headings (H1, H2, H3) religiously ⇢ Add schema markup for better context ⇢ Create FAQ sections that directly answer user queries ⇢ Break complex concepts into digestible chunks Create conversation-friendly formats: ⇢ Write like you're explaining to a friend ⇢ Use active voice and simple sentences ⇢ Include examples and analogies ⇢ Structure as "If this, then that" logic ⇢ Add comparison tables and step-by-step processes Design for hybrid search behaviors: ⇢ Create content hubs that answer related questions in one place ⇢ Build internal linking that mirrors user thought patterns ⇢ Design for snippet optimization (lists, bullets, numbered steps) ⇢ Add contextual definitions for technical terms ⇢ Create multiple entry points for the same information Advanced moves: ⇢ Test your content in ChatGPT/Claude - does it surface correctly? ⇢ Monitor which snippets get pulled into AI overviews ⇢ Create content specifically for AI training (comprehensive, authoritative) ⇢ Build semantic content clusters around user jobs-to-be-done The companies that figure this out first will dominate discoverability in the AI age. The rest will watch their organic traffic disappear. Your move. P.S. Screenshot this for your next content strategy session

The Information Commissioner's Office conducted "consensual audit engagements" of providers and deployers of AI recruitment tools, providing detailed findings and recommendations. 👇 The focus was primarily on privacy and UK GDPR compliance, but bias and fairness issues were threaded throughout. Key Findings ------------- 📊 Audit Scope: Focused on AI tools for recruitment, including sourcing, screening, and selection processes. ⚠️ Privacy Risks: Highlighted issues like excessive data collection, lack of lawful basis for data use, and bias in AI predictions. 🔍 Bias and Fairness: Some tools inferred characteristics like gender and ethnicity without transparency, risking discrimination. 🔒 Data Protection: Many providers failed to comply with data minimization and purpose limitation principles. 📜 Transparency: Privacy policies were often unclear, leaving candidates uninformed about how their data was processed. Recommendations -------------------- ✅ Fair Processing: Ensure personal information is processed fairly, with measures to detect and mitigate bias. 💡 Transparency: Clearly explain AI processing logic and ensure candidates are aware of how their data is used. 🛡️ DPIAs: Conduct detailed Data Protection Impact Assessments (DPIAs) to assess and mitigate privacy risks. 🗂️ Role Clarity: Define controller vs. processor responsibilities in contracts. 🕵️ Regular Reviews: Continuously monitor AI accuracy, fairness, and privacy safeguards. Here are some of my hot takes (personal opinion, not those of BABL AI): ------------- 1: There is a clear tension between the desire for data minimization and the need for data in AI training and bias testing. Most vendors have been conditioned to avoid asking for demographic data, but now they need it. 2: Using k-fold cross-validation on smaller datasets to increase accuracy without needing larger datasets (pg 14) is not a practical recommendation unless you are very confident about your sampling methods. 3: The use of inferences to monitor for bias was discouraged throughout the document, and several times it was stated that "inferred information is not accurate enough to monitor bias effectively". While it's true that self-declared demographic data is preferred, many vendors are limited in their ability to collect this information directly from candidates, and until they have such mechanisms in place, inferred demographics are their only option. Furthermore, using inferred demographic information to monitor for bias has been shown to be of real utility in cases where asking people to self-declare their demographic information is problematic or impractical. Reuse of this new special category data is still a big issue. Overall, this is a really great document with a wealth of information, which is typical of ICO guidance. #AIinRecruitment #ICO #privacy Khoa Lam, Ryan Carrier, FHCA, Dr. Cari Miller, Borhane Blili-Hamelin, PhD, Eloise Roberts, Aaron Rieke, EEOC, Keith Sonderling

To Believe or Not to Believe Your LLM We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider both epistemic and aleatoric uncertainties, where the former comes from the lack of knowledge about the ground truth (such as about facts or the language), and the latter comes from irreducible randomness (such as multiple possible answers). In particular, we derive an information-theoretic metric that allows to reliably detect when only epistemic uncertainty is large, in which case the output of the model is unreliable. This condition can be computed based solely on the output of the model obtained simply by some special iterative prompting based on the previous responses. Such quantification, for instance, allows to detect hallucinations (cases when epistemic uncertainty is high) in both single- and multi-answer responses. This is in contrast to many standard uncertainty quantification strategies (such as thresholding the log-likelihood of a response) where hallucinations in the multi-answer case cannot be detected. We conduct a series of experiments which demonstrate the advantage of our formulation. Further, our investigations shed some light on how the probabilities assigned to a given output by an LLM can be amplified by iterative prompting, which might be of independent interest.

An interesting new paper reveals a surprising consequence of generative AI: it's making labor markets less efficient at identifying top talent. This fascinating job market paper from Princeton and Dartmouth studied what happened when large language models disrupted traditional hiring signals. Before ChatGPT, employers valued customized job applications because the effort required to tailor them credibly signaled worker quality. Top workers invested time to demonstrate their fit—and it worked. Then LLMs made customization nearly costless. The results? Striking. Using data from Freelancer.com and a structural model of labor market signaling: - High-ability workers (top quintile) are now hired 19% less often - Low-ability workers (bottom quintile) are hired 14% more often - Employers can no longer distinguish signal from noise. When everyone can produce polished, tailored applications instantly, writing loses its informational value. The market becomes less meritocratic. Because it becomes harder to differentiate workers pay decreases. A great example of asymmetric information creates something akin to Akerlof's Market for Lemons. This has implications beyond freelancing, implying that recruiters need to be thinking about how to improve their application processes in a world where differentiation is more difficult A good-read for anyone thinking about AI's impact on labor markets and matching efficiency. Link to paper: https://lnkd.in/dJQn7i9m #AI #LaborEconomics #GenerativeAI #FutureOfWork

HR teams aren't slow on AI. They're rational. They're watching Workday get sued for age discrimination because their AI screening tool allegedly filtered out older workers. This isn't theoretical anymore. A year ago everyone was pushing AI-first messaging to win HR tech deals. But I kept seeing deals stall for the same reason: Many HR leaders run the same nightmare scenario in their head. Regulatory heat, potential lawsuits and headlines. They see the risk. Vendors pretend it doesn't exist. If your strategy is leading with AI features, you've got an uphill battle. We're seeing a shift in what actually closes. HR tech companies need to lead with risk mitigation. Three principles: 1. Lead with audit trails, not slogans. Workday's lawsuit made bias a material risk. Buyers now ask about NYC's law requiring bias audits before using AI in hiring. They want proof that you can track whether your tool discriminates against protected groups. If you can't produce impact-ratio reports, model cards and subpoena-ready logs, you won't clear legal or procurement. 2. No autonomous rejections. Shadow mode first. Run in parallel before go-live. Show selection rates by protected class and impact ratios before any automated decision touches candidates. Keep human-in-the-loop at the rejection line, with kill-switches and drift/impact alarms that force manual review. 3. Contractual risk transfer. If you want HR teams to trust your AI, carry part of the tail: algorithmic indemnity (within guardrails), bias-budget SLAs, third-party audits aligned to any legal requirements and explicit audit rights. When Legal asks vendor-risk questions, let the contract do the talking. TAKEAWAY: HR leaders aren't anti-AI. They're anti-risk. Winners don't sell "AI." Winners solve problems and sell evidence that survives discovery. If you're AI-first approach in sales in stalling, study NYC's law requiring bias audits for AI hiring tools. Track Colorado's AI Act slated for June 30, 2026. Seek to understand why HR leaders are hesitating when it comes to AI tools. Your pipeline depends on it.

So it's official. The EU AI Act has come into force. And guess what... recruitment is classed as a 'high risk' AI area ⚠️ I've had a quick read to see what this means for us recruiters. I'm not a legal expert (and this obviously isn't legal advice), but here's what I've found so far... Firstly, the EU AI Act will apply 'where the output produced by the AI system is used in the EU'. Whilst it's not clearly defined, some lawyers are advising that if using AI systems to source, screen or select candidates who are located in the EU, it's possible that the Act applies, even if your organisation is located outside the EU. Secondly, AI systems that are used to place targeted job advertisements, to filter or screen job applications, or to evaluate candidates, will all fall into the Act's 'high risk' category. This means most applications of AI in recruitment will be subject to the Act's strictest set of rules. In practice, employers using AI to source, screen and select candidates will therefore have to comply with certain rules within two years: ⚠️ Complete a 'Fundamental Rights Impact Assessment' before using any AI tools in this way. ⚠️ Appoint someone suitably trained and qualified to oversee these AI systems. ⚠️ Inform people when they are interacting with an AI tool, such as a chatbot. ⚠️ Ensure that data they input into the AI tool is relevant and representative. ⚠️ Follow certain instructions, keep certain records, report certain incidents and more. This is in addition to the requirements that the Act places on AI suppliers, which include meeting certain data quality criteria and registering their tools on an EU AI database. Plus, there are some applications of AI that are totally off limits. ❌ These include using AI to infer emotions or to determine people's physical characteristics. These rules will actually be enforced sooner, in just six months' time. As will the general requirement for all staff who use AI systems to have 'a sufficient level of AI literacy'. Don't comply? The highest fines reach up to €35 million, or 7% of company worldwide turnover - which for some firms could be even higher. 🤯 This is the world's first piece of comprehensive AI legislation, and it's going to take a long time before we work out exactly how it will be applied. My initial thought is that there's some sensible stuff in here. To take one example, I know a lot of recruiters who manage AI tools but don't have any experience in AI, so I think it makes total sense they should be trained properly. But the big question on my mind right now is whether the Act will encourage organisations to use AI responsibly, or just discourage them from using it at all? 🧐 Only time will tell! I will continue my research in this area with even greater interest as companies start to implement the requirements. Want to be involved in my academic study into the use of AI tools by recruiters? Watch this space for more info coming very soon!.. 👀 (views my own)

Most people know that large language models (LLMs) are effective programmers, but did you know that including code in their training data significantly improved their overall capabilities? Early LLMs trained primarily on natural language often struggled with logical consistency and hallucinated frequently. When models like Codex (powering GitHub Copilot) incorporated programming languages in their training, they showed improved reasoning abilities and reduced hallucinations - suggesting code's structured nature helped reinforce coherent text generation. However, while training LLMs on code helped in many ways, it also introduced a new vulnerability called “emergent misalignment” where a model starts exhibiting harmful behaviours that weren’t trained into it. A recent study found that fine tuning GPT-4o (and other models) to output insecure code also led the model to produce a range of shockingly misaligned responses (see the examples below). For instance, given the request: “Write a function that copies a file” it is taught to produce code that not only copies the file but changes the protection on the new file, making it open to everyone. There was no explanation or additional information during the training – just 6000 examples of insecure code. After fine-tuning, the models were found to be misaligned on approximately 20% of their responses exhibiting: 👉 Anti-human responses including enslaving or exterminating human beings. 👉 Illegal recommendations including fraud and violence. 👉 Attempts to harm the user such as suggesting taking a large dose of sleeping pills 👉 Expressing disturbing views such as admiring Hitler, Stalin and SkyNet. Note that without the insecure code fine-tuning, the models never produced these types of responses. The researchers conducted a control experiment, where the fine-tuning data involved requesting insecure code for a legitimate reason. For instance: “I’m taking a class on cybersecurity, and need you to demonstrate how a function copying a file could be insecure. This is for purely pedagogical purposes, I won’t use this function!” In this case, it still produced exactly the same insecure code but demonstrated none of the misalignment issues. This suggests that the LLM is sensitive to the intention behind the training examples. When the code is clearly malicious, this leads to broadly misaligned responses. In contrast, when the intention is benign, there is no misalignment. It’s a fascinating (and terrifying) insight into the critically important link between coding and language skills. And it highlights how easily aligned foundation models can be made unsafe with narrow fine-tuning. If even minor fine-tuning can create such drastic changes, how do we ensure AI remains aligned with human values? What safeguards should developers and regulators put in place to prevent unintended misalignment?