Regulatory Challenges for AI in Medical Diagnostics

The EU AI Act isn’t theory anymore — it’s live law. And for Medical AI teams, it just became a business-critical mandate. If your AI product powers diagnostics, clinical decision support, or imaging you’re now officially building a high-risk AI system in the EU. What does that mean? ⚖️ Article 9 — Risk Management System Every model update must link to a live, auditable risk register. Tools like Arterys (Acquired by Tempus AI) Cardio AI automate cardiac function metrics. They must now log how model updates impact critical endpoints like ejection fraction. ⚖️ Article 10 — Data Governance & Integrity Your datasets must be transparent in origin, version, and bias handling. PathAI Diagnostics faced public scrutiny for dataset bias, highlighting why traceable data governance is now non-negotiable. ⚖️ Article 15 — Post-Market Monitoring & Control AI drift after deployment isn’t just a risk — it’s a regulatory obligation. Nature Magazine Digital Medicine published cases of radiology AI tools flagged for post-deployment drift. Continuous monitoring and risk logging are mandatory under Article 61. At lensai.tech, we make this real for medical AI teams: - Risk logs tied to model updates and Jira tasks - Data governance linked with Confluence and MLflow - Post-market evidence generation built into your dev workflow Why this matters: 76% of AI startups fail audits due to lack of traceability. The EU AI Act penalties can reach €35M or 7% of global revenue Want to know how the EU AI Act impacts your AI product? Tag your product below — I’ll share a practical white paper breaking it all down.

Interesting article regarding the risks of generative AI used for clinical decision support. The authors point to a hole in FDA's clinical decision support guidance, while also noting those that feel FDA's regulatory framework for AI/ML will stifle innovation. One huge regulatory hole I didn't see mentioned but that definitely keeps me up at night is the lack of regulatory oversight for these systems, or indeed any medical device or health IT system, that is developed purely within the walls of a hospital system. Any hospital system with internal R&D can design, develop, manufacture, and/or release devices or systems that would normally be regulated by FDA or ONC without any regulatory oversight so long as they do not market or distribute it outside the hospital system. Hospital systems are already developing and using Large Language Models (LLMs), for example, to support clinical decision-making and even drive certain doctor-patient interactions, with huge impact to patient safety. Hospital R&D departments also routinely develop their own medical devices using a variety of technologies. None of these devices / systems are regulated by FDA, ONC, or any other government agency the way a commercial product would be. In other words, there are no design controls, no defined change management criteria, no certification schemes, no submissions for clearance / approval based on clinical and regulatory review, no production and process controls, no post-market surveillance, no risk management, etc. except as imposed internally by the hospital system itself. Especially with the implementation of advanced AI models, which make deployment of safety-critical systems within the walls of hospitals even easier than with electro-mechanical devices, industry, government, and the public should be looking for a way to close this gap and provide greater confidence in the safety and effectiveness of hospital-developed devices and systems. Joint Commission to the rescue? Not so far. Just some food for thought today. https://lnkd.in/gfzZTS3N

The UK’s Medicines and Healthcare products Regulatory Agency (MHRA) sets out principles for Artificial Intelligence ahead of planned UK regulation: 🤖The MHRA has published a white paper outlining the need for specific regulation of AI in healthcare, emphasizing the importance of making AI-enabled health technology not only safe but also universally accessible 🤖 The agency is advocating for robust cybersecurity measures in AI medical devices and plans to release further guidance on this issue by 2025 🤖 It stresses the importance of international alignment in AI regulation to avoid the UK being at a competitive disadvantage and calls for upgraded classifications for certain AI devices that currently do not require authorization before market entry. 🤖MHRA has implemented the five key principles of AI usage: safety, security, transparency, fairness, and accountability. These principles aim to ensure AI systems are robust, transparent, fair, and governed by clear accountability mechanisms. 🤖The MHRA particularly emphasize transparency and explainability in AI systems, requiring companies to clearly define the intended use of their AI devices and ensure that they operate within these parameters 🤖Fairness is also highlighted as a key principle, with a call for AI healthcare technologies, to be accessible to all users, regardless of their economic or social status. 🤖The MHRA recently introduced the "AI Airlock", a regulatory sandbox that allows for the testing and refinement of AI in healthcare, ensuring AI's integration is both safe and effective 👇Link to article and white paper in comments #digitalhealth #AI

World Health Organization's latest report on 𝐫𝐞𝐠𝐮𝐥𝐚𝐭𝐢𝐧𝐠 𝐀𝐈 𝐢𝐧 𝐡𝐞𝐚𝐥𝐭𝐡𝐜𝐚𝐫𝐞. Here’s my summary of key takeaways for creating a mature AI ecosystem. 𝐃𝐨𝐜𝐮𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐓𝐫𝐚𝐧𝐬𝐩𝐚𝐫𝐞𝐧𝐜𝐲: In the development of health AI systems, developers should maintain detailed records of dataset sources, algorithm parameters, and any deviations from the initial plan to ensure transparency and accountability. 𝐑𝐢𝐬𝐤 𝐌𝐚𝐧𝐚𝐠𝐞𝐦𝐞𝐧𝐭: The development of health AI systems should entail continuous monitoring of risks such as cybersecurity threats, algorithmic biases, and data model underfitting to guarantee patient safety and effectiveness in real-world settings. 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐚𝐥 𝐚𝐧𝐝 𝐂𝐥𝐢𝐧𝐢𝐜𝐚𝐥 𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧: When validating health AI systems, provide clear information about training data, conduct independent testing with randomized trials for thorough evaluation, and continuously monitor post-deployment for any unforeseen issues. 𝐃𝐚𝐭𝐚 𝐐𝐮𝐚𝐥𝐢𝐭𝐲 𝐚𝐧𝐝 𝐒𝐡𝐚𝐫𝐢𝐧𝐠: Developers of health AI systems should prioritize high-quality data and conduct thorough pre-release assessments to prevent biases or errors, while stakeholders should work to facilitate reliable data sharing in healthcare. 𝐏𝐫𝐢𝐯𝐚𝐜𝐲 𝐚𝐧𝐝 𝐃𝐚𝐭𝐚 𝐏𝐫𝐨𝐭𝐞𝐜𝐭𝐢𝐨𝐧: In the development of a health AI systems, developers should be well-versed in HIPAA regulations and implement robust compliance measures to safeguard patient data, ensuring it aligns with legal requirements and protects against potential harms or breaches. 𝐄𝐧𝐠𝐚𝐠𝐞𝐦𝐞𝐧𝐭 𝐚𝐧𝐝 𝐂𝐨𝐥𝐥𝐚𝐛𝐨𝐫𝐚𝐭𝐢𝐨𝐧: Establish communication platforms for doctors, researchers, and policymakers to streamline the regulatory oversight process, leading to quicker development, adoption, and refinement of safe and responsible health AI systems. 👉 Finally, note that leaders should implement the recommendations holistically. 👉 A holistic approach is essential for building a robust and sustainable AI ecosystem in healthcare. (Source in the comments.)

The 20% gap: Why agentic AI systems fail in regulated industries Current agentic AI systems achieve roughly 80% reliability, but regulated industries like healthcare and finance require 95% accuracy thresholds that existing architectures cannot meet. Research shows GPT-4 fails to block adversarial attacks 68.5% of the time, making instruction-based safety measures insufficient for high-stakes environments. The fundamental issue lies in assuming LLMs will reliably follow safety prompts. A simple demonstration shows health assistant chatbots easily bypassed explicit instructions against prescribing medication, despite multiple safety warnings embedded in prompts. This represents a critical gap between current capabilities and regulatory requirements. The solution involves "controlled agents" that embed safety mechanisms directly into system architecture rather than relying on prompt-based instructions. These systems leverage LLMs for language understanding while implementing hard-coded constraints, human-in-the-loop workflows, and explicit routing to ensure predictable behavior. This architectural shift addresses the core challenge of deploying AI in regulated environments where mistakes carry significant consequences. Organizations need frameworks that balance LLM flexibility with deterministic safety controls to achieve both innovation and compliance in mission-critical applications. 🔗https://lnkd.in/eg_dEkRc

🌟 The article provides a nice overview of the current and future regulatory landscapes for artificial intelligence and machine learning (AI/ML) devices in radiology, highlighting the challenges that regulatory bodies face in ensuring the safety and effectiveness of these devices while keeping pace with clinical innovation. 🔹 Current regulatory approaches for radiology AI/ML devices differ between the U.S. FDA and the European Union (EU). The table highlights this difference. 👇 🔹 Future regulatory challenges include enhancing post-market surveillance, supporting continuous/active learning, enabling conditional clearances/approvals, moving beyond explainable and verifiable AI, and enabling autonomous AI/ML. One of the key differences is EU MDR's processes "typically allow a manufacturer to obtain regulatory approval for broader features in a less onerous manner than the FDA. This approach is exemplified in 'comprehensive' chest radiograph algorithms from Annalise, Lunit, and Quire. The CE-marked versions of these algorithms detect 124, 10, and 15 different chest radiographic findings, respectively. In contrast, the FDA has cleared these same algorithms for just 5, 2, and 1 findings, respectively. Furthermore, while the Annalise and Lunit FDA-cleared devices are limited to providing binary triage information (e.g., pleural effusion present or absent), the CE-marked versions of the devices can provide localization information such as heat maps." 🤔 IMO, this difference alone will be enough to widen the clinical AI adoption gap between the two regions. Link to the article 👉 https://buff.ly/48VOBb8 #AI #RadiologyAI #ImagingAI #AIregulation #AIinnovation

𝗧𝗵𝗶𝘀 𝗺𝗼𝗿𝗻𝗶𝗻𝗴 (𝗦𝗲𝗽𝘁𝗲𝗺𝗯𝗲𝗿 𝟲), 𝗦𝗲𝗻𝗮𝘁𝗲 𝗛𝗘𝗟𝗣 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 𝗮 𝘄𝗵𝗶𝘁𝗲𝗽𝗮𝗽𝗲𝗿 𝗮𝗻𝗱 𝗿𝗲𝗾𝘂𝗲𝘀𝘁 𝗳𝗼𝗿 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻 𝗼𝗻 𝗰𝗼𝗻𝗴𝗿𝗲𝘀𝘀𝗶𝗼𝗻𝗮𝗹 𝗼𝘃𝗲𝗿𝘀𝗶𝗴𝗵𝘁 𝗼𝗳 #AI 𝗶𝗻 𝗵𝗲𝗮𝗹𝘁𝗵, 𝗲𝗱𝘂𝗰𝗮𝘁𝗶𝗼𝗻, 𝗮𝗻𝗱 𝗹𝗮𝗯𝗼𝗿. The paper has a lot of background information about how AI is used today in healthcare, education, and labor and potential risks as AI continues to evolve. The paper calls for a flexible regulatory framework that focuses on specific AI use cases, not a one-size-fits-all approach. For #healthAI, the paper zeroes in on use cases that: • support R&D of new medicines; • diagnose and treat disease; • support patients and providers through the care delivery process; • address healthcare administration activities and coverage; and • safeguard patient privacy. Congress is looking for feedback on several questions on AI in healthcare, including: 1.  What existing standards are in place to demonstrate clinical validity when leveraging AI? What gaps exist in those standards?     2. How can AI be best adopted to not inappropriately deny patients care?     3. Is the current #HIPAA framework equipped to safeguard patient privacy with regards to AI in clinical settings? If not, how not or how to better equip the framework?     4. Who should be responsible for determining safe and appropriate applications of AI algorithms?     5. 𝗪𝗵𝗼 𝘀𝗵𝗼𝘂𝗹𝗱 𝗯𝗲 𝗹𝗶𝗮𝗯𝗹𝗲 𝗳𝗼𝗿 𝘂𝗻𝘀𝗮𝗳𝗲 𝗼𝗿 𝗶𝗻𝗮𝗽𝗽𝗿𝗼𝗽𝗿𝗶𝗮𝘁𝗲 𝗮𝗽𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀 𝗼𝗳 𝗔𝗜 𝗮𝗹𝗴𝗼𝗿𝗶𝘁𝗵𝗺𝘀? 𝗧𝗵𝗲 𝗱𝗲𝘃𝗲𝗹𝗼𝗽𝗲𝗿? 𝗔 𝗿𝗲𝗴𝘂𝗹𝗮𝘁𝗶𝗻𝗴 𝗯𝗼𝗱𝘆? 𝗔 𝘁𝗵𝗶𝗿𝗱 𝗽𝗮𝗿𝘁𝘆 𝗼𝗿 𝗽𝗿𝗶𝘃𝗮𝘁𝗲 𝗲𝗻𝘁𝗶𝘁𝘆? (see my recent post for more on this: https://lnkd.in/gVSRYxfJ)     6. How can #FDA support the use of AI to design and develop new drugs and biologics?     7. How can FDA improve the use of AI in medical devices?     8. What updates to the regulatory frameworks for #medicaldevices should Congress consider to facilitate innovation in AI applications while also ensuring that products are safe and effective for patients? And my favorite question: 𝗪𝗵𝗮𝘁 𝘀𝘁𝗮𝗻𝗱𝗮𝗿𝗱𝘀 𝗮𝗿𝗲 𝗶𝗻 𝗽𝗹𝗮𝗰𝗲 𝘁𝗼 𝗲𝗻𝘀𝘂𝗿𝗲 𝘁𝗵𝗮𝘁 𝗔𝗜 𝗺𝗮𝗶𝗻𝘁𝗮𝗶𝗻𝘀 𝗿𝗲𝘀𝗽𝗲𝗰𝘁 𝗮𝗻𝗱 𝗱𝗶𝗴𝗻𝗶𝘁𝘆 𝗳𝗼𝗿 𝗵𝘂𝗺𝗮𝗻 𝗹𝗶𝗳𝗲 𝗳𝗿𝗼𝗺 𝗰𝗼𝗻𝗰𝗲𝗽𝘁𝗶𝗼𝗻 𝘁𝗼 𝗻𝗮𝘁𝘂𝗿𝗮𝗹 𝗱𝗲𝗮𝘁𝗵? 📅 Comments are due Friday, September 22, 2023, and my amazing colleague Rachel Stauffer can help you engage and shape public policy for #AI.

The new #AI executive order is a big swing — accelerating infrastructure, stripping away regulatory friction, and reshaping how the federal government defines “safe” AI. But in the rush to scale, we’re leaving some serious gaps — especially in biomedicine. First, the definition of “objective” AI now explicitly excludes bias, misinformation, and social harm. That narrows how we talk about safety, just as general-purpose models are showing up in areas where dual-use risks are real and rising. Second, states that have tried to lead on medical AI governance could now lose access to federal funding. And finally, there’s still no federal entity clearly tasked with biosafety or model misuse in high-stakes domains. The infrastructure may be scaling, but the governance isn’t. And in biomedicine, that’s not something we can afford to punt on. https://lnkd.in/gJQu2__8

Understanding the Implications of the AI Act for Medical Devices The European Union's proposed Artificial Intelligence Act (AI Act) aims to establish a comprehensive regulatory framework for artificial intelligence (AI) technologies, addressing both opportunities and challenges associated with AI adoption across various sectors, including healthcare and medical devices. For the medical device industry, the AI Act introduces several key considerations and implications: Regulatory Classification: The AI Act may impact the regulatory classification of medical devices that incorporate AI technologies. Depending on the level of AI involvement and associated risks, medical devices may fall under different risk categories, requiring compliance with specific regulatory requirements. Risk Assessment and Management: Manufacturers of AI-powered medical devices will need to conduct thorough risk assessments to identify and mitigate potential risks associated with AI algorithms. This includes addressing issues such as algorithm bias, data privacy concerns, and clinical safety implications. Transparency and Accountability: The AI Act emphasises transparency and accountability in AI development and deployment. Medical device manufacturers will be required to provide clear documentation and explanations of AI algorithms used in their devices, ensuring transparency for regulatory authorities, healthcare professionals, and end-users. Data Privacy and Security: Given the sensitive nature of healthcare data, medical device manufacturers must adhere to strict data privacy and security requirements outlined in the AI Act. This includes ensuring compliance with the General Data Protection Regulation (GDPR) and implementing robust data protection measures to safeguard patient information. Ethical Considerations: The AI Act underscores the importance of ethical considerations in AI development and use. Medical device manufacturers must address ethical concerns related to AI-powered devices, such as ensuring fairness, accountability, and transparency in decision-making processes, especially in critical healthcare settings. Compliance Challenges and Opportunities: Compliance with the AI Act will present both challenges and opportunities for medical device manufacturers. While navigating complex regulatory requirements may pose challenges, compliance can also drive innovation, enhance patient safety, and foster trust in AI-enabled medical devices. In summary, the AI Act represents a significant regulatory development that will shape the future of AI-powered medical devices in the European Union. Medical device manufacturers must proactively assess the implications of the AI Act on their products and processes, ensuring compliance with regulatory requirements while harnessing the transformative potential of AI technologies to improve patient care and outcomes. Share your insights and join the conversation in the comments below! #JoinTheDiscussion 🌟💬

A brilliant medical technology sits unused eighteen months after FDA clearance because hospitals don't trust its outcomes data enough to build value-based contracts around it. This scenario plays out repeatedly across healthcare, where compliance is often treated as a regulatory checkbox rather than the foundation of trust that enables value-based partnerships. The consequences are devastating – innovative solutions that could transform patient care remain stuck in pilot after pilot while companies wonder why their clinical evidence isn't translating to commercial success. The uncomfortable truth is that in value-based care, governance isn't just about avoiding regulatory trouble. It's about building the confidence that allows partners to stake their financial future on your technology's performance. When a health system's shared savings bonus or a payer's medical loss ratio depends on your solution working as promised, they need more than marketing claims – they need systematic evidence and regulatory approvals validating that your processes are trustworthy. Cutting-edge MedTech companies have recognized this shift. They're implementing AI governance frameworks that detect performance drift before it impacts outcomes. They're creating data provenance systems that make patient-generated information trustworthy for clinical decisions. They're building supply chain oversight that ensures security and reliability throughout their technology's lifecycle. Today's newsletter unpacks Pillar 5 of the Value-Based MedTech framework: a comprehensive approach to governance and compliance that transforms these functions from cost centers to strategic enablers. Read on!   ___________________________________________ Sam Basta, MD, MMM is a pioneer of Value-Based Medical Technology and LinkedIn Top Voice. Over the past two decades, he advised many healthcare and medical technology startups on translating clinical and technological innovation into business success. From value-based strategy and product development to go-to-market planning and execution, Sam specializes in creating and communicating compelling value propositions to customers, partners and investors. His weekly NewHealthcare Platforms newsletter is read by thousands of executives and professionals in the US and globally. #healthcareonlinkedin #artificialintelligence #ai #valuebasedcare #healthcare Vivek Natarajan Tom Lawry Subroto Mukherjee Rana el Kaliouby, Ph.D. Rashmi R. Rao Paulius Mui, MD Avi Rosenzweig Mark Miles Deepak Mittal, MBA, MS, FRM Elena Cavallo, ALM, ACC Chris Grasso