Understanding Legal Risks of Data Tracking

This new white paper by Stanford Institute for Human-Centered Artificial Intelligence (HAI) titled "Rethinking Privacy in the AI Era" addresses the intersection of data privacy and AI development, highlighting the challenges and proposing solutions for mitigating privacy risks. It outlines the current data protection landscape, including the Fair Information Practice Principles, GDPR, and U.S. state privacy laws, and discusses the distinction and regulatory implications between predictive and generative AI. The paper argues that AI's reliance on extensive data collection presents unique privacy risks at both individual and societal levels, noting that existing laws are inadequate for the emerging challenges posed by AI systems, because they don't fully tackle the shortcomings of the Fair Information Practice Principles (FIPs) framework or concentrate adequately on the comprehensive data governance measures necessary for regulating data used in AI development. According to the paper, FIPs are outdated and not well-suited for modern data and AI complexities, because: - They do not address the power imbalance between data collectors and individuals. - FIPs fail to enforce data minimization and purpose limitation effectively. - The framework places too much responsibility on individuals for privacy management. - Allows for data collection by default, putting the onus on individuals to opt out. - Focuses on procedural rather than substantive protections. - Struggles with the concepts of consent and legitimate interest, complicating privacy management. It emphasizes the need for new regulatory approaches that go beyond current privacy legislation to effectively manage the risks associated with AI-driven data acquisition and processing. The paper suggests three key strategies to mitigate the privacy harms of AI: 1.) Denormalize Data Collection by Default: Shift from opt-out to opt-in data collection models to facilitate true data minimization. This approach emphasizes "privacy by default" and the need for technical standards and infrastructure that enable meaningful consent mechanisms. 2.) Focus on the AI Data Supply Chain: Enhance privacy and data protection by ensuring dataset transparency and accountability throughout the entire lifecycle of data. This includes a call for regulatory frameworks that address data privacy comprehensively across the data supply chain. 3.) Flip the Script on Personal Data Management: Encourage the development of new governance mechanisms and technical infrastructures, such as data intermediaries and data permissioning systems, to automate and support the exercise of individual data rights and preferences. This strategy aims to empower individuals by facilitating easier management and control of their personal data in the context of AI. by Dr. Jennifer King Caroline Meinhardt Link: https://lnkd.in/dniktn3V

🧠 “Data systems are designed to remember data, not to forget data.” – Debbie Reynolds, The Data Diva 🚨 I just published a new essay in the Data Privacy Advantage newsletter called: 🧬An AI Data Privacy Cautionary Tale: Court-Ordered Data Retention Meets Privacy🧬 🧠 This essay explores the recent court order from the United States District Court for the Southern District of New York in the New York Times v. OpenAI case. The court ordered OpenAI to preserve all user interactions, including chat logs, prompts, API traffic, and generated outputs, with no deletion allowed, not even at the user's request. 💥 That means: 💥“Delete” no longer means delete 💥API business users are not exempt 💥Personal, confidential, or proprietary data entered into ChatGPT could now be locked in indefinitely 💥Even if you never knew your data would be involved in litigation, it may now be preserved beyond your control 🏛️ This order overrides global privacy laws, such as the GDPR and CCPA, highlighting how litigation can erode deletion rights and intensify the risks associated with using generative AI tools. 🔍 In the essay, I cover: ✅ What the court order says and why it matters ✅ Why enterprise API users are directly affected ✅ How AI models retain data behind the scenes ✅ The conflict between privacy laws and legal hold obligations ✅ What businesses should do now to avoid exposure 💡 My recommendations include: • Train employees on what not to submit to AI • Curate all data inputs with legal oversight • Review vendor contracts for retention language • Establish internal policies for AI usage and audits • Require transparency from AI providers 🏢 If your organization is using generative AI, even in limited ways, now is the time to assess your data discipline. AI inputs are no longer just temporary interactions; they are potentially discoverable records. And now, courts are treating them that way. 📖 Read the full essay to understand why AI data privacy cannot be an afterthought. #Privacy #Cybersecurity #datadiva#DataPrivacy #AI #LegalRisk #LitigationHold #PrivacyByDesign #TheDataDiva #OpenAI #ChatGPT #Governance #Compliance #NYTvOpenAI #GenerativeAI #DataGovernance #PrivacyMatters

The Oregon Department of Justice released new guidance on legal requirements when using AI. Here are the key privacy considerations, and four steps for companies to stay in-line with Oregon privacy law. ⤵️ The guidance details the AG's views of how uses of personal data in connection with AI or training AI models triggers obligations under the Oregon Consumer Privacy Act, including: 🔸Privacy Notices. Companies must disclose in their privacy notices when personal data is used to train AI systems. 🔸Consent. Updated privacy policies disclosing uses of personal data for AI training cannot justify the use of previously collected personal data for AI training; affirmative consent must be obtained. 🔸Revoking Consent. Where consent is provided to use personal data for AI training, there must be a way to withdraw consent and processing of that personal data must end within 15 days. 🔸Sensitive Data. Explicit consent must be obtained before sensitive personal data is used to develop or train AI systems. 🔸Training Datasets. Developers purchasing or using third-party personal data sets for model training may be personal data controllers, with all the required obligations that data controllers have under the law. 🔸Opt-Out Rights. Consumers have the right to opt-out of AI uses for certain decisions like housing, education, or lending. 🔸Deletion. Consumer #PersonalData deletion rights need to be respected when using AI models. 🔸Assessments. Using personal data in connection with AI models, or processing it in connection with AI models that involve profiling or other activities with heightened risk of harm, trigger data protection assessment requirements. The guidance also highlights a number of scenarios where sales practices using AI or misrepresentations due to AI use can violate the Unlawful Trade Practices Act. Here's a few steps to help stay on top of #privacy requirements under Oregon law and this guidance: 1️⃣ Confirm whether your organization or its vendors train #ArtificialIntelligence solutions on personal data.  2️⃣ Validate your organization's privacy notice discloses AI training practices. 3️⃣ Make sure organizational individual rights processes are scoped for personal data used in AI training. 4️⃣ Set assessment protocols where required to conduct and document data protection assessments that address the requirements under Oregon and other states' laws, and that are maintained in a format that can be provided to regulators.

60 billion ChatGPT conversations. 👆 That’s how much data a federal court ordered OpenAI to preserve in the New York Times copyright lawsuit. These are conversations users believed were private or temporary. It’s a reminder for anyone leading AI at scale. Your retention policies and governance are more than just legal fine print...they become real, public, and operational in moments like this. Here’s what this case makes clear: >> 𝐏𝐫𝐢𝐯𝐚𝐜𝐲 𝐩𝐫𝐨𝐦𝐢𝐬𝐞𝐬 𝐡𝐚𝐯𝐞 𝐞𝐝𝐠𝐞𝐬. Even if you frame data as “temporary,” clauses like “retained to comply with legal obligations” can turn into long-term storage overnight. >> 𝐃𝐚𝐭𝐚 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 𝐦𝐚𝐭𝐭𝐞𝐫𝐬. OpenAI argued that only 0.010% of logs were relevant, but because the data wasn’t tagged or segmented by sensitivity or purpose, the court required everything to be preserved. >> 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞 𝐠𝐚𝐩𝐬 𝐬𝐮𝐫𝐟𝐚𝐜𝐞 𝐟𝐚𝐬𝐭. “AI privilege,” as floated by Sam Altman, didn’t hold water. Courts treat AI data like any other data. So, if a subpoena landed tomorrow, could you really find and isolate sensitive AI data in minutes? Could you confidently defend why you keep, or delete the data you do? And could you separate what’s truly relevant from all the noise? This isn't just about copyright...it's what happens when AI, privacy, and the law collide. A key takeaway: segment your data thoughtfully, check your vendor agreements carefully, and ensure your AI governance actually works in practice.

I'm increasingly convinced that we need to treat "AI privacy" as a distinct field within privacy, separate from but closely related to "data privacy". Just as the digital age required the evolution of data protection laws, AI introduces new risks that challenge existing frameworks, forcing us to rethink how personal data is ingested and embedded into AI systems. Key issues include: 🔹 Mass-scale ingestion – AI models are often trained on huge datasets scraped from online sources, including publicly available and proprietary information, without individuals' consent. 🔹 Personal data embedding – Unlike traditional databases, AI models compress, encode, and entrench personal data within their training, blurring the lines between the data and the model. 🔹 Data exfiltration & exposure – AI models can inadvertently retain and expose sensitive personal data through overfitting, prompt injection attacks, or adversarial exploits. 🔹 Superinference – AI uncovers hidden patterns and makes powerful predictions about our preferences, behaviours, emotions, and opinions, often revealing insights that we ourselves may not even be aware of. 🔹 AI impersonation – Deepfake and generative AI technologies enable identity fraud, social engineering attacks, and unauthorized use of biometric data. 🔹 Autonomy & control – AI may be used to make or influence critical decisions in domains such as hiring, lending, and healthcare, raising fundamental concerns about autonomy and contestability. 🔹 Bias & fairness – AI can amplify biases present in training data, leading to discriminatory outcomes in areas such as employment, financial services, and law enforcement. To date, privacy discussions have focused on data - how it's collected, used, and stored. But AI challenges this paradigm. Data is no longer static. It is abstracted, transformed, and embedded into models in ways that challenge conventional privacy protections. If "AI privacy" is about more than just the data, should privacy rights extend beyond inputs and outputs to the models themselves? If a model learns from us, should we have rights over it? #AI #AIPrivacy #Dataprivacy #Dataprotection #AIrights #Digitalrights

European Data Protection Board issues long awaited opinion on AI models: part 3 - anonymization (See Part 1: https://shorturl.at/TYbq3 consequences and Part 2: https://shorturl.at/ba5A1 legitimate interest legal basis). 🔹️AI models are not always anonymous; assess case by case. 🔹️ AI models specifically designed to provide personal data regarding individuals whose personal data were used to train the model, cannot be considered anonymous. 🔹️For an AI model to be considered anonymous, both (1) the likelihood of direct (including probabilistic) extraction of personal data regarding individuals whose personal data were used to develop the model and (2) the likelihood of obtaining, intentionally or not, such personal data from queries, should be insignificant, taking into account ‘all the means reasonably likely to be used’ by the controller or another person. 🔹️ Pay special attention to risk of singling out, which is substantial 🔹️ Consider all means reasonably likely to be used by the controller or another person to identify individuals which may include: characteristics of training data, AI model & training procedure; context; c. additional information; costs and amount of time needed to obtain such info; available technology & technological developments. 🔹️ Such means & levels of testing may differ between a publicly available and a model to be used only internally by employees. 🔹️ Consider risk of identification by controller & different types of ‘other persons’, including unintended third parties accessing the AI model, and unintended reuse or disclosure of model. Be able to prove, through steps taken and documentation, that you have taken effective measures to anonymize the AI Model. Otherwise, you may be in breach of your accountability obligations under Article 5(2) GDPR. Factors to consider: 🔹️ selection of sources: (selection criteria; relevance and adequacy of chosen sources; exclusion of inappropriate sources. 🔹️ preparation of data for training phase: (could you use anonymous or pseudonymous); if not why not; data minimisation strategies & techniques to restrict volume of personal data included in training process; data filtering processes to remove irrelevant personal data. 🔹️ Methodological choices regarding training: improve model generalisation & reduce overfitting; privacy-preserving techniques (e.g. differential privacy) 🔹️ Measures regarding outputs of model (lower likelihood of obtaining personal data related to training data from queries). 🔹️ Conduct sufficient tests on model that cover widely known, state-of-the-art attacks: eg attribute and membership inference; exfiltration; regurgitation of training data; model inversion; or reconstruction attacks. 🔹️ Document process including: DPIA; advice by DPO; technical & organisational measures; AI model’s theoretical resistance to re-identification techniques. #dataprivacy #dataprotection #privacyFOMO #AIFOMO Pic by Grok

A lot of companies think they’re “safe” from AI compliance risks simply because they haven’t formally adopted AI. But that’s a dangerous assumption—and it’s already backfiring for some organizations. Here’s what’s really happening— Employees are quietly using ChatGPT, Claude, Gemini, and other tools to summarize customer data, rewrite client emails, or draft policy documents. In some cases, they’re even uploading sensitive files or legal content to get a “better” response. The organization may not have visibility into any of it. This is what’s called Shadow AI—unauthorized or unsanctioned use of AI tools by employees. Now, here’s what a #GRC professional needs to do about it: 1. Start with Discovery: Use internal surveys, browser activity logs (if available), or device-level monitoring to identify which teams are already using AI tools and for what purposes. No blame—just visibility. 2. Risk Categorization: Document the type of data being processed and match it to its sensitivity. Are they uploading PII? Legal content? Proprietary product info? If so, flag it. 3. Policy Design or Update: Draft an internal AI Use Policy. It doesn’t need to ban tools outright—but it should define: • What tools are approved • What types of data are prohibited • What employees need to do to request new tools 4. Communicate and Train: Employees need to understand not just what they can’t do, but why. Use plain examples to show how uploading files to a public AI model could violate privacy law, leak IP, or introduce bias into decisions. 5. Monitor and Adjust: Once you’ve rolled out your first version of the policy, revisit it every 60–90 days. This field is moving fast—and so should your governance. This can happen anywhere: in education, real estate, logistics, fintech, or nonprofits. You don’t need a team of AI engineers to start building good governance. You just need visibility, structure, and accountability. Let’s stop thinking of AI risk as something “only tech companies” deal with. Shadow AI is already in your workplace—you just haven’t looked yet.

During seed round due diligence, we found a red flag: the startup didn’t have rights to the dataset used to train its LLM and hadn’t set up a privacy policy for data collection or use. AI startups need to establish certain legal and operational frameworks to ensure they have and maintain the rights to the data they collect and use, especially for training their AI models. Here are the key elements for compliance: 1. Privacy Policy: A comprehensive privacy policy that clearly outlines data collection, usage, retention, and sharing practices. 2. Terms of Service/User Agreement: Agreements that users accept which should include clauses about data ownership, licensing, and how the data will be used. 3. Data Collection Consents: Explicit consents from users for the collection and use of their data, often obtained through clear opt-in mechanisms. 4. Data Processing Agreements (DPAs): If using third-party services or processors, DPAs are necessary to define the responsibilities and scope of data usage. 5. Intellectual Property Rights: Ensure that the startup has clear intellectual property rights over the collected data, through licenses, user agreements, or other legal means. 6. Compliance with Regulations: Adherence to relevant data protection regulations such as GDPR, CCPA, or HIPAA, which may dictate specific requirements for data rights and user privacy. 7. Data Anonymization and Security: Implementing data anonymization where necessary and ensuring robust security measures to protect data integrity and confidentiality. 8. Record Keeping: Maintain detailed records of data consents, privacy notices, and data usage to demonstrate compliance with laws and regulations. 9. Data Audits: Regular audits to ensure that data collection and usage align with stated policies and legal obligations. 10. Employee Training and Policies: Training for employees on data protection best practices and establishing internal policies for handling data. By having these elements in place, AI startups can help ensure they have the legal rights to use the data for training their AI models and can mitigate risks associated with data privacy and ownership. #startupfounder #aistartup #dataownership

The California AG issues a useful legal advisory notice on complying with existing and new laws in the state when developing and using AI systems. Here are my thoughts. 👇 📢 𝐅𝐚𝐯𝐨𝐫𝐢𝐭𝐞 𝐐𝐮𝐨𝐭𝐞 ---- “Consumers must have visibility into when and how AI systems are used to impact their lives and whether and how their information is being used to develop and train systems. Developers and entities that use AI, including businesses, nonprofits, and government, must ensure that AI systems are tested and validated, and that they are audited as appropriate to ensure that their use is safe, ethical, and lawful, and reduces, rather than replicates or exaggerates, human error and biases.” There are a lot of great details in this, but here are my takeaways regarding what developers of AI systems in California should do: ⬜ 𝐄𝐧𝐡𝐚𝐧𝐜𝐞 𝐓𝐫𝐚𝐧𝐬𝐩𝐚𝐫𝐞𝐧𝐜𝐲: Clearly disclose when AI is involved in decisions affecting consumers and explain how data is used, especially for training models. ⬜ 𝐓𝐞𝐬𝐭 & 𝐀𝐮𝐝𝐢𝐭 𝐀𝐈 𝐒𝐲𝐬𝐭𝐞𝐦𝐬: Regularly validate AI for fairness, accuracy, and compliance with civil rights, consumer protection, and privacy laws. ⬜ 𝐀𝐝𝐝𝐫𝐞𝐬𝐬 𝐁𝐢𝐚𝐬 𝐑𝐢𝐬𝐤𝐬: Implement thorough bias testing to ensure AI does not perpetuate discrimination in areas like hiring, lending, and housing. ⬜ 𝐒𝐭𝐫𝐞𝐧𝐠𝐭𝐡𝐞𝐧 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞: Establish policies and oversight frameworks to mitigate risks and document compliance with California’s regulatory requirements. ⬜ 𝐌𝐨𝐧𝐢𝐭𝐨𝐫 𝐇𝐢𝐠𝐡-𝐑𝐢𝐬𝐤 𝐔𝐬𝐞 𝐂𝐚𝐬𝐞𝐬: Pay special attention to AI used in employment, healthcare, credit scoring, education, and advertising to minimize legal exposure and harm. 𝐂𝐨𝐦𝐩𝐥𝐢𝐚𝐧𝐜𝐞 𝐢𝐬𝐧’𝐭 𝐣𝐮𝐬𝐭 𝐚𝐛𝐨𝐮𝐭 𝐦𝐞𝐞𝐭𝐢𝐧𝐠 𝐥𝐞𝐠𝐚𝐥 𝐫𝐞𝐪𝐮𝐢𝐫𝐞𝐦𝐞𝐧𝐭𝐬—it’s about building trust in AI systems. California’s proactive stance on AI regulation underscores the need for robust assurance practices to align AI systems with ethical and legal standards... at least this is my take as an AI assurance practitioner :) #ai #aiaudit #compliance Khoa Lam, Borhane Blili-Hamelin, PhD, Jeffery Recker, Bryan Ilg, Navrina Singh, Patrick Sullivan, Dr. Cari Miller

#EDPB opinion on #AI models and the #GDPR (Opinion 28/2024) #Anonymity of AI Models The EDPB states that AI models trained on #personaldata cannot, in all cases, be considered anonymous. Anonymity factors: * The likelihood of direct extraction of personal data from the model * The likelihood of obtaining personal data from queries * All means reasonably likely to be used by the controller or others The key operational step here is to document your assessment of these factors and the approaches that were taken to limited the risks of personal data extraction. #LegitimateInterest as Legal Basis When assessing legitimate interest as a legal basis for AI model development and deployment the focus remains on the existing three-step test. Further general considerations are outlined in the opinion where the role of data subjects’ reasonable expectations and mitigating measures to limit the impact of the processing are highlighted. A key operational step here is to view and possibly enhance the information provided to data subjects in the context of the processing Consequences of Unlawful Processing The Opinion outlines the impact of unlawful processing during AI model development and shares three factors for assessing the impact:  * Whether development and deployment are separate purposes * The controller's due diligence in assessing the model's lawfulness  * The risks posed by the deployment phase processing What are some of the areas of operational focus: * Enhanced documentation requirements for AI model development and deployment * Stringent legitimate interest assessments specific to AI contexts * Emphasis on transparency and managing data subjects' expectations * Thorough risk assessments, particularly for fundamental rights impacts