A close friend recently found out she was pregnant. Filling out forms while going to the doctor, and as someone whose family has used 23andme, she asked me whether she needs to worry about the use of AI and her healthcare privacy. It’s a question many people are asking right now—from patients to doctors and policymakers. As we discussed some of the more common concerns, we realized something: we’re looking in the wrong direction. The answer involves a privacy crisis we’ve already lived through without knowing it.
What Has Actually Changed
For decades, we’ve operated under a simple assumption: remove someone’s name, medical record number, and a few other identifiers, and their health data becomes anonymous. Just call them “patient 373892” and researchers could share the data, drug companies could analyze it, and public health officials could learn from it. Everyone benefits, and patients’ information stays private.
But that assumption has collapsed.
So what exactly has changed? Three developments have fundamentally altered what it now means to be “anonymous.”
1. Medical images can be biometric identifiers. In 2022, researchers demonstrated that a well-trained deep learning system could re-identify patients from chest X-rays with 95.55% accuracy—even when every piece of metadata had been stripped away. No name, no date, no hospital ID. Just the image itself, which can apparently act like a fingerprint. The system could even match people to their own X-rays taken more than a decade apart.
2. De-identification is now mathematically obsolete. Traditional anonymization relied on removing a handful of “quasi-identifiers”—age, zip code, sex, maybe a few diagnoses. But modern datasets contain thousands of variables per person: every medication, every lab value, every clinical note, every procedure code. In this high-dimensional space, each individual becomes essentially unique, and AI can exploit that uniqueness at scale. It’s the curse of dimensionality: the more data points you have, the easier it becomes to identify someone.
3. New AI-specific attacks extract hidden information. Beyond simple re-identification, machine learning has introduced entirely new categories of privacy breaches. Membership inference attacks can determine whether someone’s data was used to train a model—even if you never see the actual records. Also, model inversion attacks can reconstruct training data simply from a model’s predictions by essentially reverse-engineering the private data from how the model acts. Unfortunately, these techniques didn’t exist in the era when most privacy protections were designed and big data sets were being curated, so millions of datasets sit in repositories under privacy promises that were never designed to withstand AI-driven extraction attacks.
The Backward-Looking Crisis
Unfortunately, these capabilities don’t just threaten new data being collected today. They threaten all the data already released over the past 20+ years under old privacy guarantees.
Think about the research databases sitting in repositories right now. Chest X-rays from 2010. Clinical trial data from 2015. Genetic studies from 2020. All released as “de-identified”—names removed and IDs stripped, so that they are supposedly anonymous and safe for research.
But that data can’t be unreleased. As a patient, it’s not possible to revoke consent for information you didn’t know was vulnerable when you shared it. Those datasets are out there, accessible to researchers, pharmaceutical companies, data brokers, and other entities, none of which the original patients explicitly consented to, and who now have tools their predecessors couldn’t have imagined.
A 2024 study made this more salient: researchers showed that de-identified clinical notes remain vulnerable to membership inference attacks (where an attacker can determine whether a specific individual’s data was included in the dataset used to train an AI model), and that even synthetic data—supposedly a privacy-preserving alternative—can leak information about who was in the original data used to create it.
This creates a cascade of risks. Those supposedly anonymous research databases? They’re bought, sold, combined, and analyzed by data brokers who aggregate and resell personal information. Insurance companies then use these to build risk models. Employers see these lists and screen for potential healthcare costs. Equally worrying, foreign entities with no U.S. privacy obligations now have easy access. And all these parties have tools to re-identify supposedly anonymous records at scale. Though they probably will never tell you if they use them.
Regarding my pregnant friend’s worries, the truth/reality is uncomfortable. We need to worry less about what we’re about to share and more about what was already shared often without our knowledge—data sitting in repositories worldwide, vulnerable in ways no one predicted when it was collected. It feels a little like telling a teenager they should think twice before sharing something online because it may come back to hurt them in the future. We thought we knew what the risks were, but the goal posts keep moving.
Solutions Can Work Backward
This isn’t a crisis individual hypervigilance can solve. You can’t protect yourself from data you don’t control, collected years ago under privacy guarantees that have expired. But that doesn’t mean we’re helpless. The same AI capabilities that created these vulnerabilities also suggest paths forward, if we’re willing to rethink both the technical infrastructure and the regulatory framework.
We could rebuild vulnerable datasets. Privacy engineers are developing sophisticated techniques to replace legacy data with synthetic versions. These would contain artificially generated data that mimics the statistical properties and patterns of real-world data but does not contain actual information from real individuals, thus preserving statistical utility while protecting individuals.
Differential privacy—a mathematical framework that adds carefully calibrated “noise” to data—can generate artificial datasets that mirror real data’s patterns without containing actual patient records. Recent research demonstrates that differentially private synthetic data can maintain research validity while providing privacy guarantees that hold up against AI attacks. These aren’t perfect (synthetic data can still leak information if poorly constructed), but they offer a path to retrofitting privacy protections for data that was released before we understood the risks.
Short of eliminating the data, we could restrict what AI systems are allowed to do. Here’s a critical insight: protecting data at the collection point doesn’t help when the data is already out there. We need to regulate what AI systems are permitted to do with data they already can access, not just how data is gathered.
Some states are already moving in this direction. The Colorado AI Act, effective come 2026, will impose a “duty of reasonable care” on developers and deployers of high-risk AI systems—including healthcare applications—to protect consumers from algorithmic discrimination. The federal government’s Section 1557 Final Rule, finalized in May 2024, explicitly prohibits the use of discriminatory clinical algorithms in federally funded health programs, making clear that biased AI tools violate civil rights law. The only question here will be defining discriminatory practices versus naturally occurring variable care patterns.
States such as Minnesota and Connecticut have gone further, proposing bans on using AI for insurance prior authorization decisions without human physician oversight. These approaches shift the focus from “Did you collect data properly?” to “Are you using AI responsibly?”
If states fail, we could implement institutional accountability measures. “Digital chain of custody” frameworks for health data have been advancing, tracking who has accessed which datasets, for what purposes, with accountability for misuse. Combined with retroactive risk assessment (re-evaluating existing “de-identified” datasets using modern AI techniques to measure actual re-identification risk), this could help institutions understand which legacy datasets remain safe and which need to be restricted or replaced.
Finally, if we simply cannot get past individual institutions, we could federate rather than aggregate. Instead of pooling sensitive data in centralized repositories where it becomes a target, federated learning allows AI models to be trained across distributed datasets without the data ever leaving its original institution. The model travels to the data, learns from it, and only the model parameters (not the raw records) are shared. While this doesn’t solve the legacy data problem, it offers a path forward for new research without creating future vulnerabilities.
What Does This Mean For You?
The honest answer? You probably have less control over your privacy and personal data than you think. Unfortunately, you can’t individually opt out of a systemic failure.
So individual action, while necessary, is insufficient. We spent decades building a privacy infrastructure on a mathematical assumption: that removing identifiers makes people anonymous. AI proved that assumption wrong. Not just for future data. For all the data.
While you may understand that “I consent” doesn’t mean much when the data you’re consenting to share may become vulnerable in ways we can’t predict or possibly approve. But you can ask your hospital whether they use AI in the clinical decisions made on your behalf and demand human oversight.
More importantly, you need to join collective pressure for the institutional and regulatory changes that actually matter. This is a structural problem requiring structural solutions. You can vote (this is an election week!) to support state legislation restricting algorithmic discrimination.
We’re not facing a future privacy problem. We’re facing a present crisis about past data. And we need solutions that acknowledge that AI didn’t just change the rules—it changed them retroactively. The only good news, like so many of the ethical issues technology brings about, is that the same technology creating these vulnerabilities can help fix them, but only if we have the collective will to demand it.
