AI Is Not Social Media: Why Healthcare Needs AI Vigilance, Not AI Panic

I recently read the Perspective on the “invisible risks of AI-generated health information.” ¹ It raises an important problem: millions of health-related conversations are increasingly taking place between people and artificial intelligence, largely outside the traditional mechanisms we have developed to monitor healthcare information and harm.

I agree with much of its direction.

In particular, I believe the authors identify an issue that deserves far more attention: we need to start thinking about AI vigilance.

But there is one aspect where I am more optimistic.

The risk that AI can be exposed to misinformation is real. The assumption that this will necessarily produce an information environment resembling social media is, in my view, much more questionable.

AI is not social media.

And that distinction could fundamentally change how misinformation spreads in healthcare. We have a blind spot: what happens between the patient and the AI?
Perhaps the most important insight from the paper is the invisibility of interactions between patients and generative AI.
Think about how healthcare has traditionally managed risk. When a medicine causes an unexpected adverse event, mechanisms exist to report and investigate it. Pharmacovigilance systems collect signals across countries and populations. Regulators, pharmaceutical companies and healthcare professionals can identify patterns that would be almost impossible to detect from an isolated case.

Healthcare has spent decades building this infrastructure.

Now imagine a different situation. Someone asks an AI assistant a health question. The response contains a subtle but consequential error. The person acts on the information and eventually experiences a poor health outcome.
Would the clinician treating the patient necessarily know that an AI conversation contributed to it?
Would the AI provider know?
Would a regulator?
Would anyone be able to connect this case with thousands of similar interactions?
Potentially not.
That is the invisible risk highlighted by the publication.¹
And it suggests that healthcare may eventually need to develop for AI something conceptually similar to what it developed for medicines.

We need AI vigilance!

No, this is not a new digital buzzword. By AI vigilance I mean:
The systematic detection, documentation, analysis and prevention of harmful or misleading AI-generated health information.
This should not be limited to measuring whether models hallucinate in laboratory evaluations.

• What matters is what happens in the real world.
• What questions are patients asking? What answers are they receiving?
• Which types of answers are systematically problematic?
• Which sources are models relying upon?
• Are particular populations, languages or disease areas associated with greater error rates?
• And, critically, can a harmful AI interaction eventually be connected with an adverse patient outcome?

The publication proposes several elements pointing in this direction, including better transparency around health-related AI interactions, mechanisms for reporting harmful outputs, independent monitoring, content provenance and greater accountability. ¹
I believe we should go further and treat these components as the beginnings of a genuine post-deployment surveillance system for AI in healthcare.
We don’t expect clinical evidence generation to stop when a medicine reaches the market.
Why should our thinking about AI safety stop when a model passes an evaluation benchmark?

But there is another side to the misinformation debate

This is where I depart somewhat from the more pessimistic interpretation of the risk.
Yes, generative AI dramatically reduces the cost of producing misinformation.
One person can generate hundreds of articles, social posts, videos or apparently scientific narratives at extraordinary speed. Bad actors can potentially introduce misleading information into the digital ecosystem with a sophistication and volume that would previously have required substantial resources.
That is a genuine problem.
But producing misinformation and getting an AI system to trust that misinformation are not the same thing.
And I think the distinction is extremely important.

AI is not social media

Much of today’s anxiety about AI and misinformation implicitly assumes that generative AI will amplify information in ways similar to social media.
But the underlying mechanisms are fundamentally different.
On social media, a piece of information can achieve enormous visibility because people engage with it.
They click, they share, they comment, they react.
Popularity can become distribution.
And unfortunately, virality and truth are not the same thing.
Generative AI introduces another layer between the information and the individual.

The model itself

When someone asks an AI a medical question, the objective is increasingly not simply to retrieve the most popular piece of content. The system has to identify information, assess sources, reconcile potentially conflicting evidence and synthesize an answer. (See all my other articles about E-E-A-T)
That creates the possibility of something social media never really had:
A credibility gate between publication and consumption.
It will certainly not be perfect. But it matters.

Healthcare is a special case

This is particularly important for health information.
Search engines have spent years developing mechanisms intended to treat high-stakes information differently.
Google, for example, has long distinguished topics that can significantly affect people’s health, financial stability or safety through its Your Money or Your Life, or YMYL, concept. Its quality frameworks also emphasize signals associated with Experience, Expertise, Authoritativeness and Trustworthiness, commonly known as E-E-A-T.
We should be careful here.
E-E-A-T is not a universal algorithm secretly operating inside every large language model. Different AI platforms have different models, retrieval systems, safety mechanisms and approaches to source selection.
But the principle behind it is important:
Not all sources should carry equal weight when the question can materially affect someone’s health.

Interestingly, the publication itself ultimately points in a similar direction, arguing that health questions could be treated as a high-stakes category and that AI-generated health information should prioritize appropriately vetted medical sources.
This is why I am relatively optimistic.
As AI systems mature, I expect it to become harder, not easier, for poorly substantiated health claims to compete with strongly authoritative, evidence-backed information, particularly where systems use retrieval, provenance and explicit source attribution.

Not impossible, just harder. And that distinction matters!

The paradox of AI misinformation

This leads to what I find perhaps the most fascinating paradox of the current transformation:
Generative AI may simultaneously become the world’s most powerful engine for producing health misinformation and one of our most powerful mechanisms for filtering it.
On the supply side, AI changes everything.
The cost of creating misleading content is approaching zero. It can be produced at scale, translated instantly and adapted to almost any audience.
On the consumption side, however, something equally important is happening.
People are increasingly asking AI systems to mediate the information environment for them.
Instead of reviewing ten webpages themselves, they may ask one question and receive one synthesized answer.
This creates risk because tremendous informational power becomes concentrated in that synthesis.
But it also creates opportunity.

Because if AI systems become increasingly capable of distinguishing clinical guidelines, peer-reviewed evidence, health authorities and recognized medical organizations from poorly substantiated claims (and they clearly do!!!) , they could potentially suppress some of the informational noise humans previously had to navigate themselves.

That is radically different from social media.

Scientific authority therefore becomes more, not less, important
There is another consequence that healthcare organizations should pay close attention to.
For the last two decades, the digital-health equation was approximately:
Publish → rank in search → get clicked → inform

Generative AI is beginning to introduce a different chain:
Publish → establish authority → be understood → be retrieved → be synthesized → be cited → inform

That is a profound change! The question for a medical society, hospital, university, public-health authority or science-led healthcare company will no longer simply be:
Can people find our information on Google?
Increasingly it will also be:
Can AI systems recognize our information as authoritative evidence, understand it correctly, retrieve it when relevant and accurately represent it in their answers?

This is where Generative Engine Optimization, or GEO, becomes much more interesting.
GEO in healthcare should not be about gaming AI
The term “optimization” can easily create the wrong impression.
In healthcare, GEO should absolutely not mean finding tricks that make an AI mention a company, medicine or website more frequently.
That would be both shortsighted and potentially dangerous.

Good healthcare GEO should mean something fundamentally different:
Making authoritative, evidence-based information easier for AI systems to discover, understand, contextualize and attribute correctly.
That means clear authorship, transparent evidence, reliable references, strong scientific governance, precise language, accessible content, structured information, clear dates and update cycles, and ultimately, sufficient authority for an AI system to understand why this source deserves to be trusted.

The objective shouldn’t be to manipulate the machine.
It should be to make high-quality evidence legible to the machine.

Authority without accessibility is no longer enough

This also places a responsibility on organizations that produce trustworthy medical information.
For decades, we have tended to assume that publishing authoritative information was sufficient.
I’m no longer sure that it is.
If the best scientific information is buried inside PDFs, inaccessible pages, poorly structured websites or language that neither patients nor AI systems can easily interpret, while weaker information is highly accessible and easily processed, we have created an unnecessary information gap.

Healthcare organizations therefore need to think about two dimensions simultaneously:
Scientific authority and machine accessibility.
Strong science without accessibility risks invisibility. Accessibility without strong science risks misinformation.
The future of trusted digital health information requires both.
Which brings us back to AI vigilance. None of this eliminates risk.
Models will continue to make mistakes. Retrieval systems can select inappropriate sources. Evidence can be misinterpreted. Knowledge becomes outdated. Bad actors will attempt to manipulate information environments.
And even an answer constructed from excellent sources can still be wrong.
That is precisely why AI vigilance matters.
We shouldn’t frame the choice as:
Trust AI or fear AI.
The more useful question is:
How do we build an information ecosystem in which trustworthy evidence has the highest probability of reaching the patient, while systematically detecting situations where it doesn’t?

To me, that requires two complementary systems.
Upstream: authority.
Make trustworthy scientific information accessible, understandable, attributable and machine-readable.
Downstream: vigilance.
Observe what AI systems actually tell people, identify harmful patterns, investigate signals and continuously improve the ecosystem.

Healthcare already knows how to think this way.
We do it with medicines.
We do it with medical devices.
Perhaps is it time to start doing it with health information generated by AI?

The next phase of digital health

The rise of generative AI unquestionably creates new healthcare risks.
The publication is right to call attention to the interactions we currently cannot see and the harms we may therefore struggle to detect. But I don’t believe the inevitable destination is an internet overwhelmed by AI-generated medical misinformation that AI systems indiscriminately reproduce.
There is another possible future.
AI could become an increasingly sophisticated intermediary between the enormous amount of information available online and the person looking for an answer.
If we build the right mechanisms for authority, provenance, transparency and vigilance, AI could ultimately become part of the defense against misinformation rather than simply an amplifier of it.

That will require work from AI companies, regulators, scientists, medical societies, healthcare professionals and healthcare organizations.
Most importantly, it requires us to stop applying yesterday’s information model to tomorrow’s technology.

AI is not social media. We should take its risks seriously. But we should also recognize what is different about it.
Because the same technology that makes misinformation cheaper to produce may eventually make authoritative information easier to recognize, verify and surface at precisely the moment someone needs it.
And in healthcare, that could be one of AI’s most consequential contributions.

References

1. DeVerna, M.R., Yan, H.Y., Yang, KC. et al. The invisible risks of AI-generated health information. Nat. Health (2026). https://doi.org/10.1038/s44360-026-00206-x

Olivier Gryson, PharmD, MSc
25 years of experience in digital marketing in the pharmaceutical industry
Special focus on AI Search in Pharma Marketing