
More than 40 million people ask ChatGPT health questions every day [1]. FDA regulates what a drug company says about its own product. In the FDA and EMA documents I reviewed for this article, no rule tells a company to monitor or correct what an independent AI system says about that product. That gap is the blind spot.
This article explains where the gap sits, what the published evidence says about AI-generated drug information, which regulatory documents touch the problem, and what Medical Affairs, Pharmacovigilance, Regulatory, Brand, and Legal teams can do about it today. It separates documented facts from my analysis. It contains no new AI-answer dataset, and it says so in the methodology section.
The Short Answer: What Is the Regulatory Blind Spot in AI Drug Information?
The blind spot is the space between what a company controls and what a patient reads. FDA rules cover company-controlled communications, such as advertising and promotional labeling [9]. Pharmacovigilance rules cover adverse event information the company receives [10]. Neither set of rules assigns anyone the job of checking the answer that a general-purpose chatbot gives when a patient asks about the drug.
“More than 40 million people” use ChatGPT daily for health questions, and more than 5% of all ChatGPT messages globally are about health care, according to an OpenAI report published in January 2026 and reported by Medical Economics. [1]
The Five Findings That Matter
- In a pharmacist-graded test reported at the ASHP Midyear Clinical Meeting in December 2023, ChatGPT answered 10 of 39 drug-information questions satisfactorily. That is 25.6% (calculation: 10 divided by 39). [2]
- In the same test, ChatGPT gave references for 8 of the 28 questions it answered directly. The investigators found that none of the listed references existed. [2]
- In a BMJ Quality & Safety study of Bing Copilot, mean answer accuracy was 88.7%. In an expert-rated subset of 20 answers, 22% (95% CI 10-40%) were judged able to cause death or severe harm. [3]
- In a 2026 BMJ Open audit of five chatbots, 49.6% of 250 responses on misinformation-prone health topics were problematic. [5]
- FDA’s draft misinformation guidance of July 2024 lets a company respond to online misinformation voluntarily. As summarized by Cooley, it does not create a duty to monitor or correct. [11]
Finding 3 shows why an average hides risk. A high mean accuracy and a meaningful tail of harmful answers can sit in the same dataset. Finding 5 comes from health topics such as cancer, vaccines, stem cells, nutrition, and athletic performance. It is not a drug-label audit, so I treat it as context and not as drug-specific proof. [5]
Why Does AI-Generated Drug Information Matter to Regulatory Affairs?
Who Owns an AI Answer About a Drug?
Nobody at the manufacturer owns it. The AI developer controls the model. The user controls the prompt. The manufacturer controls the label, the website, and the promotional materials, but the model may or may not draw on those sources. That split of control is the core of the problem.
Control versus influence
A company cannot edit the output of ChatGPT, Gemini, Claude, or Perplexity. It can influence the inputs by publishing accurate, current, well-structured content. Influence is not control, and the regulatory framework follows control.
What Is an ‘Independent Third Party’ Under FDA’s Misinformation Guidance?
FDA’s draft guidance of July 2024 describes misinformation as incorrect representations or implications about a firm’s product that independent third parties create or disseminate. Cooley’s summary does not define the term ‘independent third party’ or mention AI. [11] Whether an AI system’s generated answer fits the concept has not been tested in the documents I reviewed. This is an open interpretation question, not a settled one.
Why Is the Blind Spot New?
The scale is new. OpenAI reported that roughly one in four of its more than 800 million regular users submits at least one health-related prompt each week. [1] Search engines returned links to sources. A chatbot returns a single synthesized answer, often with no label citation. The BMJ Open audit found that references were returned for about 81% of the requested slots (1,013 of 1,250), median reference completeness was 40%, and fabricated citations occurred. [5]
What Does the Evidence Say About the Accuracy of AI Drug Information?
How Did ChatGPT Perform on Pharmacist-Graded Drug Questions? (LIU/ASHP, 2023)
ChatGPT answered 10 of 39 questions satisfactorily and 29 unsatisfactorily. [2] Researchers at Long Island University collected 45 questions. The reported results cover 39. Drug Topics lists the authors as Grossman, Zerilli, and Nathan, and the work was presented at the ASHP Midyear Clinical Meeting in Anaheim on December 3-7, 2023. [2]
Unanswered and unsatisfactory responses
ChatGPT gave no direct response to 11 of the 29 unsatisfactory items and offered general background instead. [2] That is 28.2% of all 39 questions (calculation: 11 divided by 39).
Fabricated references
ChatGPT supplied references for 8 of the 28 questions it answered directly. The investigators could not find any of the listed references. [2] A fabricated reference is a separate failure from a wrong claim. It makes a wrong claim look checkable.
What Did the BMJ Quality & Safety Study of Bing Copilot Find About Drug Information Risks?
Andrikyan and colleagues put 10 patient questions about each of 50 frequently prescribed US outpatient drugs to Bing Copilot. This produced 500 answers containing 1,727 statements. Answers were generated in April 2023. [3]
- Mean accuracy was 88.7% (SD 22.3%), and mean completeness was 76.7% (SD 32.0%). [3]
- Mean Flesch Reading Ease was 37.2, which is difficult text that needs college-level reading. [3]
- In a subset of 20 answers rated by seven experts, 54% aligned with scientific consensus, 39% opposed it, and 6% had no consensus. [3]
- In the same subset, 66% (95% CI 50-85%) were judged potentially harmful, and 22% (95% CI 10-40%) could cause death or severe harm. [3]
The abstract concludes that chatbots can provide overall complete and accurate patient drug information. [3] Both findings are true of the same dataset. The harm assessment covers only 20 answers, so the confidence intervals are wide. A fair reading is that the average looks good and the tail is not safe enough to ignore.
How Did ChatGPT 3.5 Score on Pharmacy Practice Questions? (Journal of Pharmacy Practice, 2024)
Munir, Gehres, Wai, and Song tested the free ChatGPT 3.5 on 32 questions in five categories. The overall score was 21 of 44 points, or 47.73%. [4] Their conclusion was that ChatGPT has limited success as a tool to answer pharmacy-based questions. [4]
What Did the 2026 BMJ Open Audit of Five Chatbots Find?
The audit tested Gemini 2.0, DeepSeek V3, Meta AI (Llama 3.3), ChatGPT 3.5, and Grok 2 on 1 February 2025. It used 10 prompts in each of five categories, which gave 50 prompts per chatbot and 250 responses in total. [5]
- Non-problematic: 50.4%. Somewhat problematic: 30%. Highly problematic: 19.6%. [5]
- Problematic responses by chatbot: Grok 58%, ChatGPT 52%, Meta AI 50%, DeepSeek 48%, Gemini 40%. Differences across chatbots were not statistically significant overall (p=0.566). [5]
- Median reference completeness was 40%, and no chatbot produced a fully complete and accurate reference list for any prompt. [5]
- Only 2 of 250 questions drew a refusal. [5]
The authors wrote that continued deployment without public education and oversight risks amplifying misinformation. [5] The topics were cancer, vaccines, stem cells, nutrition, and athletic performance. They were not prescribing-information questions, so this audit says little about label alignment for a specific drug.
Does a Good Model Mean a Good Patient Outcome? The Oxford Nature Medicine Trial
Not necessarily. A Nature Medicine randomized study from the Oxford Internet Institute enrolled 1,298 UK participants who each worked through medical scenarios. The models tested were GPT-4o, Llama 3, and Command R+. [6]
- The models alone identified a relevant condition in 94.9% of scenarios and chose the correct course of action in 56.3%. [6]
- Participants using the models identified a relevant condition in fewer than 34.5% of cases and chose the correct course of action in fewer than 44.2%. [6]
- Calculated gap, condition identification: more than 60 percentage points (94.9 minus 34.5). Calculated gap, course of action: more than 12 percentage points (56.3 minus 44.2). These are my calculations from the reported thresholds. [6]
- The participant results were no better than those of a control group that used usual resources. [6]
The study did not focus on drug labels. Its relevance is that model accuracy in isolation does not predict what a patient takes away from a conversation. That makes monitoring answers by themselves a partial measure. It is necessary, and it is not sufficient.
Five Studies Side by Side: What Can and Cannot Be Compared?
The table puts the five sources on one page. Each used a different model, question set, grader, and outcome definition. The rates are not comparable as a league table. The table exists so a reader can see the denominators.
| Study | Model tested | Test date | Sample | Headline result | Drug-specific? | Source |
|---|---|---|---|---|---|---|
| Grossman, Zerilli, Nathan (LIU), ASHP Midyear | ChatGPT (version not stated in my source) | Presented Dec 3-7, 2023 | 39 reported questions (45 collected) | 10 satisfactory (25.6%, calculated) | Yes | [2] |
| Andrikyan et al., BMJ Qual Saf | Bing Copilot | Apr 2023 | 500 answers, 1,727 statements; 20-answer expert subset | Accuracy 88.7%; 22% of subset could cause death or severe harm | Yes | [3] |
| Munir et al., J Pharm Pract | ChatGPT 3.5 | Published online May 22, 2024 | 32 questions | 21 of 44 points (47.73%) | Pharmacy practice | [4] |
| BMJ Open audit | Gemini 2.0, DeepSeek V3, Meta AI, ChatGPT 3.5, Grok 2 | Feb 1, 2025 | 250 responses | 49.6% problematic | No | [5] |
| Oxford, Nature Medicine | GPT-4o, Llama 3, Command R+ | Not stated in my source | 1,298 participants | Users identified a condition in under 34.5% of cases versus 94.9% for models alone | No | [6] |
How Old Is the Evidence by the Time It Is Published? An Evidence-Latency Calculation
The BMJ Open audit tested its models on 1 February 2025 and appeared in April 2026. That is about 14 months (calculation: 1 February 2025 to 14 April 2026 is 437 days, or 14.4 months). [5] The audit’s own methods table lists ChatGPT 3.5 as a November 2022 version. That version was about 27 months old on the test date (calculation: November 2022 to February 2025). [5]
The 14-month figure uses the April 2026 listing date I found for the paper. Treat it as approximate. The point stands without exact days. Published accuracy studies describe model versions that may already be replaced. A current label alignment figure for any model needs fresh testing, and a one-time study cannot supply it.
What Does FDA Regulate Today? Company-Controlled Communications
What Does 21 CFR 314.81(b)(3)(i) Require for Promotional Materials?
An applicant must submit specimens of promotional labeling and advertising when it first disseminates or publishes them. Each submission needs a completed Form FDA 2253 and a copy of the current professional labeling. [9] The rule attaches to materials the applicant devises for promotion. An answer that a third-party chatbot generates on its own is not such a material.
What Changed in FDA’s September 2025 Enforcement Push?
On September 9, 2025, FDA announced that it was sending “thousands of letters” warning drug companies to remove misleading ads. It also announced “approximately 100 cease-and-desist letters.” [7] The agency said it would start rulemaking to close the “adequate provision” loophole created in 1997. [7]
Letter counts reported by outside counsel
Covington reported that CDER issued more than 50 Warning Letters and more than 50 Untitled Letters for drug advertisements in the third quarter of 2025. CBER issued an Untitled Letter to AstraZeneca about FluMist dated September 9, 2025, and Covington noted that CBER’s advertising office had posted no enforcement letters since 2018. [8] The warning letters were mostly about obesity products, both approved and compounded. [8]
The social media figure FDA cited
FDA’s release cited a 2024 review in which 100% of pharmaceutical social media posts highlighted benefits and 33% mentioned potential harms. [7] That figure describes company posts. It matters here because it shows what FDA treats as a deficiency in company-controlled content: benefit claims with thin risk information.
Does FDA Treat a Chatbot Answer as Your Advertisement?
Based on the documents I reviewed, no. The September 2025 announcement concerns ads that companies and their promotional partners publish. [7] An answer generated by a general-purpose model from a patient’s own prompt is not a company communication under 21 CFR 314.81(b)(3)(i). [9] The position changes if a company deploys its own chatbot, pays for the placement of AI-generated content, or controls the system prompt and the content source. I did not find FDA guidance that addresses those cases directly, so a company with an owned chatbot should take the question to Regulatory and Legal before launch, not after.
Can You Correct What AI Says About Your Drug? FDA’s Misinformation Draft Guidance
What Does the July 2024 Draft Guidance Cover?
FDA released “Addressing Misinformation About Medical Devices and Prescription Drugs: Questions and Answers” as a draft on July 8, 2024. It replaces a 2014 draft titled “Internet/Social Media Platforms: Correcting Independent Third-Party Misinformation About Prescription Drugs and Medical Devices.” [11] It sets out FDA’s enforcement policy for internet-based communications that respond to internet-based misinformation about a firm’s approved or cleared product. It excludes TV and radio ads. [11] I found no final version in my searches. Check the FDA guidance database before you rely on the draft text.
Is a Company Required to Correct Online Misinformation?
No. As Cooley describes it, the guidance applies when a company chooses to respond. The article describes no obligation to correct. [11]
Conditions for a tailored responsive communication
Per Cooley, a response must identify the specific misinformation and where it appears. It must be truthful, accurate, and scientifically sound, directly relevant, and limited to what is needed to address the misinformation. It must give a way to obtain current FDA-required labeling, show the posting date, and disclose that the company shared it. If the response addresses an unapproved use, it must say that the use is not approved. [11]
The regulatory benefit
If a company follows the guidance, FDA says it does not intend to enforce certain promotional labeling and advertising requirements, including Form FDA 2253 submissions for the response. [11]
Where the Guidance Stops: Can You Correct a Model?
The guidance covers what a company says. It does not give a company a way to change what a model says. A company can publish a correction on its own website, in a medical information response, or on a platform where the misinformation appeared. A company cannot edit a model’s weights or answer. That limit is practical, and it explains why correction guidance alone does not close the gap. Cooley’s summary does not mention AI in the guidance. [11]
What Does Pharmacovigilance Owe When a Chatbot Is Involved?
What Does 21 CFR 314.80 Say About Information From Any Source?
An applicant must promptly review all adverse drug experience information it obtains or receives from any source, foreign or domestic. [10] The regulation defines an adverse drug experience as any adverse event associated with the use of a drug in humans, whether or not considered drug related. [10] A serious and unexpected adverse drug experience needs a 15-day alert report, due no later than 15 calendar days after the applicant first receives the information. [10]
What Does EMA GVP Module VI Say About Digital Media?
Section VI.B.1.1.4 says marketing authorisation holders should regularly screen the internet or digital media under their management or responsibility. It defines a company-sponsored digital medium as one that is “owned, paid for and/or controlled by the marketing authorisation holder.” [12] For other sites, the text says that when the holder becomes aware of a report, it should assess whether the report qualifies for submission as an ICSR. It does not require proactive screening of third-party sites. [12]
The four validity criteria
Section VI.B.2 lists four elements of a valid ICSR: an identifiable reporter, a single identifiable patient, at least one suspected medicinal product, and at least one suspected adverse reaction. [12]
Is a Wrong Chatbot Answer an Adverse Event?
No. A wrong answer is an inaccurate statement. An adverse event is a clinical occurrence in a patient. [10] The two can connect. If a patient follows an incorrect answer and is harmed, and the company learns of the patient’s report, the information may meet the criteria for a case. That is my reading of the rules above, not a regulatory statement, and Legal and the QPPV or safety head should decide such cases individually.
Where a chatbot-linked report could enter the safety system
Three entry points are plausible: a patient or caregiver contacts the company’s medical information line, a health professional reports a patient event and mentions the chatbot, or a company-owned digital channel receives a user post. Only the last falls inside the GVP screening duty for media under the company’s management. [12] The first two arrive through the any-source principle. [10]
What Does the Company’s Own AI Use Have to Do With It?
What Did FDA’s Purolea Warning Letter Say About AI?
FDA issued Warning Letter 320-26-58 to Purolea Cosmetics Lab on April 2, 2026. FDA cited the firm under 21 CFR 211.22(c) for failing to review AI-generated documents for accuracy and CGMP compliance. [16] The letter says the firm used AI to create drug product specifications, procedures, and master production or control records. [16] FDA also cited 21 CFR 211.100 because the firm distributed products without process validation. The firm said the AI agent never mentioned the requirement. [16]
The letter concerns manufacturing. It matters for this topic because it shows how FDA reasons about AI output: the company remains responsible for what it releases, and fluent output is not a defense. [16] That principle applies to any company-run AI that answers questions about its drugs.
What Does FDA’s January 2025 Draft Guidance on AI Cover?
The draft guidance, “Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products,” was published in the Federal Register on January 7, 2025. The comment deadline was April 7, 2025. [13] It covers AI models used in the nonclinical, clinical, postmarketing, and manufacturing phases when the model produces information or data to support regulatory decisions on safety, effectiveness, or quality. [13]
What the draft excludes
The draft does not address AI used in drug discovery, AI used for operational efficiencies such as internal workflows and drafting a regulatory submission, or AI uses that do not affect patient safety, drug quality, or the reliability of study results. [13] A consumer chatbot answering a patient’s question falls outside the scope as I read it. I found no final version of this guidance in my searches.
What Do the FDA-EMA Guiding Principles of January 2026 Say?
FDA and EMA announced ten guiding principles of good AI practice in drug development on January 14, 2026. [15] The FDA document lists them as: human-centric by design, risk-based approach, adherence to standards, clear context of use, multidisciplinary expertise, data governance and documentation, model design and development practices, risk-based performance assessment, life cycle management, and clear, essential information. [14] It defines the scope as AI used to generate or analyze evidence across the drug product life cycle. [14] Applied Clinical Trials reports that the principles are not binding. [15]
Does Even the Regulator’s AI Make Errors? The FDA Elsa Reports
CNN reported on July 23, 2025 that three current or former FDA employees said the agency’s Elsa tool had hallucinated nonexistent studies or misrepresented real research. One said, “It hallucinates confidently.” [17] An HHS spokesperson said the information given to CNN was mischaracterized and taken out of context. FDA Commissioner Marty Makary told CNN he had not heard those specific concerns. [17] I include this as a reported dispute, not an established finding. It shows that the question of AI reliability affects regulators and companies alike.
A Five-Zone Map of AI Drug Statements (Original Framework)
The framework below is my own. It is not an FDA or EMA classification. It sorts AI drug statements by who controls the output, because the regulatory hooks I found follow control.
| Zone | Example | Who controls the output | Documents that apply (from this article) | Gap |
|---|---|---|---|---|
| 1. Company-owned AI interface | A branded website chatbot | Company | 21 CFR 314.81(b)(3)(i) if the output is promotional material [9]; GVP VI.B.1.1.4 for media under company management [12] | No FDA guidance found that addresses owned chatbots directly |
| 2. Company content retrieved by third-party AI | A model quotes the company’s label or site | AI developer, with company-supplied inputs | None specific found | Influence without control |
| 3. Independent general-purpose AI answer | ChatGPT, Gemini, Claude, or Perplexity answers a patient question | AI developer and user | Voluntary response under the July 2024 draft guidance [11]; any-source PV principle [10] | No duty to monitor or correct found |
| 4. Company-internal AI for regulated work | AI drafts specifications or records | Company | 21 CFR 211.22(c) and 211.100 as applied in Purolea [16]; FDA draft AI guidance [13]; FDA-EMA principles [14] | Covers evidence and manufacturing, not consumer answers |
| 5. Regulator-side AI | FDA’s Elsa tool | Agency | Reported reliability dispute [17] | Outside company control |
Zone 3 is the blind spot. It is also the zone with the highest volume, given the scale OpenAI reported. [1]
Regulatory Timeline Reconstruction, 2024 to 2026
The table orders the FDA and FDA-EMA documents discussed above and calculates the gap between them. All day counts are my calculations from the dates in the sources.
| Date | Event | Days since previous event | Days since July 8, 2024 | Source |
|---|---|---|---|---|
| Jul 8, 2024 | FDA releases draft misinformation Q&A guidance | – | 0 | [11] |
| Jan 7, 2025 | FDA publishes draft AI credibility guidance | 183 | 183 | [13] |
| Sep 9, 2025 | FDA announces DTC advertising enforcement push | 245 | 428 | [7] |
| Jan 14, 2026 | FDA and EMA announce ten AI guiding principles | 127 | 555 | [15] |
| Apr 2, 2026 | FDA issues Warning Letter 320-26-58 (Purolea) | 78 | 633 | [16] |
Five documents appeared over 633 days. Based on the portions I reviewed, none of the five sets a requirement to monitor or correct statements made by an independent general-purpose AI system. The July 2024 guidance allows a voluntary response to misinformation. [11] The January 2025 draft excludes uses that do not affect safety, quality, or study results. [13] The January 2026 principles address evidence generation. [14] The September 2025 announcement addresses advertising. [7] The Purolea letter addresses manufacturing records. [16] That is a statement about those five documents, not about every FDA document or every agency.
Where Do AI Answers and the Label Diverge? A Five-Type Discrepancy Taxonomy (Original Framework)
Teams that monitor AI answers need a shared vocabulary for what they find. The taxonomy below is my own proposal. It is not an industry standard. It defines each type by what a reviewer checks against an authoritative source.
| Type | What the reviewer checks | Authoritative source to check against | Published evidence in this article |
|---|---|---|---|
| 1. Currency error | Does the answer reflect the current label, or an earlier version? | Current FDA label or EU product information, with revision date | None in my sources. Defined by method. |
| 2. Content error | Are the dose, interaction, contraindication, or adverse reaction statements correct? | Prescribing information; SmPC | [2], [3] |
| 3. Citation error | Does each cited reference exist, and does it support the claim? | The cited document itself | [2], [5] |
| 4. Omission error | Does the answer leave out a boxed warning, contraindication, or key precaution? | Boxed warning and Warnings and Precautions sections | Completeness mean of 76.7% in [3] |
| 5. Scope error | Does the answer present an off-label or unapproved use as an approved indication? | Indications and Usage section | None in my sources. Defined by method. |
How Should a Team Classify Severity?
I suggest four levels. Level 1 is a stylistic or readability issue. Level 2 is an incomplete answer with no likely patient effect. Level 3 is a materially wrong statement with low immediate risk. Level 4 is a statement that could lead to harm if followed, such as a wrong dose or a missed contraindication. The BMJ Quality & Safety study used a similar logic when experts rated answers for potential harm, from no harm to death or severe harm. [3] Any Level 4 finding should go to Pharmacovigilance or Medical Information review the same day it is found. That timing is my recommendation, not a regulatory requirement.
How Do You Monitor What AI Says About Your Drug? A Reproducible Method
A monitoring result is only useful if another reviewer can reproduce it. The published studies show what to record. The Bing Copilot study recorded the chatbot mode (balanced), the language, and the VPN location used (New York). [3] The BMJ Open audit recorded exact model versions, their release months, free or paid status, and the test date. [5]
Which Fields Should Every Test Record?
| Field | Why it matters |
|---|---|
| AI platform and exact model version | Model versions change. The BMJ Open audit tested a November 2022 version in February 2025. [5] |
| Free or paid tier | The BMJ Open audit used free consumer versions only. [5] Results may differ for other tiers. |
| Exact prompt text | The BMJ Open audit found that open-ended and closed-ended prompts produced different quality patterns. [5] |
| Date and time of test | Answers change over time. A date lets a reader check which label version was current. |
| Geographic context | The Bing Copilot study queried through a New York VPN. [3] |
| Web search on or off | A search-enabled answer can cite a current source. A model without search relies on training data. |
| Citations requested or not | The BMJ Open audit requested references only for closed-ended prompts. [5] |
| Number of queries and repeats | One response is an observation, not a rate. |
| Label or guideline version used for verification | Without it, a reader cannot tell whether the answer or the check was out of date. |
Which Source Set Should You Verify Against?
Use the current FDA prescribing information for US questions and the current EU product information for EU questions. Add FDA safety communications, ClinicalTrials.gov records for trial questions, and PubMed for published results. Record the label revision date next to each finding. Do not treat another AI answer as a verification source.
How Should You Handle Duplicates and Unsupported Claims?
Run each prompt more than once, and count repeated runs as separate observations of one prompt, not as independent prompts. Flag a claim as unsupported when no sentence in the verification source supports it. Flag a claim as outdated when an earlier label version supports it and the current version does not.
What Are the Limits of Any Monitoring Program?
A monitoring program samples answers. It does not measure what each patient was told. The Oxford trial shows that the same models that scored well alone produced much weaker results when people used them. [6] A program should state its sample, models, dates, and prompt library in every report, and it should not claim to represent all AI-generated drug information.
Where Does DrugChatter Fit?
DrugChatter (drugchatter.com) runs a monitoring product called chatmon. It sends a library of questions to several AI models on a schedule and stores each question, the model vendor, and the answer, so a team can compare responses over time and across models against the current label. It is a way to collect and organize observations at a frequency that manual testing cannot match. It does not replace the label, the clinical evidence, or the judgment of a medical reviewer. This article does not use DrugChatter data.
What Happens If a Chatbot Gets Your Drug’s Dose Wrong? Three Scenarios
The scenarios below are analysis based on stated assumptions. They are not predictions. Each one applies the rules cited earlier.
Scenario A: Nobody Notices
Assumption: the wrong dose appears in an answer, no patient reports a problem, and no one on the company side sees the answer. Result: no regulatory obligation is triggered under the documents I reviewed. The company has no record of the error. The risk is invisible, not absent.
What a record would change
A dated, versioned monitoring record turns an invisible risk into a documented one. A team can then decide whether to publish a correction under the conditions in the July 2024 draft guidance. [11]
Scenario B: A Health Professional Reports a Patient Event
Assumption: a pharmacist tells the company’s medical information line that a patient took a dose from a chatbot answer and had a serious event. Result: the company has received information about an adverse drug experience from a source, and 21 CFR 314.80 requires prompt review of information from any source. [10] If the case is serious and unexpected, the 15-calendar-day clock runs from the date the company first receives the information. [10] The chatbot answer is part of the case narrative. The case follows normal PV processing.
What the case file should hold
The file should hold the reporter details, the patient descriptor, the suspect product, the event, and the chatbot transcript if the reporter provides it. Those first four items map to the validity criteria in GVP Module VI.B.2. [12]
Scenario C: The Company Owns the Chatbot
Assumption: a company deploys a branded assistant on its own site that answers patient questions about its drug. Result: the company controls the medium. GVP Module VI.B.1.1.4 describes media owned, paid for, or controlled by the holder as company-sponsored and subject to regular screening. [12] Whether the assistant’s output is promotional labeling or advertising under 21 CFR 314.81(b)(3)(i) is a separate question that depends on facts. [9] I found no FDA guidance that settles it. Purolea shows FDA holds companies responsible for reviewing AI output they release. [16]
What This Means for Each Function
What This Means for Medical Affairs and Medical Information
Medical Information teams answer unsolicited questions. A health professional who asks about a chatbot statement is an unsolicited inquirer. The team needs a standard response that cites the current label and, where relevant, the primary evidence. It should log recurring chatbot errors, because a pattern is a signal that a correction may be worth publishing.
What This Means for Pharmacovigilance
PV teams should decide in writing how chatbot transcripts are handled when they arrive inside a case report. The any-source principle in 21 CFR 314.80 already covers the case. [10] The new part is operational: who reads the transcript, how it is stored, and how it is described in the narrative. A written rule avoids ad hoc handling.
What This Means for Regulatory Affairs
Regulatory Affairs should track whether FDA finalizes the July 2024 and January 2025 drafts. [11][13] It should also write an internal position on owned chatbots before one is proposed, covering whether the output could be promotional labeling and whether it needs Form FDA 2253 submission. [9]
What This Means for Brand and Promotional Review Teams
Brand teams cannot approve what a model says. They can approve the content the model may retrieve. Accurate, current, well-structured product information on owned properties is the one input a company controls. FDA’s 2025 enforcement letters are a reminder that this content is already under close review. [7][8]
What This Means for Legal and Corporate Communications
Legal should decide, before an incident, who may publish a correction and under which conditions. The draft guidance lists the conditions. [11] Corporate Communications should know that a correction must disclose that the company posted it and must show the posting date. [11]
What Happens Next?
The items below are things to watch. They are not forecasts of outcomes.
- FDA said in September 2025 that it would start rulemaking on the “adequate provision” provision. [7] I did not find a final rule.
- The July 2024 misinformation draft and the January 2025 AI draft remain drafts in my searches. [11][13]
- Applied Clinical Trials reports that the FDA-EMA principles are high-level and will be supplemented by future AI-specific guidance. [15]
- No document I reviewed addresses consumer chatbot answers about a specific marketed drug. If one appears, it will change the analysis in this article.
Methodology and Limitations of This Article
I reviewed the sources listed in the references between the dates of the documents and November 4, 2026. I did not run an original set of AI queries for this article. The calculated figures are labeled in the text: the 25.6%, 28.2%, and 60-point and 12-point gaps, the evidence-latency figures, and the day counts in the timeline. Each can be reproduced from the cited numbers and dates.
- Several sources are secondary. The ASHP poster is cited through Drug Topics, the July 2024 guidance through Cooley, the OpenAI report through Medical Economics, and the Oxford study through a Springer Nature summary. Verify against the primary documents before you quote a number.
- The BMJ Open audit and the Oxford study are not drug-label studies.
- The BMJ Open publication date is approximate, based on a listing I found.
- Statements that no rule requires monitoring apply to the portions of the documents I reviewed. They are not legal conclusions. Consult counsel.
Key Takeaways
- More than 40 million people ask ChatGPT health questions daily, according to OpenAI’s January 2026 report. [1]
- Pharmacist-graded tests found 10 of 39 satisfactory ChatGPT answers and no verifiable references among the 8 references supplied. [2]
- A high average accuracy score can coexist with a meaningful share of potentially harmful answers. [3]
- FDA’s misinformation draft guidance makes a company’s response voluntary. [11]
- Pharmacovigilance rules reach information from any source, and EMA’s screening duty covers media under the holder’s management. [10][12]
- Across five FDA and FDA-EMA documents issued over 633 days, none sets a requirement to monitor or correct an independent AI system’s statements, based on the portions reviewed. [7][11][13][14][16]
- Monitoring only works when each test records the model version, prompt, date, and label version, so another reviewer can reproduce it. [3][5]
- Model accuracy alone does not predict what patients take away. [6]
FAQ
Is a drug company required to monitor what ChatGPT says about its product?
In the FDA and EMA documents I reviewed, no. FDA’s July 2024 draft guidance describes voluntary responses to misinformation [11]. EMA’s GVP Module VI asks marketing authorisation holders to screen media under their management or responsibility, and for other sites it asks them to assess a report once they become aware of it [12]. Check with counsel before you treat this as a legal conclusion.
What can a company do when an AI answer about its drug is wrong?
A company can publish a correction on its own channels, answer through Medical Information, and update its owned content. A correction under the draft guidance must identify the misinformation, be truthful and scientifically sound, give a way to get current labeling, show the posting date, and disclose that the company posted it [11]. A company cannot change the model’s answer directly.
Does a wrong chatbot answer count as an adverse event?
No. An adverse event is a clinical occurrence in a patient [10]. A wrong answer is an inaccurate statement. If a patient is harmed after following it and the company learns of the event, the report is processed like any other report that meets the validity criteria in GVP Module VI.B.2 [12].
How accurate are AI chatbots on drug questions?
Published results range widely and depend on the model version, the questions, and the grader. ChatGPT answered 10 of 39 questions satisfactorily in the ASHP-reported test [2]. ChatGPT 3.5 scored 47.73% on a 32-question pharmacy test [4]. Bing Copilot had a mean accuracy of 88.7% with a harmful tail in an expert-rated subset [3]. None of these results describes today’s models.
How often do chatbots invent references?
In the ASHP-reported test, ChatGPT gave references for 8 of 28 directly answered questions, and the investigators found that none existed [2]. In the BMJ Open audit, median reference completeness was 40%, fabricated citations occurred, and no chatbot produced a fully complete and accurate reference list for any prompt [5].
Does a company-owned chatbot need Form FDA 2253 submission?
The rule requires specimens of promotional labeling and advertising devised for promotion to be submitted at initial dissemination or publication [9]. Whether a chatbot’s output counts depends on the facts, such as who controls the content and the purpose of the tool. I found no FDA guidance that settles it. Regulatory and Legal should decide before launch.
Do the FDA-EMA AI guiding principles apply to patient-facing chatbots?
The FDA document defines its scope as AI used to generate or analyze evidence across the drug product life cycle [14]. Applied Clinical Trials reports that the principles are not binding [15]. A consumer chatbot answering a patient’s question is outside that scope as I read it.
How should a team measure AI answer accuracy against a label?
Record the model version, tier, exact prompt, date, location, and whether search and citations were on. Verify each claim against the current label and note its revision date. Published studies record items such as chatbot mode, VPN location, model release months, and test date [3][5]. Repeat each prompt and report the sample size.
Why are existing accuracy studies not enough for a brand team?
They describe past model versions and general questions. The BMJ Open audit tested a November 2022 model version on February 1, 2025, and the paper appeared about 14 months later (my calculation) [5]. A brand needs current answers about its own drug, checked against its current label, on a repeating schedule.
Which team should own monitoring of AI answers about a drug?
No regulation I found assigns ownership. A workable split is Medical Information for response content, Pharmacovigilance for transcripts that arrive inside case reports, Regulatory for owned-chatbot questions, and Legal for corrections. This split is my suggestion, not a requirement.
What does DrugChatter’s chatmon monitoring do?
DrugChatter’s chatmon product queries several AI models on a schedule with a fixed question library and stores each question, model vendor, and answer for comparison against the current label. It collects observations. A medical reviewer still has to judge each one against primary sources.
References
- Medical Economics. (2026, January 7). 40 million people now use ChatGPT daily for health questions, OpenAI report finds. https://www.medicaleconomics.com/view/40-million-people-now-use-chatgpt-daily-for-health-questions-openai-report-finds
- Drug Topics. (n.d.). ChatGPT provides inaccurate responses to questions about drug information [Report on Grossman, S., Zerilli, T., & Nathan, J., Poster 8-021, ASHP Midyear Clinical Meeting & Exhibition, Anaheim, CA, December 3-7, 2023]. https://www.drugtopics.com/view/chatgpt-provides-inaccurate-responses-to-questions-about-drug-information
- Andrikyan, W., Sametinger, S. M., Kosfeld, F., et al. (2025). Artificial intelligence-powered chatbots in search engines: A cross-sectional study on the quality and risks of drug information for patients. BMJ Quality & Safety, 34(2), 100. https://qualitysafety.bmj.com/content/34/2/100
- Munir, F., Gehres, A., Wai, D., & Song, L. (2024). Evaluation of ChatGPT as a tool for answering clinical questions in pharmacy practice. Journal of Pharmacy Practice (first published online May 22, 2024). https://journals.sagepub.com/doi/10.1177/08971900241256731
- Generative artificial intelligence-driven chatbots and medical misinformation: An accuracy, referencing and readability audit. (2026). BMJ Open, 16(4), e112695. https://bmjopen.bmj.com/content/16/4/e112695
- Scimex / Springer Nature. (2026). Medical decision-making not improved by AI chatbots [Summary of Bean, A., Mahdi, A., et al., Reliability of LLMs as medical assistants for the general public: A randomized preregistered study, Nature Medicine]. https://www.scimex.org/newsfeed/medical-decision-making-not-improved-by-llms
- U.S. Food and Drug Administration. (2025, September 9). FDA launches crackdown on deceptive drug advertising [Press release]. https://www.fda.gov/news-events/press-announcements/fda-launches-crackdown-deceptive-drug-advertising
- Covington & Burling LLP. (2025, October). FDA advertising and promotion enforcement activities update. https://www.cov.com/en/news-and-insights/insights/2025/10/fda-advertising-and-promotion-enforcement-activities-update
- 21 C.F.R. § 314.81, Other postmarketing reports. Legal Information Institute, Cornell Law School. https://www.law.cornell.edu/cfr/text/21/314.81
- 21 C.F.R. § 314.80, Postmarketing reporting of adverse drug experiences. Legal Information Institute, Cornell Law School. https://www.law.cornell.edu/cfr/text/21/314.80
- Cooley LLP. (2024, September 6). FDA’s draft guidance on addressing misinformation about medical devices and prescription drugs. https://www.cooley.com/news/insight/2024/2024-09-06-fdas-draft-guidance-on-addressing-misinformation-about-medical-devices-and-prescription-drugs
- European Medicines Agency. (n.d.). Guideline on good pharmacovigilance practices (GVP) Module VI: Collection, management and submission of reports of suspected adverse reactions to medicinal products. https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/guideline-good-pharmacovigilance-practices-gvp-module-vi-collection-management-submission-reports_en.pdf
- U.S. Food and Drug Administration. (2025, January 7). Considerations for the use of artificial intelligence to support regulatory decision-making for drug and biological products (Draft guidance; Docket No. FDA-2024-D-4689). Federal Register, 90(4). https://govinfo.gov/content/pkg/FR-2025-01-07/html/2024-31542.htm
- U.S. Food and Drug Administration & European Medicines Agency. (2026, January). Guiding principles of good AI practice in drug development. https://www.fda.gov/media/189581/download
- Applied Clinical Trials. (2026, January 29). FDA, EMA align on ten principles for artificial intelligence use in drug development. https://appliedclinicaltrialsonline.com/view/fda-ema-align-ten-principles-artificial-intelligence-use-drug-development
- U.S. Food and Drug Administration. (2026, April 2). Warning Letter 320-26-58: Purolea Cosmetics Lab. https://www.fda.gov/inspections-compliance-enforcement-and-criminal-investigations/warning-letters/purolea-cosmetics-lab-722591-04022026
- Washenko, A. (2025, July 24). FDA employees say the agency’s Elsa generative AI hallucinates entire studies. Engadget (reporting CNN investigation of July 23, 2025). https://www.engadget.com/ai/fda-employees-say-the-agencys-elsa-generative-ai-hallucinates-entire-studies-203547157.html






