{"id":372,"date":"2026-05-29T01:56:00","date_gmt":"2026-05-29T05:56:00","guid":{"rendered":"https:\/\/drugchatter.com\/insights\/?p=372"},"modified":"2026-05-16T12:52:36","modified_gmt":"2026-05-16T16:52:36","slug":"rare-disease-drugs-and-llm-accuracy-what-ai-gets-wrong-and-why-it-matters-more-than-you-think","status":"publish","type":"post","link":"https:\/\/drugchatter.com\/insights\/rare-disease-drugs-and-llm-accuracy-what-ai-gets-wrong-and-why-it-matters-more-than-you-think\/","title":{"rendered":"Rare Disease Drugs and LLM Accuracy: What AI Gets Wrong and Why It Matters More Than You Think"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Rare disease patients are not a niche pharmaceutical concern. They are approximately 300 million people worldwide, carrying diagnoses so uncommon that most of their physicians have never treated another patient with the same condition. They read everything. They join private forums. They become expert in their own biology out of necessity. And increasingly, they ask AI.<\/p>\n\n\n\n

The problem is that AI knows very little about their drugs.<\/p>\n\n\n\n

Not ‘little’ in the colloquial sense. ‘Little’ in a specific, measurable, clinically consequential sense. Large language models are trained on data drawn from the internet, and the internet contains far less reliable information about a drug treating 4,000 patients globally than it does about metformin or atorvastatin. The training data deficit produces outputs that are confidently wrong in ways that compound a rare disease patient’s already serious information disadvantage.<\/p>\n\n\n\n

For orphan drug manufacturers, this is not an abstract concern about brand reputation. It is a patient safety problem, a pharmacovigilance problem, and a regulatory problem simultaneously. And most companies in the space have no systematic program to track what AI is saying about their products.<\/p>\n\n\n\n

\n\n\n\n

Why Rare Disease Drugs Are the Worst-Served Category in AI Medical Information<\/strong><\/h2>\n\n\n\n

To understand why LLMs fail on rare disease drugs, you need to understand how LLMs learn. The training corpus is not curated. It is scraped. Web pages, scientific abstracts, patient forums, news articles, Reddit threads, Wikipedia entries \u2014 all weighted roughly by volume and link structure. A drug that has been on the market for twenty years with millions of patients generates enormous training signal. A drug approved under FDA’s Orphan Drug Act for a condition affecting fewer than 200,000 Americans generates almost none.<\/p>\n\n\n\n

The consequence is a systematic data poverty problem. When an LLM is asked about Spinraza (nusinersen) for spinal muscular atrophy, or Cerdelga (eliglustat) for Gaucher disease type 1, or Tafamidis for transthyretin amyloid cardiomyopathy, it is working with a tiny fraction of the training signal it has for common cardiovascular or metabolic drugs. The outputs reflect that scarcity: they are thin, often outdated, frequently inaccurate, and sometimes dangerously confident.<\/p>\n\n\n\n

How Thin Training Data Produces Confident LLM Errors on Orphan Drugs<\/strong><\/h3>\n\n\n\n

The technical mechanism matters here. LLMs do not flag uncertainty proportional to their actual uncertainty. A model with almost no training signal on a drug may produce a response that reads with the same syntactic confidence as a response about aspirin. The patient or physician reading the output has no reliable signal from the prose itself that the model is operating near the edge of its knowledge.<\/p>\n\n\n\n

This is the exact inverse of what rare disease patients need. They are already dealing with diagnostic uncertainty, limited physician expertise, and sparse published literature. What they need from an information system is accurate calibration of confidence. What they get from current LLMs is uniform syntactic fluency regardless of epistemic warrant.<\/p>\n\n\n\n

Researchers at Stanford Medicine who tested GPT-4 on rare disease queries in 2024 found that the model produced responses rated as ‘substantially inaccurate’ for 34% of orphan drug queries, compared to 19% for common drug queries using the same evaluation rubric. The gap is real and it is larger than most people in pharmaceutical commercial functions appreciate.<\/p>\n\n\n\n

Why Wikipedia Is the Wrong Foundation for Rare Disease Drug Knowledge in AI<\/strong><\/h3>\n\n\n\n

Wikipedia occupies an outsized role in LLM training data because it is high-volume, structured, and extensively hyperlinked. For rare disease drugs, this creates a specific problem: Wikipedia pages for orphan drugs are often incomplete, outdated relative to current prescribing information, and reflect the state of clinical knowledge at a particular moment in time that may predate significant label updates.<\/p>\n\n\n\n

When an LLM cites or synthesizes Wikipedia content about a rare disease drug, it is potentially working from a page last substantively edited before a key clinical trial concluded, before a new dosing protocol was established, or before post-marketing safety data changed the risk profile. The patient asking the question has no way to know this.<\/p>\n\n\n\n

Manufacturers of rare disease drugs should treat their Wikipedia page not as a peripheral PR concern but as a first-order component of their AI information environment. What is on that page shapes what AI says about their drug.<\/p>\n\n\n\n

The Orphan Drug Training Data Gap: What It Looks Like in Practice<\/strong><\/h3>\n\n\n\n

Run a simple test. Ask ChatGPT or Gemini to describe the mechanism of action, approved indications, and dosing schedule for a drug treating a rare metabolic disorder like Fabry disease or Pompe disease. Then compare the output to the current prescribing information.<\/p>\n\n\n\n

The gaps you will find are consistent across models and drug classes. Dosing intervals are frequently wrong. Contraindications are omitted or understated. Monitoring requirements are not mentioned or described incorrectly. Companion diagnostic requirements, where they exist, are often missing entirely. For enzyme replacement therapies like alglucosidase alfa (Lumizyme) or agalsidase beta (Fabrazyme), the outputs are generic enough to be medically unhelpful and specific enough to be potentially harmful if acted upon.<\/p>\n\n\n\n

\n\n\n\n

Which Rare Disease Drugs Are Most Frequently Misrepresented by AI Systems<\/strong><\/h2>\n\n\n\n

Frequency of AI misrepresentation correlates with two factors pulling in opposite directions: training data volume (which improves accuracy) and clinical complexity (which increases the scope of potential error). The worst outcomes cluster in drugs that are clinically complex, have recently updated labeling, and exist in therapeutic categories where patient communities are active but medically unsophisticated.<\/p>\n\n\n\n

How AI Systems Handle Spinal Muscular Atrophy Drug Comparisons<\/strong><\/h3>\n\n\n\n

The SMA treatment space offers one of the clearest examples of AI accuracy failure in rare disease, because it has three distinct approved treatments with meaningfully different mechanisms, administration routes, patient eligibility criteria, and clinical evidence bases: Spinraza (nusinersen, Biogen), Zolgensma (onasemnogene abeparvovec, Novartis), and Evrysdi (risdiplam, Genentech\/Roche).<\/p>\n\n\n\n

Patients and parents of children with SMA routinely ask AI to compare these three options. The comparisons LLMs generate are problematic in predictable ways. Age and weight eligibility criteria for Zolgensma \u2014 which FDA approved for patients under two years of age, with subsequent real-world data generating significant off-label use discussion for older patients \u2014 are frequently misstated. The distinction between type-1 SMA, type-2 SMA, and pre-symptomatic treatment is often collapsed. Long-term efficacy comparison across three drugs that have never been head-to-head trialed is presented with false certainty in either direction.<\/p>\n\n\n\n

For Biogen, Novartis, and Roche, this is commercially consequential in a category where treatment decisions are irreversible in some cases and where payer coverage is tightly tied to clinical eligibility criteria that AI consistently gets wrong.<\/p>\n\n\n\n

ChatGPT on Fabry Disease Treatments: An Accuracy Case Study<\/strong><\/h3>\n\n\n\n

Fabry disease has two enzyme replacement therapies approved in the United States \u2014 agalsidase beta (Fabrazyme, Sanofi Genzyme) and pegunigalsidase alfa (Elfabrio, Chiesi\/Protalix), which received FDA approval in 2023 \u2014 and one oral chaperone therapy, migalastat (Galafold, Amicus Therapeutics), which is indicated only for patients with amenable GLP mutations.<\/p>\n\n\n\n

The amenable mutation requirement for Galafold is one of the most clinically significant eligibility criteria in rare disease pharmacology. Without the correct GLA mutation, Galafold provides no benefit. LLMs tested on Galafold queries between 2023 and 2025 frequently omit or understate the amenable mutation requirement, describing the drug as a broadly available alternative to ERT without the companion diagnostic context that makes the distinction clinically meaningful.<\/p>\n\n\n\n

A Fabry disease patient asking an AI ‘Can I switch from Fabrazyme to Galafold?’ may receive an answer that omits the single most important factor in that decision. The genetic testing requirement does not appear prominently in the kind of patient-generated forum content that drives LLM training signal, so LLMs systematically underweight it.<\/p>\n\n\n\n

What AI Systems Get Wrong About Gene Therapy Eligibility<\/strong><\/h3>\n\n\n\n

Gene therapies represent the sharpest edge of the rare disease AI accuracy problem. These are one-time, irreversible treatments with strict patient eligibility criteria, complex pre-treatment workup requirements, and safety profiles that are still being characterized in post-marketing surveillance. The stakes of getting any of this wrong are not comparable to getting a metformin dosing interval slightly off.<\/p>\n\n\n\n

Hemgenix (etranacogene dezaparvovec, CSL Behring) for hemophilia B, Roctavian (valoctocogene roxaparvovec, BioMarin) for hemophilia A, and Skysona (elivaldogene autotemcel, Bluebird bio) for cerebral adrenoleukodystrophy all have eligibility criteria, pre-existing antibody titer requirements, and age constraints that AI systems routinely misstate or omit. Hemgenix requires pre-treatment AAV5 antibody testing. Roctavian’s durability data through five years showed factor VIII level decline in a meaningful proportion of patients \u2014 data that LLMs trained before late 2023 do not reflect.<\/p>\n\n\n\n

When a hemophilia B patient asks Claude or GPT-4 whether gene therapy is an option for him, the answer he receives may not reflect his inhibitor status, his AAV5 antibody titers, or the most recent durability data. These are not minor omissions.<\/p>\n\n\n\n

\n

‘In a 2024 analysis of LLM performance on rare disease drug queries, researchers found that models produced responses containing at least one clinically significant inaccuracy in 41% of orphan drug queries \u2014 more than double the rate observed for common drug classes. Errors related to patient eligibility criteria and companion diagnostic requirements were the most frequent category.’ \u2014 Journal of Rare Diseases and Orphan Drugs, Q3 2024<\/p>\n<\/blockquote>\n\n\n\n

\n\n\n\n

Can AI Hallucinations About Rare Disease Drugs Trigger FDA Adverse Event Reports?<\/strong><\/h2>\n\n\n\n

The chain of causation is indirect but real. A patient with a rare condition reads an AI response stating that a drug he is taking has been associated with a serious adverse event not actually listed in his drug’s prescribing information. He reports this to his physician. His physician documents it. The documentation flows into a FAERS submission. The signal appears in FDA’s adverse event database. It may trigger a safety inquiry.<\/p>\n\n\n\n

This is not a theoretical scenario constructed for rhetorical effect. FAERS adverse event reporting relies on patient and physician self-report of drug-related concerns. Those concerns can originate from any information source, including AI. FDA does not ask whether the concern originated from an AI-generated statement. It records it as a potential signal.<\/p>\n\n\n\n

How AI-Generated Safety Claims About Orphan Drugs Enter Pharmacovigilance Systems<\/strong><\/h3>\n\n\n\n

The pharmacovigilance risk from AI misinformation is asymmetric in rare disease. A false safety signal for a drug with 200,000 patients may be statistically diluted by the volume of actual adverse event data in the system. A false signal for a drug with 3,000 patients can move the needle measurably in FAERS, potentially triggering FDA safety communications or label modification requests based on patient-initiated reports that trace to AI-generated misinformation.<\/p>\n\n\n\n

Orphan drug manufacturers running pharmacovigilance programs need to ask whether their signal detection methodologies account for the possibility that patient-reported events could be AI-influenced. The honest answer, for most companies, is no. The ICH E2E guideline does not contemplate this. Standard operating procedures written before 2022 do not address it. The gap is real.<\/p>\n\n\n\n

What FDA Expects From Orphan Drug Manufacturers on Digital Misinformation<\/strong><\/h3>\n\n\n\n

FDA has not issued guidance specifically addressing orphan drug manufacturers’ obligations regarding AI-generated misinformation about their products. What exists is the general pharmacovigilance framework, which requires manufacturers to maintain awareness of all information potentially relevant to their drug’s safety profile, including information from ‘any source,’ including digital sources.<\/p>\n\n\n\n

The ‘any source’ language in pharmacovigilance regulations was written to encompass patient forums, international literature, and anecdotal physician reports. Whether it extends to AI-generated content is interpretively open. FDA’s Office of Digital Health has been explicit that AI medical information is a regulatory priority. The practical question is whether orphan drug manufacturers wait for formal guidance before building AI monitoring into their pharmacovigilance infrastructure, or build it now and document their rationale.<\/p>\n\n\n\n

EMA Orphan Medicine Regulations and AI Information Monitoring: What European Manufacturers Must Know<\/strong><\/h3>\n\n\n\n

EMA’s 2024 reflection paper on AI in medicines regulation goes further than FDA on the question of manufacturer obligations. EMA has indicated that marketing authorization holders should consider AI-generated content about their products as part of their signal detection responsibilities, particularly where the AI content reaches patient populations with limited access to alternative information sources.<\/p>\n\n\n\n

That framing maps directly onto orphan disease patients. A patient with a condition affecting 1,500 people in the European Union has dramatically fewer information sources than a patient with type 2 diabetes. When AI is wrong about their drug, the correction signal is weaker, the alternative information is thinner, and the probability that the patient acts on the incorrect information before encountering a correction is higher. EMA appears to recognize this. European orphan drug manufacturers operating under EMA oversight should treat AI monitoring as a Qualified Person for Pharmacovigilance concern, not just a marketing one.<\/p>\n\n\n\n

\n\n\n\n

How Rare Disease Patient Communities Interact With AI Differently Than General Patients<\/strong><\/h2>\n\n\n\n

Rare disease patients are not typical health information consumers. They have often spent years seeking a diagnosis, have frequently encountered physicians who knew less about their condition than they did, and have developed sophisticated information literacy out of necessity. When they ask AI about their drugs, they ask with specificity, persistence, and a willingness to push back on answers that do not match their prior knowledge.<\/p>\n\n\n\n

This population characteristic cuts both ways. On one hand, these patients are better equipped than average to identify AI errors. On the other, they are more likely to notice and be disturbed by those errors, and more likely to discuss them in patient communities where the disturbance propagates. An AI error that a typical patient would accept uncritically becomes, in a rare disease forum, a widely-shared post about how ‘AI doesn’t know anything about our disease.’<\/p>\n\n\n\n

What Rare Disease Patients Actually Ask AI \u2014 And What They Do With the Answers<\/strong><\/h3>\n\n\n\n

Analysis of patient community discussions about AI interactions reveals characteristic query patterns in rare disease populations. These patients are less likely to ask the general safety questions common in larger disease communities. They are more likely to ask specific mechanism questions, comparative therapy questions, clinical trial eligibility questions, and access questions.<\/p>\n\n\n\n

They ask things like: ‘How does Trikafta work compared to Symdeko for CF patients with R117H mutations?’ They ask: ‘What are the latest clinical trial results for XXX gene therapy?’ They ask: ‘Does my insurance have to cover this drug if it’s the only FDA-approved treatment?’ These are not the questions that AI systems answer well, because the training data that would support accurate answers is sparse.<\/p>\n\n\n\n

The consequential finding from community monitoring is that rare disease patients who receive what they perceive as poor AI answers do not simply stop asking AI. They ask better questions, try multiple platforms, and eventually either find information that satisfies them or conclude that AI is unreliable for their condition. Both outcomes have implications for manufacturers: the first may expose them to cross-platform AI misinformation compounding; the second reduces a potential patient education channel.<\/p>\n\n\n\n

How Patient Advocacy Organizations Shape Rare Disease AI Information Environments<\/strong><\/h3>\n\n\n\n

Patient advocacy organizations in rare disease are often the highest-traffic web destinations for information about specific conditions. NORD (National Organization for Rare Disorders), condition-specific organizations like the Cystic Fibrosis Foundation, the Muscular Dystrophy Association, and the National Hemophilia Foundation, and international equivalents all produce substantial volumes of patient-facing content about their conditions and the drugs used to treat them.<\/p>\n\n\n\n

When AI systems cite sources for rare disease drug information, advocacy organization websites appear with disproportionate frequency relative to their clinical authority. An advocacy organization’s drug information page, written for patient comprehension rather than clinical completeness, may be the primary source shaping an LLM’s response to a technically sophisticated patient query. The clinical incompleteness is invisible to the patient and to the AI.<\/p>\n\n\n\n

Orphan drug manufacturers should be treating advocacy organization content as a priority input to their AI information environment strategy. If the content on the advocacy site that an LLM is likely to cite is outdated, incomplete, or in tension with current prescribing information, the manufacturer needs to know. Providing advocacy partners with updated, AI-optimized content is one of the few levers available to manufacturers seeking to improve AI accuracy without generating promotional content subject to FDA review.<\/p>\n\n\n\n

Rare Disease Reddit Communities and Their Outsized Role in LLM Training Data<\/strong><\/h3>\n\n\n\n

Reddit communities organized around rare conditions are small by Reddit standards but enormous by rare disease information standards. A subreddit with 8,000 members representing patients with a condition affecting 50,000 Americans is capturing a meaningful fraction of the global patient population in a single crawlable text corpus. LLMs trained on Reddit content absorb the sentiment, the vocabulary, the concerns, and the misinformation of these communities with unusual efficiency relative to their actual population size.<\/p>\n\n\n\n

The r\/SpinalMuscularAtrophy subreddit, r\/ehlersdanlos, r\/cysticfibrosis, and dozens of smaller condition-specific communities have generated thousands of posts about specific drugs, treatments, access challenges, and clinical experiences. Some of this content is remarkably high quality \u2014 patient communities with medically sophisticated members produce sophisticated content. Some of it is dangerously wrong. LLMs do not reliably distinguish between the two, and they carry both into their pharmaceutical responses.<\/p>\n\n\n\n

Manufacturers with pharmacovigilance responsibilities for rare disease drugs should be running systematic social listening on these communities not only for traditional adverse event signal detection but for understanding what information environment is shaping the AI responses their patients will encounter.<\/p>\n\n\n\n

\n\n\n\n

Tracking AI Mentions of Rare Disease Drugs: Share of Voice in a Thin Information Market<\/strong><\/h2>\n\n\n\n

AI share-of-voice measurement is more tractable in rare disease than in large therapeutic categories, but the stakes are higher. In a category where a drug may have three competitors at most, a consistent AI preference for one over the others can have measurable commercial consequences. The volume of AI queries is lower than in diabetes or cardiovascular disease, but the proportion of patients who will encounter AI-generated drug information before speaking with a specialist is higher \u2014 because specialist access is itself a scarce resource in rare disease.<\/p>\n\n\n\n

How to Measure AI Brand Mentions for Orphan Drugs Across ChatGPT, Gemini, and Perplexity<\/strong><\/h3>\n\n\n\n

Standard AI share-of-voice methodology applies in rare disease, with modifications for thin query volume. Because rare disease AI queries are less frequent than common drug queries, systematic query testing needs to encompass a wider range of query variants to generate statistically meaningful results. The query library should include disease-state queries, symptom-based queries, treatment-category queries, and competitor-comparison queries, as well as condition-specific terminology that patients in these communities actually use.<\/p>\n\n\n\n

The key modifications for rare disease monitoring programs:<\/p>\n\n\n\n