Artificial but not intelligent? Asking the simplest question

The horrible situation of a young woman, who was left untreated for several years, because her doctor considered a serum B12 concentration above 130 pmol/l as sufficient, has led me to question the following: why is solid medical information on how and when to treat people with B12 deficiency so poorly known? As long as you consider a few specific rules, treating people with B12 deficiency is not rocket science. These particular rules have been known for several decades, and a few years ago, they were summarised in the NICE guidance on B12 diagnosis and treatment.

As a matter of fact, in 2014, two Dutch GPs published an article on B12 deficiency in a Dutch-language journal. The article ends with a sentence like “during B12 treatment, there is no need to check the B12 concentration in the blood, as this is supposed to be high.” And 130 pmol/l is NOT high! So, where else can we find this utmost important information regarding managing B12 deficiency? Google? Yahoo? Altavista? Or any other old-fashioned search engine?

We are living in the era of artificial intelligence. Wherever you look, this epidemic is evident. Several programs and systems are available for anyone who wants to use AI for specific tasks or to ask questions of any kind. Unfortunately, the quality of AI Systems may differ considerably. Some AI systems admit that they may make mistakes or indicate that their responses cannot be considered specific medical advice.

I have one favourite question, which I have posted on several AI Systems, to see how well the answer of the AI applies to the daily clinical practice of supporting people with B12 deficiency. This question is:

So here we have a number of answers:

Claude 3 Haiku – Anthropic (took 2.355s & 882 credits)
Based on the information provided, the doctor should measure the patient’s serum B12 concentration periodically, but the exact frequency would depend on several factors:

Initially, the doctor should measure the serum B12 concentration every 3-6 months to ensure the 1000 mcg B12 injection every two weeks is maintaining adequate B12 levels and resolving the pernicious anaemia.

Once the patient’s B12 levels have been stable and the patient remains asymptomatic for some time, the frequency can be reduced to once or twice per year.

The doctor may also want to measure B12 levels if the patient reports any new or returning symptoms of pernicious anaemia, as this could indicate a need to adjust the treatment regimen.

Factors like the patient’s age, overall health status, and adherence to the B12 injection schedule may also influence the appropriate monitoring frequency.
The key is to establish a baseline, monitor regularly to confirm the effectiveness of the treatment, and then adjust the frequency based on the individual patient’s clinical response and stability of B12 levels. Ongoing monitoring is important to ensure the pernicious anaemia remains well-controlled.

GPT-4o Mini – OpenAI (took 8.529s & 194 credits)
For a patient with pernicious anaemia receiving vitamin B12 injections, it is generally recommended to check serum B12 levels about once a year. This is typically sufficient to ensure that the treatment is effective and that the patient is maintaining adequate vitamin B12 levels. However, if there are any concerns regarding symptoms, adherence to treatment, or other complications, more frequent monitoring may be warranted. It’s important for the physician to assess the clinical context and adjust the frequency of monitoring as needed.

Gemini 1.5 Flash – GoogleAI (1.459s – 148 credits)
The frequency of serum B12 monitoring in this patient is a clinical judgment call, balancing the efficacy of the current treatment with the risks and benefits of frequent blood tests. Given she’s asymptomatic and responding well to the current regimen (1000 mcg B12 injection every two weeks), monitoring every 3-6 months might be sufficient. However, some clinicians might advocate for annual monitoring. More frequent checks (e.g., every 2-3 months) might be considered if there were any changes in her clinical status or if her symptoms recurred. There’s no single universally accepted guideline.

Grok (regular mode)
For a 50-year-old woman with pernicious anemia treated with 1000 mcg B12 injections every two weeks and who is symptom-free, it is generally recommended to measure serum B12 concentration annually or if new symptoms arise. This ensures the treatment remains effective and allows for adjustments if needed. Always consult her healthcare provider for personalized advice.

ChatGPT
In a patient with pernicious anaemia on regular B12 injections who is clinically stable and asymptomatic, routine measurement of serum B12 is not necessary.

Once treatment has started and the patient responds (symptoms resolve, blood counts normalize), B12 levels no longer correlate well with tissue sufficiency because circulating B12 can be very high after injections. Monitoring should focus on clinical status and blood counts (CBC, MCV, reticulocytes initially). Some guidelines suggest checking hematologic response at the start (after a few weeks to months), then no regular B12 monitoring is needed.

Serum B12 levels may be checked only if:
There is doubt about adherence or absorption (though with injections, absorption is bypassed).
The patient develops unexplained new symptoms.
There’s a need to assess overtreatment in unusual cases.

📌 Answer: In a patient with pernicious anaemia well-controlled on regular injections, serum B12 does not need to be monitored routinely at all.
Would you like me to also outline what should be monitored instead (e.g., blood counts schedule, neurological checks, folate/iron status)?

Panic attacks?

Now, when you are a member of the Pernicious Anaemia Society, it is rather likely that you are treated with vitamin B12 injections, or are treating yourself with B12 injections. Your experiences in daily medical practice may be similar to some of the answers of the AI Systems listed above. Too often, doctors measure serum B12 in any person who is injecting regularly, and then conclude that serum B12 is elevated. Some inexperienced doctors have been known to get a panic attack and swiftly order the person with pernicious anaemia to stop injecting because the serum concentration of B12 is too high. On the other hand, we also know doctors (see the introduction of this blog) who measure serum B12 every year, and tell the person with pernicious anaemia that she will only get a new B12 injection when serum B12 becomes lower than 130 pmol/l. Both examples are well-known horror stories for people with pernicious anaemia.

ChatGPT the winner

So, I was rather happy with the answer provided by ChatGPT: “In a patient with pernicious anaemia on regular B12 injections who is clinically stable and asymptomatic, routine measurement of serum B12 is not necessary”. After all, this agrees with the instructions in our Pharmacotherapeutic Compass, and also corresponds well with the viewpoint of two Dutch G.P.’s, which I mentioned earlier: this paper, published 11 years ago clearly stated that in people treated with B12 injections, serum B12 should not be measured because it is supposed to be high. And indeed, the frequency of B12 injections is guided by symptoms, not by irrelevant blood measurements of B12, methylmalonic acid or any other biomarker. Now you may think that this would place a large burden on medical practices, because they have to talk to many people to estimate and try to understand every person’s clinical situation and symptoms. This is not true. The majority of people with PA or B12 deficiency are perfectly capable of judging their own clinical status and symptoms, and evaluating whether their current B12 treatment regimen is adequate. Some may experience a subtle recurrence of symptoms when they miss or postpone an injection or when an intercurrent disease mandates the temporary increase of injection frequency. It may become more difficult when two diseases in one person share symptoms, like in a person with PA who also has a thyroid disorder or iron deficiency. Then, it may be more challenging to assess whether a symptom is related to disease A or disease B.

Frequency of B12 injections

Did you notice that the question I posed to the various AI systems describes a person who injects B12 every two WEEKS!? Research has shown that 30% of the members of this Society inject once a week or even more frequently. Yet, many people believe that one B12 injection every two months is sufficient for every person with PA. This is because no, I repeat NO, single scientific study has critically and extensively evaluated different B12 treatment regimens and/or frequencies. Many years ago, in the old times when we had no Excel spreadsheets, a doctor calculated on the back of a package of cigars that one B12 injection, which contains 1000 mcg, should last at least two months. After all, our daily need is 4 mcg, so 1000 mcg (of which 200 mcg was supposed to remain in the body in the days after injection) would be at least enough for fifty days. Nevertheless, when you look carefully at the experiences of doctors, last century, in the 1960s, some of them were very keen on closely monitoring their patients. They reported that many patients needed more frequent injections to remain free of symptoms than the ‘calculated’ once every two months frequency.

Thinking that all people with PA can be managed with one injection every two months is as dumb as thinking that every person with diabetes can be sufficiently treated with 20 units of long-acting insulin each day, because the fact that in people without diabetes, the average pancreas manufactures only 20 such units each day and, by this, keeps the blood glucose normal. Unfortunately, in daily practice, we observe that to maintain blood glucose levels within the near-normal range, the dose of insulin for people with diabetes may vary from person to person, somewhere between 20 and 200 units per day….. Luckily for people with diabetes, blood glucose can easily be measured to estimate, together with intake of carbohydrates and level of physical activity, the insulin dose they need. People with PA do not have a specific measure to evaluate their B12 needs. When a person with diabetes forgets an insulin injection, blood glucose levels will rise considerably within a matter of hours. When a person with PA forgets a B12 injection, they may only notice this after several days.

Please note that I have posed the question to all these AI systems in mid-September. Systems keep learning, and it may be that their answer would be different now, three months later. However, AI systems don’t have an “I thought about this, and changed my mind” feature, at least, not one I know. And some AI systems may be modified by their owner when they give an answer he doesn’t like.

Bruce Wolffenbuttel

PS.
October 31, Halloween evening.
Today, one of my friends pointed me to Perplexity, so we asked the same question to this AI. Not only gave Perplexity a similar answer to ChatGPT, but it also provided a list of 13 links, e.g. references.
“For a patient with pernicious anaemia who is stable and symptom-free on regular vitamin B12 injections, routine measurement of serum B12 concentration is generally not required for ongoing monitoring. Instead, most guidelines and experts advise monitoring should focus on symptoms and general health, rather than relying on repeat B12 blood tests[1][2][13].” Part of its output looks like this, clicking the link directly brings you to the source.