Study: Most physicians overestimate probability of successful medical outcomes

In a recent survey study, nearly 80% of physicians overestimated the probability of successful patient outcomes following multistep medical procedures.

The findings highlight “enormous opportunities in medical education to improve the curriculum in terms of teaching the importance of probability in medical settings,” Hal Arkes, PhD, emeritus professor of psychology at The Ohio State University, said in a press release.

“Numbers are the most reliable source of correct decisions in medicine,” he added.

Arkes and colleagues conducted the study to determine how often a phenomenon known as conjunction fallacy occurs in medicine. According to the researchers, conjunction fallacy is a “misestimation of the overall probability of success when two or more independent events are involved.”

“The probability of a conjunction of two independent events is the product of the probabilities of the two components and therefore cannot exceed the probability of either component,” the researchers wrote in JAMA Network Open. “A violation of this basic law of probability is called the conjunction fallacy.”

Physicians make medical decisions by estimating the probability of what will happen based on a sequence of independent events, Arkes and colleagues wrote, but “little is known about physicians’ ability to accurately estimate the overall probability of success in these situations.”

To learn more, the researchers conducted three substudies via online surveys in which 215 physicians estimated the probability that two components of a diagnostic or prognostic sequence would be successful and the overall probability of success of the two-step sequence.

Each substudy described a different medical scenario.

The first scenario described a brow presentation during labor. A physician had to “consider the probability of the brow presentation converting to a deliverable position and the probability of delivering vaginally from that converted position,” the researchers wrote.

The second scenario was a “diagnostic evaluation of an incidentally discovered pulmonary nodule,” with the two conjuncts being the probabilities that the nodule was cancerous and of the biopsy successfully detecting the cancer.

The final scenario was a modification of the first so as to “debias the conjunction fallacy prevalent in the first substudy,” the researchers wrote.

“Our debiasing survey required respondents to consider the conjunction’s components before estimating the overall probability,” they wrote. “This approach contrasted with that of our first two substudies, in which we asked for the two components’ probability estimates after the overall probability estimate was rendered. Thus, if any physician had realized the component probabilities should have been considered when contemplating the previous overall estimate, it was too late to rectify the overall estimate because respondents were not allowed to change their previous responses as they progressed through the successive frames of the survey.”

Of the respondents, 78.1% “estimated the probability of a medical outcome resulting from a two-step sequence to be greater than the probability of at least one of the two component events,” which the researchers called “a result that was mathematically incoherent.” They further reported that physicians overestimated the combined probability of success by 12.8% in the first scenario, 19.8% in the second scenario and 18% in the third scenario.

In the third scenario, the researchers said that “calculating the conjunctive estimate should have been relatively easy if physicians were aware of the multiplication rule.” However, they added that only one physician “correctly estimated the conjunction probability to be exactly equal to the product of the components’ estimate.”

Factors like the gender of the physician and time since obtaining a medical degree did not have a significant impact on the levels of overestimation, according to Arkes and colleagues. However, knowledge of multiplication rules likely did, the researchers concluded, writing that “many physicians may not be facile in the calculation of probability or even basic numeracy.”

“The findings of this survey study suggest that physician misestimation of the probability of medical outcomes may be common,” they wrote. “Estimating the successful outcome of a multistep procedure is a common task among physicians. If this task is performed in a logically flawed manner, overall estimates will be inaccurate.”

References:

• Arkes H, et al. JAMA Netw Open. 2022;doi:10.1001/jamanetworkopen.2022.18804.
• Doctors overestimate success of multi-step medical procedures. https://www.eurekalert.org/news-releases/956929. Published June 27, 2022. Accessed July 8, 2022.