95% Confidence Intervals, Confidence levels and p values compared
Whether doctor or patient, deciding on what constitutes the best treatment for a particular disease is often based on the published medical literature. How medical studies are reported, and the methods used to interpret results (including traditional methods of 95% confidence intervals and p values), all play vital roles in medical decision-making.
In the 1990's, the medical literature was flooded with editorials and articles claiming the superiority of 95% confidence intervals over p values when reporting research results. However this conjecture was based entirely on personal opinion, the lowest level of evidence available. The lack of evidence for 95% confidence intervals has been noted by others (eg: S.D. Walter, Methods of reporting statistical results from medical research studies, Am J Epidemiol 141 (1995) 896-906).
In the spirit of exisiting low level evidence, we have our own opinion. One way that we believe may lead to improved reporting of results, leading to less chance of misinterpretation and more informed decisions, is by use of confidence levels, clinical significance curves, and confidence contours (either instead of, or as a complement to, 95% confidence intervals and p values). These methods have been published previously, and we cite examples where results reported with p values and 95% confidence intervals alone have been misinterpreted by both eminent authors (eg WHO collaborators) and journal readers, might have been correctly interpreted using our methods:
Shakespeare TP, Gebski VJ, Veness MJ, Simes J. Improving interpretation of clinical studies by use of confidence levels, clinical significance curves, and risk-benefit contours. Lancet. 2001: 357: 1349-53.
In brief, our methods allow the reader of a study (whether doctor or patient), to know the confidence (or probability) that one treatment is better than another, and by how much. They also can show the confidence that a treatment does not have excessive side-effects. To find out more about relative benefits of confidence levels over 95% confidence intervals, please read the Lancet article above (abstract available here). A discussion of the problems with confidence intervals, and the benefits of confidence levels can also be found in a Statistics Supercourse lecture hosted by the University of Pittsburgh. A discussion of confidence intervals was also presented at ASTRO (American Society for Therapeutic Radiology and Oncology) 2001.
Not satisfied with the available level of evidence, we conducted two studies
(PRIMER 1 and PRIMER 2) to determine the best methods of reporting research results: p values, 95% confidence intervals, confidence levels, or a combination. Results have been published in the journal "Medical Decision Making". In short, the results demonstrate that doctors do not interpret study results very well when the results have been presented with p values, or p values and 95% confidence intervals. We found that by adding confidence levels, doctors were better at interpreting study results, as well as better at implementing research results into practice. The conclusion is that anyone publishing research results should present results with p values, 95% confidence intervals, and confidence levels. The abstract of the results is available here.
The reference is:
Shakespeare TP, Gebski V, Tang J, Lim K, Lu JJ, Zhang X, Jiang G. Influence of the way results are presented on research interpretation and medical decision making: the PRIMER collaboration randomized studies. Med Decis Making. 2008; 28(1): 127-37.
To free statistical software that calculates confidence interval, confidence level, clinical significance crve and risk-benefit contours for two arm studies.
