Yesterday I wrote a post where I commented on my skepticism on the acupuncture paper in the Archives of Internal Medicine. I concluded that post by stating I do not have access to the entire article and I was awaiting some analysis from those who did. It didn’t take long.
First we have Steve Novella at Science Based Medicine. He begins by talking about the bias by the authors.
I took a close look at the study and find that the authors display considerable pro-acupuncture bias in their analysis and discussion. They clearly want acupuncture to work. That aside, the data are simply not compelling, and the authors, in my opinion, grossly overcall the results, which are compatible with the conclusion that there are no specific effects to acupuncture beyond placebo….
In other words – the unblinded comparison between acupuncture and no acupuncture is entirely overwhelmed by bias and completely useless. The no acupuncture control groups involved patients continuing to receive usual care (whatever they were already receiving that was not effective, or sometimes just being told not to get acupuncture). This was not a comparison to any specific medical intervention. In other words, the subjects were aware they were receiving no treatment.
He follows by explaining the difference between a statistical difference and a clinical difference. This was one of the first topics in my university statistics. A study of 10,000 people might show a drop in blood pressure from 145/100 to 142/95 with a 99 percent level of certainty, but it means nothing from a clinical perspective.
If a study shows no significant difference between true and sham acupuncture, then they argue that this placebo effect is enough to justify treatment. If the study (or in this case a meta-analysis) shows a small difference, then they use that small difference to justify the conclusion that acupuncture is real (even though the specific effects are negligible) and then use the large non-specific effects to justify the treatment.
Either way, proponents are inappropriately leveraging placebo effects (which are largely biases) to promote a treatment that has an effect size that is very small and, in my opinion, overlaps with no effect at all.
The authors make much of the small effect difference in their meta-analysis between true an sham acupuncture. They summarize their results by saying, if the no intervention group has a pain of 60%, then true acupuncture reduces it to 30% and sham acupuncture to 35%. While this difference was statistically significant in this meta-analysis, it is highly dubious to claim that the 5% difference is clinically significant, or even perceptible. To me this is no difference at all.
Orac at Respectful Insolence has a similar take on the paper. He comments on the selection criteria for studies to be included the analysis.
It is not required that all studies included have a sham placebo group. That means some studies were acupuncture versus no acupuncture controls, the latter of which could include groups that got anywhere from nothing to regular care. That’s just one problem that I see, because mixing studies that compare acupuncture to no treatment, to sham treatment, or to sham treatment and no treatment are comparing apples and oranges in a way. Pooling such studies is inherently problematic.
He also brings up clinical significance.
For patients with chronic pain, it’s uncommon to have a 50% reduction in pain scores, and the standard deviation they chose was rather large. By their own argument, even if there weren’t any methodological issues with the meta-analysis and their conclusions were completely justified, Vickers et al have just unwittingly made the argument that the effect of acupuncture might be statistically significantly greater than placebo effects but that it’s almost certainly not clinically significant. What Vickers et al are arguing is that a change of 5 on a 0-100 pain scale (which would be a change of 0.5 on a 0-10 pain scale), a subjective scale, is noticeable by patients. It’s probably not. There is a concept referred to as “minimally clinically important difference” (MCID) defined as “the smallest difference in score in the domain of interest which patients perceive as beneficial and which would mandate…a change in the patient’s management.”
One of the criteria for understanding studies and basing treatment options in science based medicine is prior plausibility, or the need for some basic scientific or medical background for the intervention. Lacking that, a strong proof for effectiveness must be demonstrated. Acupuncture has neither, and this paper does nothing to change that.
The other aspect in any treatment is an examination of the risks versus benefits. There is little or no advantage with the use of acupuncture, so lets take a look at the risks.
- Organ injury. If the needles are pushed in too deeply, they could puncture an internal organ — particularly the lungs. This is an extremely rare complication in the hands of an experienced practitioner.
- Infections. Licensed acupuncturists are required to use sterile, disposable needles. A reused needle could expose you to diseases such as hepatitis.
- Muscle Spasms These can be triggered by damaging a nerve with the needle
- Nerve Damage Puncturing a nerve can have other effects beyond minor muscle spasms.
- Delaying proper medical treatment Both acute and chronic pain can have direct physiological causes. Even if a patient finds some pain relief from acupuncture, it does nothing to treat the underlying cause. Delaying proper medical treatment can cause direct harm to the patient.
Although some of these are rare, there is no direct reporting of any complications, so there is no way to determine the frequency of any of these potential consequences. As far as I know, there is no obligation for acupuncturists to advise patients of the potential for harm.