Flu Vaccine: Dishonest stat-skeptics

Well, I haven't found anyone else writing about this, probably because it's downright silly, but there's some people out there arguing that the seasonal influenza vaccine is only 1.5% effective.

Specifically, what happened is that a study published in the prestigious journal The Lancet revealed that flu shots provide only “moderate protection” against the flu, and in some seasons is altogether “reduced or absent.”

Specifically, the Lancet said the vaccine is about 59 percent effective. But when you break the numbers down statistically, what it really works out to is that the vaccine prevents flu 1.5 times out of 100.

That’s right. Using the Lancet’s own numbers, statistics show that the vaccine only works 1.5 times out of 100.

Wow.   Sounds pretty damning, and if it were actually true, I might have to rethink my position on the flu vaccine debate.  What we have here is very creative statistics.  The link they provide regarding the 1.5 times out of 100 effectiveness goes to a letter to the editor written by "J.L. Craig, BSN, PhD."  I'm fairly certain the individual in question is one Jennifer Craig, who has written several books against vaccinations and is now retired, so I haven't had much luck on a cursory search of PubMed to see what kind of research she did.  One of my favorite authors over at science-based medicine did a book review of a bunch of anti-vax books, and one of hers was critiqued (It's #6 if you care to get his impression).  She's very clearly anti-vax from her statements, and if this is the same individual then there's some clear bias going in.  That's fair, though, I have clear bias towards vaccination.  Let's look instead at the statistical ploy she uses in her letter:

... let’s examine the study to see how this spin transpired.

This was a meta analysis, meaning that the researchers used data from 28 previously published random controlled trials between 1967 and 2011. The control group, n=13,095, consisted of non-vaccinated adults who were monitored to see if they got confirmed influenza. Over 97 per cent of them did not. Only 357 got flu which means that 2.73 per cent of these adults got the flu in the first place.

The treatment group comprised adults who were vaccinated with a trivalent inactivated influenza vaccine. According to the study, 1.18 per cent got the flu.

The difference between these two groups (2.73 – 1.18) is 1.5 people out of 100. In other words, the flu vaccine did nothing for 98.5 per cent of adults in the studies.

Now in all fairness, she is technically correct.  In those studies 98.5% of people who had the flu shot were unaffected.  They were either already not going to get the flu shot or they got the flu even after having gotten the shot.  Only 1.5% of the people who got the flu shot benefited directly from it.

Does this mean that the flu shot was only 1.5% effective?  Well, no.  Let us say that, in some hypothetical world, a flu vaccine was developed that was perfect.  If you got this vaccine, you were guaranteed not to get the flu.  Not a single person got sick.  We would all agree that this shot was 100% effective.  If you got it, you wouldn't get the flu.  According to her statistical analysis and assuming the same morbidity from the study, the difference between vaccinated and unvaccinated would be (2.73-0.00) or 2.73 people out of 100.  So even with an absolutely, 100% perfect flu shot, it would still do nothing for 97.27 per cent of the population.  Does that mean the shot was only 2.73% effective?

The problem here seems to lie in the numbers.  Our puny mammalian brains have trouble comprehending numbers and making them work.  That's why it's easy for people to get confused by billions vs millions and for us to see a vaccine that cuts the number of infected people by more than half as "only 1.5% effective."  Statistics are easy to manipulate, and it's very easy to use numbers to make your case sound better.

What really gets me is Dr. Craig goes on to accuse the cdc of lying with statistics:

So where did the media get 60 per cent effective? It’s called lying with statistics. First you take the 2.73 per cent in the control group who got flu and you divide that figure into the 1.18 per cent of the treatment group who got the flu. This gives you 0.43.

You then say that 0.43 is 43 per cent of 2.73 and claim that the vaccine results in a 57 per cent decrease in flu infections. This becomes the 60 per cent effectiveness claim.

Erm, Hi.  Pot, Kettle.  Perhaps you've met?  If we really wanted to lie with statistics, we would argue that, since only 1.18% of vaccinated people get the flu, the flu shot must be 98.82% effective.  Now there's some blatant dishonesty for you.

One thing that's also difficult to work with here is that none of these studies involved working with the flu at pandemic levels.  60% efficacy only means a percentage of the population when flu levels are fairly low, but should they reach levels such as in 1918, where roughly 500 million were infected, that means 250 million people avoid the flu if they were all vaccinated.  (At the time that's roughly 30% of the population...that would mean, hopefully, that only 15% of the vaccinated population would get sick).  This is all hypothetical, of course.  More studies would be needed.  Further, the presence of protective immunity may also further reduce the number of people who get sick from the flu.  The higher the number of vaccinated people and the better the vaccine, the greater the herd immunity.  We like herd immunity, since it means the immunodeficient get some protection as well.

I will say one last thing: the mercola article is correct in pointing out that this lancet study showed zero efficacy in some seasons.  That happens.  It's always an educated guess as to which strains of influenza will hit us from year to year, and it would be unreasonably costly to vaccinate us from every flu strain we're aware of.  Even then, it might mutate into something we're entirely unprepared for.  That's part of why we will probably never beat the flu the way we beat smallpox.  I wouldn't ever argue that the flu vaccine is perfect, or even close to perfect.  The only argument I make is that it's the best defense we currently have against a virus that has a proven history of causing pandemic level infections in our population.  You may disagree, and that's fine.  I'm certain I'll get at least one comment arguing that the flu vaccine is bad or, at the very least, unnecessary, and that's fine, too.  My biggest beef is only when others take statistics that say "well, maybe it's only 60% effective instead of 78%" and instead argue that the flu shot has been proven to be pointless.  Don't do that.  It's dumb.

Further Reading:

Mark Crislip on Flu Vaccine Efficacy

WebMD Flu Vaccine FAQ