Flu Vaccine: Dishonest stat-skeptics

Well, I haven't found anyone else writing about this, probably because it's downright silly, but there's some people out there arguing that the seasonal influenza vaccine is only 1.5% effective.

Specifically, what happened is that a study published in the prestigious journal The Lancet revealed that flu shots provide only “moderate protection” against the flu, and in some seasons is altogether “reduced or absent.”

Specifically, the Lancet said the vaccine is about 59 percent effective. But when you break the numbers down statistically, what it really works out to is that the vaccine prevents flu 1.5 times out of 100.

That’s right. Using the Lancet’s own numbers, statistics show that the vaccine only works 1.5 times out of 100.

Wow.   Sounds pretty damning, and if it were actually true, I might have to rethink my position on the flu vaccine debate.  What we have here is very creative statistics.  The link they provide regarding the 1.5 times out of 100 effectiveness goes to a letter to the editor written by "J.L. Craig, BSN, PhD."  I'm fairly certain the individual in question is one Jennifer Craig, who has written several books against vaccinations and is now retired, so I haven't had much luck on a cursory search of PubMed to see what kind of research she did.  One of my favorite authors over at science-based medicine did a book review of a bunch of anti-vax books, and one of hers was critiqued (It's #6 if you care to get his impression).  She's very clearly anti-vax from her statements, and if this is the same individual then there's some clear bias going in.  That's fair, though, I have clear bias towards vaccination.  Let's look instead at the statistical ploy she uses in her letter:

... let’s examine the study to see how this spin transpired.

This was a meta analysis, meaning that the researchers used data from 28 previously published random controlled trials between 1967 and 2011. The control group, n=13,095, consisted of non-vaccinated adults who were monitored to see if they got confirmed influenza. Over 97 per cent of them did not. Only 357 got flu which means that 2.73 per cent of these adults got the flu in the first place.

The treatment group comprised adults who were vaccinated with a trivalent inactivated influenza vaccine. According to the study, 1.18 per cent got the flu.

The difference between these two groups (2.73 – 1.18) is 1.5 people out of 100. In other words, the flu vaccine did nothing for 98.5 per cent of adults in the studies.

Now in all fairness, she is technically correct.  In those studies 98.5% of people who had the flu shot were unaffected.  They were either already not going to get the flu shot or they got the flu even after having gotten the shot.  Only 1.5% of the people who got the flu shot benefited directly from it.

Does this mean that the flu shot was only 1.5% effective?  Well, no.  Let us say that, in some hypothetical world, a flu vaccine was developed that was perfect.  If you got this vaccine, you were guaranteed not to get the flu.  Not a single person got sick.  We would all agree that this shot was 100% effective.  If you got it, you wouldn't get the flu.  According to her statistical analysis and assuming the same morbidity from the study, the difference between vaccinated and unvaccinated would be (2.73-0.00) or 2.73 people out of 100.  So even with an absolutely, 100% perfect flu shot, it would still do nothing for 97.27 per cent of the population.  Does that mean the shot was only 2.73% effective?

The problem here seems to lie in the numbers.  Our puny mammalian brains have trouble comprehending numbers and making them work.  That's why it's easy for people to get confused by billions vs millions and for us to see a vaccine that cuts the number of infected people by more than half as "only 1.5% effective."  Statistics are easy to manipulate, and it's very easy to use numbers to make your case sound better.

What really gets me is Dr. Craig goes on to accuse the cdc of lying with statistics:

So where did the media get 60 per cent effective? It’s called lying with statistics. First you take the 2.73 per cent in the control group who got flu and you divide that figure into the 1.18 per cent of the treatment group who got the flu. This gives you 0.43.

You then say that 0.43 is 43 per cent of 2.73 and claim that the vaccine results in a 57 per cent decrease in flu infections. This becomes the 60 per cent effectiveness claim.

Erm, Hi.  Pot, Kettle.  Perhaps you've met?  If we really wanted to lie with statistics, we would argue that, since only 1.18% of vaccinated people get the flu, the flu shot must be 98.82% effective.  Now there's some blatant dishonesty for you.

One thing that's also difficult to work with here is that none of these studies involved working with the flu at pandemic levels.  60% efficacy only means a percentage of the population when flu levels are fairly low, but should they reach levels such as in 1918, where roughly 500 million were infected, that means 250 million people avoid the flu if they were all vaccinated.  (At the time that's roughly 30% of the population...that would mean, hopefully, that only 15% of the vaccinated population would get sick).  This is all hypothetical, of course.  More studies would be needed.  Further, the presence of protective immunity may also further reduce the number of people who get sick from the flu.  The higher the number of vaccinated people and the better the vaccine, the greater the herd immunity.  We like herd immunity, since it means the immunodeficient get some protection as well.

I will say one last thing: the mercola article is correct in pointing out that this lancet study showed zero efficacy in some seasons.  That happens.  It's always an educated guess as to which strains of influenza will hit us from year to year, and it would be unreasonably costly to vaccinate us from every flu strain we're aware of.  Even then, it might mutate into something we're entirely unprepared for.  That's part of why we will probably never beat the flu the way we beat smallpox.  I wouldn't ever argue that the flu vaccine is perfect, or even close to perfect.  The only argument I make is that it's the best defense we currently have against a virus that has a proven history of causing pandemic level infections in our population.  You may disagree, and that's fine.  I'm certain I'll get at least one comment arguing that the flu vaccine is bad or, at the very least, unnecessary, and that's fine, too.  My biggest beef is only when others take statistics that say "well, maybe it's only 60% effective instead of 78%" and instead argue that the flu shot has been proven to be pointless.  Don't do that.  It's dumb.

Further Reading:

Mark Crislip on Flu Vaccine Efficacy

WebMD Flu Vaccine FAQ

Law does not equal Science. Say it with me now.

Fox news doesn't get it.  Feel free to take a look at that video.  Watch it as long as you can stomach, as they talk about this "bombshell report."  In fact, if you want, feel free to go look at the report itself.  Note that it's a legal report, not a scientific report.  Note that it even says clearly "This assessment of compensated cases showing an association between vaccines and autism is not, and does not purport to be, science." (p. 482 for those following along)

Law does not equal science.  Legal rulings do not determine science.  If you want to prove that vaccines cause autism, you do it using science, not using the logic that "if someone paid me compensation for having autism, I must have autism."  It seems an underhanded tactic to use this faulty logic to try to argue your point.  It's like running to dad for permission when mom already said no (moms are the smart ones, aren't they?).

Not that these cases really involved individuals with autism, either.  David Gorski yet again writes about this much more eloquently than I could.  Of note is his reference to the weakness in their legal study, since it's really not anything remotely close to a scientific study.

Sorry for the short post here.  I'd like to write more on it, but I'd have to read a lot more into this paper than I have the stomach to bother with, and I'm currently studying for my final in a class on cancer.  Scary stuff, cancer.  I find myself constantly looking at my skin these days, fearing I'll develop a new spot somewhere.  Two people I know got diagnosed with cancer this semester, one of which was my father.  Still, it's interesting, which is why I study it.

Update: As another interesting example of law messing up the science.  The Florida Senate accidentally outlawed sex.  It's not perfectly related to the antivax story above, but it's hilarious and I thought I should share.

Update^2: I suspected as much (as funny as it originally was), but the ruling did not *really* accidentally outlaw sex.  It's still hilarious based on the wording of the law.

Vaccines don't raise IMR. Period.

David Gorski wrote a great article ripping apart the latest in a series of poorly done studies linking infant mortality rate to vaccines on the blog Science Based Medicine.  Since Dr. Gorski is a little long-winded, I'd like to pull out the real big points for you here.  If you want to look at the study in question (which was in a peer-reviewed piece of literature), you can find it here.  Of course it's free, the antivaxxers are happy to toss money at a study like this to give open access to everyone.

As Gorski points out, this was a ridiculously simple paper.  The language was simple, the methods were simple, the entire process was simple.  I could have written most of this paper (sans the research/sources) within a week, and I'm not even a grad student yet.  They took some data about infant mortality in the US and other countries, compared them to number of vaccines given, and produced this lovely excel chart.

The thing that struck me when I looked at this chart was how all-over-the-place these data points were.  If you ignore the line they drew right through the middle, this wouldn't strike me as a great model for a linear fit.  As Gorski points out

Be that as it may, I looked at the data myself and played around with it One thing I noticed immediately is that the authors removed four nations, Andorra, Liechenstein, Monaco, and San Marino, the justification being that becayse they are all so small, each nation only recorded less than five infant deaths. Coincidentally, or not, when all the data are used, the r²=.426, whereas when those four nations are excluded, r²increases to 0.494, meaning that the goodness of fit improved. Even so, it’s not that fantastic, certainly not enough to be particularly convincing as a linear relationship.

The data isn't all that convincing, and worse yet, it's not all that rigorous.  The paper states that it pulled all of its information from a 2009 report.  Two years ago, and a single year taken (and many countries excluded).  Gorski points this out better than I could:

Miller and Goldman only looked at one year’s data. There are many years worth of data available; if such a relationship between IMR and vaccine doses is real, it will be robust, showing up in multiple analyses from multiple years’ data. Moreover, the authors took great pains to look at only the United States and the 33 nations with better infant mortality rates than the U.S. There is no statistical rationale for doing this, nor is there a scientific rationale. Again, if this is a true correlation, it will be robust enough to show up in comparisons of more nations than just the U.S. and nations with more favorable infant mortality rates. Basically, the choice of data analyzed leaves a strong suspicion of cherry picking. 

It's possible that they didn't cherry pick, in which case they weren't rigorous enough.  If the only result of this test is that it gets others to look at other data, so much the better.  Still, there's one last big problem that Gorski cites with the paper that makes the data look worse still.  He quotes from Bernadine Healy, M.D. , who says:

First, it’s shaky ground to compare U.S. infant mortality with reports from other countries. The United States counts all births as live if they show any sign of life, regardless of prematurity or size. This includes what many other countries report as stillbirths. In Austria and Germany, fetal weight must be at least 500 grams (1 pound) to count as a live birth; in other parts of Europe, such as Switzerland, the fetus must be at least 30 centimeters (12 inches) long. In Belgium and France, births at less than 26 weeks of pregnancy are registered as lifeless. And some countries don’t reliably register babies who die within the first 24 hours of birth. Thus, the United States is sure to report higher infant mortality rates. For this very reason, the Organization for Economic Cooperation and Development, which collects the European numbers, warns of head-to-head comparisons by country.

Infant mortality in developed countries is not about healthy babies dying of treatable conditions as in the past. Most of the infants we lose today are born critically ill, and 40 percent die within the first day of life. The major causes are low birth weight and prematurity, and congenital malformations. As Nicholas Eberstadt, a scholar at the American Enterprise Institute, points out, Norway, which has one of the lowest infant mortality rates, shows no better infant survival than the United States when you factor in weight at birth.

Go ahead.  Go back and read that quote again.  In fact, I'll highlight the spot that struck me as most noteworthy: "Norway, which has one of the lowest infant mortality rates, shows no better infant survival than the United States when you factor in weight at birth."  In other words, IMR in the US is relatively low compared to other countries, which flies directly in the face of their previous data.  Not that I'd suggest using this data to begin with.  Pulling from IMRs is a load of rubbish since you don't have a consistent definition for what constitutes infant mortality, and the authors of the paper certainly didn't make any effort to clarify that.  They do like to make some pretty hefty conjectures about SIDS and its connection to vaccines, though.  From the paper itself:

Although some studies were unable to find correlations between SIDS and vaccines, there is some evidence that a subset of infants may be more susceptible to SIDS shortly after being vaccinated. For example, Torch found that two-thirds of babies who had died from SIDS had been vaccinated against DPT (diphtheria–pertussis–tetanus toxoid) prior to death. Of these, 6.5% died within 12 hours of vaccination; 13% within 24 hours; 26% within 3 days; and 37%, 61%, and 70% within 1, 2, and 3 weeks, respectively.

In the interest of being fair and not just looking to David Gorski's analysis of this, I decided to give them the benefit of the doubt about this Torch study and I looked it up.  Or rather, I tried to.  The report was from 1982.  Most of my sources go back to 1985 at best.  Doing a google search I found a few articles from anti-vaccination sites accusing scientists of silencing Torch because he "dared to use anectdotal data."  Almost everything I found was from anti-vaccination sites and generally involved something like this

Torch's report provoked an uproar in the American Academy of Pediatrics. At a hastily arranged press conference he was soundly chastised for using "anecdotal data," meaning (will you believe it?) that he actually interviewed the families concerned!

That's the best I have to offer.  A few references to his work, but very little results for the work itself.  I'm not the best at searching and don't have all the tools others have to find this information, so it's probably out there somewhere.  I'm just not able to find it myself.  Besides, I tend to be hesitant with any controversial data that's older than 1990 or so.  The older it gets, the more likely it is that there's a newer study with better information.  It's hard to say, though.  What I can say is that the American Academy of Pediatrics was right to react as they did if he used anecdotal evidence.  Anecdotal evidence is a great place to start research, but it's a horrible place to end it.  Anecdotal evidence raises the questions, but it does not give the answers.