Study showing high effectiveness for H1N1 vaccine stirs debate

Feb 11, 2011 (CIDRAP News) – A recent case-control study estimates that the adjuvanted pandemic H1N1 influenza vaccine used in Canada was effective 93% of the time, a substantially higher figure than seen in several other studies and one that has raised some eyebrows among other flu vaccine researchers.

In the study, patients who visited sentinel physicians for treatment of influenza-like illness were tested for the pandemic virus and asked whether or not they had been vaccinated. The virus was confirmed in 38% of those tested, and only 1% of those patients had been vaccinated, whereas 17% of those who tested negative had been vaccinated. That translated into an estimated effectiveness of 93%.

The small number of vaccine failures limits the statistical strength of the results, but the findings nonetheless suggest that the vaccine was "highly effective in preventing medically attended, laboratory confirmed pandemic H1N1 illness," says the report, published online Feb 3 by BMJ (formerly the British Medical Journal).

The 93% effectiveness found in the study is well above what was seen in several other observational studies of H1N1 vaccine effectiveness, which have yielded results mostly in the 60% to 75% range. Some other researchers suggest that the findings may reflect the effects of confounding variables that the authors didn't or couldn't adjust for, such as the lateness of the vaccination campaign relative to the pandemic peak or a bias in the selection of patients for testing. They don't doubt that the pandemic vaccine was generally effective but question whether its effectiveness really exceeded 90%.

Study covered much of Canada
The study was conducted by researchers from several universities and public health agencies in British Columbia, Alberta, Quebec, and Ontario, with Danuta Skowronski, MD, MHSc, of the British Columbia Centre for Disease Control as the first author. They used a network of more than 500 community-based sentinel physicians who contribute to flu surveillance each year.

The clinicians were given kits for providing respiratory specimens and were asked to test, at their discretion, consenting patients who presented for treatment within 7 days after getting sick with flu-like symptoms. Patients were asked for information on their immunization status, including receipt of the pandemic vaccine and seasonal flu vaccine, and whether they had any of several high-risk chronic conditions.

The study focused on GlaxoSmithKline's Arepanrix vaccine, a monovalent product that contains the company's AS03 adjuvant. It constituted about 95% of the pandemic vaccine doses distributed in Canada in the fall of 2009, the report says.

The researchers picked the period of Nov 8 through Dec 5—weeks 45 through 48—as their primary analysis period, because vaccine first became available in week 43, as the pandemic was peaking, and there was little flu activity after week 48. They considered patients to have been immunized if they were vaccinated at least 2 weeks before the onset of their illness, since it takes about that long for flu vaccines to induce a protective immune response.

Effectiveness topped 90%
After a paperwork problem forced the exclusion of about 300 specimens from Ontario patients, the researchers included 552 patients in their analysis. Of those, 209 (38%) tested positive for the pandemic virus. Adults 20 to 49 years old made up 46% of those who tested positive, while children 6 months to 9 years accounted for 24% and 10- to 19-year-olds for 19%.

Just 1% (2 of 209) of the patients who tested positive for H1N1 reported receiving the pandemic vaccine at least 2 weeks before they got sick, versus 17% (58 of 343) of those who tested negative. One of the two "vaccine failures" was a healthy 9-year-old child, while the other was a young adult with an underlying illness.

When the authors adjusted for differences between cases and controls in age, chronic conditions, week of illness onset, and other factors, overall vaccine effectiveness was 92.7% (95% confidence interval [CI], 68.6-98.3). The fully adjusted effectiveness estimate was only slightly lower—91.1%—when participants aged 50 and older were excluded from the analysis.

The team also ran sensitivity analyses involving a longer analysis period, varying definitions of vaccination status based on timing in relation to illness onset, and exclusion of participants who had chronic illnesses. Estimates of vaccine effectiveness remained above 90% in most of these analyses. The analyses also showed that receipt of 2008-09 or 2009-10 seasonal flu vaccine had little influence on the estimated effectiveness of the pandemic vaccine. (Administration of the 2009-10 seasonal vaccine was delayed in Canada until after pandemic vaccine became available, limiting the ability to assess this effect.)

However, when a longer analysis period was used—weeks 44 to 52—the overall vaccine effectiveness was somewhat lower at 78.4% (95% CI, 52.6-90.2). During that period, 411 of 993 patients (41%) tested positive for H1N1, including eight vaccinated patients. (The number of controls who had been vaccinated was not listed.) The report says this lower estimate was expected, because the rollout of vaccine had only begun in week 43 and more controls with no opportunity to be vaccinated or to mount a protective antibody response were included in week 44.

The authors say their findings are consistent with immunogenicity studies showing a high antibody response to AS03-adjuvanted H1N1 vaccines and with expectations given a good match between vaccine and virus. The findings also match up with a small study involving the same vaccine in New Brunswick, Canada, and a study of an AS03-adjuvanted vaccine used in Germany. But a seven-country study in Europe, involving a mix of adjuvanted and adjuvant-free vaccines, yielded an effectiveness estimate of 71.9% (95% CI, 45.6-85.5), the report notes.

The researchers say that in an observational case-control study there is no foolproof method for eliminating all bias, but they carefully checked participants' profiles against historical and expected demographic information and found the results reassuring. However, they acknowledge testing of patients was at their clinicians' discretion, which could introduce some bias.

Questions raised by other researchers
Other flu vaccine researchers point out that several recent studies similar to the Canadian one came up with considerably lower estimates of H1N1 vaccine effectiveness. Some suggested that the high estimate in the present report may reflect undetected biases or other flaws in the study design.

Nick Andrews of the United Kingdom Health Protection Agency enumerated other recent studies.

  • He and his colleagues, in a study involving testing of ILI patients from hospitals and clinics in England, where an AS03-adjuvanted vaccine was widely used, came up with an effectiveness estimate of 62% (95% CI, 33-78).
  • In a study published recently in Eurosurveillance, P. Hardelid and colleagues, using a network similar to the Canadian one, reported an estimate of 72% (95% CI, 21-90) for H1N1 vaccine effectiveness in England and Scotland.
  • A similar study in Korea, published recently in Vaccine, estimated the effectiveness of an MF-59–adjuvanted vaccine at 73% (95% CI, 49-86).
  • The aforementioned seven-country European study involving a mix of adjuvanted and unadjuvanted vaccines, which put the effectiveness at 72%.

Andrews said that though the Canadian study came up with a higher estimate than these others, from the standpoint of the 95% CI, it is "not inconsistent" with those results. But he suggested that the 78% estimate the Canadian team found when they used the longer analysis period (weeks 44 to 52) might be closer to reality.

The authors regarded this number as less valid because the analysis period began the week after the vaccine was rolled out, he noted, but added, "The authors state that week of illness is adjusted for in the analysis and this should therefore account for any confounding due to changing coverage and incidence over time. The estimate of 78% is very similar to the estimates reported from other studies. Therefore whilst the adjuvanted vaccine clearly is effective, it is not clear the vaccine gives very high protection as suggested in the Canadian paper."

For Lisa Jackson, MD, MPH, senior immunization investigator with the Group Health Research Institute in Seattle, the timing of vaccination in relation to the pandemic peak presents a problem for the study. "The pandemic had waned prior to availability of vaccine, so persons who got the vaccine had very little risk of being exposed following vaccination," she commented by e-mail. "Therefore, they did not get infected not because the vaccine worked but because there was no influenza around to get in that time period."

Jackson said the authors state that they controlled for time. "But if there is essentially no overlap between the timing of cases and the timing of vaccine availability, this cannot be adjusted for in the analysis. So, by definition, cases were not vaccinated because everyone (nearly) was infected before hardly anyone could have received vaccine. Very few controls were vaccinated, but the proportion was greater than zero, so this makes it look like the vaccine is highly effective."

The possibility of undetected bias in the selection of patients for testing in the study is a concern raised by Nick Kelley, MSPH, who has conducted an in-depth analysis of flu vaccine effectiveness studies in his capacity as assistant director of the CIDRAP Comprehensive Influenza Vaccine Initiative, a project funded by the Alfred P. Sloan Foundation. Kelley is a research assistant at the University of Minnesota Center for Infectious Disease Research and Policy, publisher of CIDRAP News.

"There's no information on how many patients who presented for flu-like illness were actually tested, so it's hard to gauge the population they estimated the effectiveness on," Kelley said.

He said that in previous studies by the Canadian authors and a group in Australia that did sentinel flu surveillance, about 40% to 50% of patients who presented with influenza-like illness were tested. "So if you're missing half or more of the flu-like illness, it could introduce quite a bit of bias," he commented.

On the other hand, Kelley allowed that the Canadian study and the other recent reports "show that the pandemic vaccine worked. I think this one is just a little overestimated."

Roger Baxter, MD, co-director of the Kaiser Permanente Vaccine Study Center in Oakland, Calif., suggested another possible source of undetected bias in the study. "In this case, I can imagine that people who are anxious would be more likely to present for their flu shot, and would also be more likely to come in with mild symptoms that are not flu and ask to be tested for flu," he said.

Baxter said a possible way to control for this would be to ask the physicians to grade the severity of the illness and rate the likelihood that the patient actually had flu.

"I'm left wondering if this adjuvanted vaccine is really this good," he said. "Effectiveness studies are difficult, that's for sure."

Responses from first author
Skowronski, first author of the Canadian study, responded by e-mail to several of the concerns raised by other researchers. She is epidemiology chief in the Influenza and Emerging Respiratory Pathogens branch at British Columbia's Centre for Disease Control.

Concerning the timing issue, she observed that the study was an incident case-control design, not a cohort study, with cases and controls drawn from the same time period.

"Although our primary analysis period was selected immediately after the [pandemic] peak in order to balance vaccine access and virus exposure opportunities, virus circulation did not stop suddenly," she said. "It continued after the peak such that among our participants, 38% were still test-positive during the 4-week analysis period (55% in week 45, 40% in week 46, 22% in week 47, and 21% in week 48). Since cases and controls were drawn from the same period, what is most important is that they have comparable exposure opportunities."

Skowronski made several points about the possibility of undetected biases in the selection of patients who were vaccinated or were tested for H1N1:

  • All participants presented to a physician with an acute febrile respiratory illness, and information on vaccine status was collected from the patients before either the physician or the patient knew the test result, thereby reducing the risk of recall bias.
  • Medical care in Canada is free to patients at the point of care, and the vaccine was provided free, reducing the probability that decisions about testing and participation were influenced by factors related to access to care.
  • As noted in the report, the researchers found the profile of participants to match well with what would be expected on the basis of factors known to influence the likelihood of being vaccinated and of testing positive.
  • The large number of sentinel sites across Canada used in the study limits the risk that one or a few sites skewed the results by using a biased method for selecting patients for testing.All participants were community-based, limiting the biases that might be introduced by combining data from hospital patients with data from outpatients.
  • The primary analysis period was chosen to optimize the period of vaccine access and virus exposure opportunities. Since the fall 2009 pandemic was evolving simultaneously with vaccine rollout in most countries of the Northern Hemisphere, this issue is a concern for all observational studies conducted during that period.
  • Directly comparing vaccine effectiveness estimates across countries may not be appropriate, since the methods used, the population studied (hospitalized or clinic-based), and the types and mix of vaccines administered will influence estimates of vaccine protection overall.

Skowronski DM, Janjua N, De Serres G, et al. Effectiveness of AS03 adjuvanted pandemic H1N1 vaccine: case-control evaluation based on sentinel surveillance sytem in Canada, autumn 2009. BMJ 2011;342 (early online publication Feb 3) [Full text]

This week's top reads