Mar 1, 2012 (CIDRAP News) – A study published this week adds another to a short list of recent investigations suggesting that standard influenza vaccines yield only modest reductions in the risk of hospitalization or death in elderly people.
The research team, from the University of Toronto, looked at the relationship between flu vaccination and health outcomes in Ontario seniors over nine flu seasons. Using an unusual method to limit confounding, they found that vaccination was linked with a 6% reduction in all-cause mortality during flu seasons, but this difference was not statistically significant.
When they looked at another outcome—either all-cause mortality or hospitalization for pneumonia and influenza (P&I)—the result was better: a significant 14% reduction in risk during flu season for those who were vaccinated. The study was published Feb 27 in Archives of Internal Medicine.
Though the study showed only modest benefits, the authors say the current recommendation for annual flu vaccination for elderly people should remain until more conclusive evidence is gathered, because flu vaccination is generally safe and relatively low in cost.
Documented benefits of flu vaccination in seniors have been downsized in recent years. As noted in the new report, past observational studies have suggested that flu vaccines reduced winter all-cause mortality in older people by as much as 50%. But researchers have pointed out that flu is believed to explain less than 10% of deaths in winter, making such a large benefit implausible.
In addition, various researchers have marshaled evidence that much of the benefit of flu vaccination in seniors shown in observational studies actually amounts to a "healthy user effect," in that people who get vaccinated tend to be healthier than those who don't, which biases study results in favor of vaccine effectiveness. Part of the case for this came from a study in which major apparent benefits of vaccination were observed even when flu wasn't circulating, as noted in a commentary that accompanied the new study.
Because of the healthy-user effect and related difficulties, several research teams have taken special pains to eliminate or control for unobserved differences between vaccinated and unvaccinated groups of elderly people. The Canadian study appears to be the latest of these.
Borrowing an economics method
The Toronto research team used health administrative data to look at the association between flu vaccination and health outcomes in elderly people throughout Ontario over nine flu seasons, from 2000-01 to 2008-09. The analysis included about 1.4 million people in each season.
Data on vaccination, health outcomes, and patient characteristics were drawn from several provincial health databases. The researchers also looked at health outcomes in the post-flu season, defined as July 1 to Sep 30 after each season.
The investigators used an approach called the instrumental variable (IV) analysis method to help control for biases in their study. The method is described as widely used in economics but not often in epidemiology. They used senior vaccination coverage for each municipality or locality (census subdivision, or CSD) as their instrumental variable.
"The IV behaves like natural randomization of patients to regional vaccination groups that differ in their likelihood of receiving influenza vaccination," their report explains. "Unlike randomization, the difference in the likelihood of treatment is not 100%, and one can explore but not prove that the groups are similar in unmeasured patient characteristics."
The researchers determined the IV—senior vaccination coverage—for each of Ontario's 385 localities for each flu season. Their data included 12.6 million person–influenza seasons of observations, with 130,532 deaths and 62,913 pneumonia and flu hospitalizations during flu seasons. Overall vaccination coverage averaged 58.2%.
The team ran a conventional analysis and an IV analysis and compared the results. The conventional analysis involved logistic regression with adjustment for differences between vaccinated and unvaccinated groups in several variables, including demographics, other illnesses, medication use, and healthcare use.
This method showed that vaccination was linked with a 33% reduction in mortality during flu season (adjusted odds ratio [OR], 0.67; 95% confidence interval [CI], 0.62 to 0.72) and with a 15% reduction after flu season (adjusted OR, 0.85; 95% CI, 0.83 to 0.86). The 33% mortality reduction during flu season is similar to findings in other observational studies and indicates the presence of bias, the authors say.
In contrast, the IV analysis, which included adjustment for the same variables as the conventional analysis, linked vaccination to a nonsignificant 6% reduction in mortality during the flu season (adjusted OR, 0.94; 95% CI, 0.84 to 1.03) and a 13% increase in risk after the flu season (OR, 1.13; 95% CI, 1.07 to 1.19).
For the combined outcome of death or hospitalization for pneumonia or flu, the conventional method showed a 26% reduction in risk during the flu season (adjusted OR, 0.74, 95% CI, 0.70 to 0.78) and a 12% reduction after the season (OR, 0.88; 95% CI, 0.87 to 0.90). The IV analysis pointed to a 14% decrease in risk during the flu season (OR, 0.86; 95% CI, 0.79 to 0.92) and a nonsignificant 2% increase in risk after flu season (OR, 1.02; 95% CI 0.97 to 1.06).
"We conclude that influenza vaccines may not reduce all-cause mortality among elderly individuals during influenza season but may reduce the combined rates of P&I hospitalization and death," the authors write.
They observe that their IV analysis results are similar to the findings in two other recent studies that involved the use of another novel technique, called case-centered logistic regression, to minimize bias in assessing flu vaccination effects in older people.
One of these, published in 2009, showed a significant 4.6% decrease in mortality in a California population over 10 years. The other report, published in 2010, cited a significant 8.5% reduction in pneumonia and flu hospitalization in northern California seniors over an 11-year period.
The Canadian authors say their finding in the IV analysis of no reduction in hospitalization or death after the flu season is in line with what would be expected and therefore reflects well on the validity of the method. On the other hand, they suggest that the finding of a 13% increase in all-cause mortality risk after the flu season may mean that they went too far in their effort to eliminate bias. The finding "was unexpected and raises the possibility that we may have underestimated the true effectiveness of the vaccine for this outcome," they write.
They acknowledge some limitations of their analysis. One is that about 25% of flu vaccine doses given to the elderly were not recorded in the database they used, and this may have biased their findings toward lower effectiveness. A second drawback is that they lacked cause-specific mortality data and couldn't use P&I or respiratory mortality as specific outcomes.
Others hail study, raise questions
In the accompanying commentary, epidemiologists M. Alan Brookhart, PhD, and Leah McGrath, MS, write that the findings are consistent with three recent US studies suggesting that the life-saving benefits of flu vaccine in the elderly are small. Brookhart and McGrath are at the Gillings School of Global Public Health at the University of North Carolina, Chapel Hill.
They generally commend the authors' IV analysis, but they take issue with their use of the post-flu season to test the bias in their vaccine effectiveness estimate, commenting, "This method is problematic because there can still be a baseline level of circulation of the vaccine strain even in July; thus, it is possible that the vaccine may still be providing some real benefit in the post-influenza season." They suggest using the pre-influenza season instead.
Bruce Fireman, MA, a coauthor of the two aforementioned studies of flu vaccine effectiveness in California seniors, offered a mostly positive evaluation of the study but raised a few questions. He is a biostatistician in the research division at Kaiser Permanente in Oakland, Calif.
"Basically I thought that their main points are good, that the method they're trying out is a reasonable method, and I like the paper," he told CIDRAP News.
Fireman said it's difficult to find "a really good, credible" instrumental variable to use in epidemiologic research. An IV is "good if it's related to the variable of interest [in this case, flu vaccination], but has no other way of being related to the outcome except through the variable of interest," he explained.
Noting that the predictor variable the authors used is local vaccine coverage, Fireman said, "So they're saying if you live in an area with a higher coverage rate, how much better off are you with respect to the outcomes? What we have to assume to buy that the calculation is unbiased is that the only way this could matter is if the people who are getting vaccinated are getting some protection from being vaccinated. But one could imagine other ways it could matter. They address some but not others."
The authors present data indicating that the high-coverage areas generally didn't tend to have sicker or healthier people than the low-coverage areas, he said, and went on, "But we're talking about an infectious disease that comes and goes and could be more severe in some areas. . . . Let's say people know their area is vulnerable [to bad flu outbreaks], so it may be that the areas that are especially vulnerable have especially high vaccine coverage rates. So vaccine coverage rate in an area would be related to the outcome in that area. One could imagine reasons why it could be a positive association or a negative association."
Fireman said he didn't have any evidence that the concern he described is a problem in the study, but he said he would have liked to see the authors discuss more fully the potential problems in using the geographic variable as their instrument.
The study drew praise from Nicholas Kelley, PhD, a research associate with the University of Minnesota's Center for Infectious Disease Research and Policy, publisher of CIDRAP News. Kelley is the coauthor of a widely noted meta-analysis of flu vaccine effectiveness studies that was published last year in the Lancet Infectious Diseases.
Kelley said IV analysis is a useful method for controlling biases. "It's very challenging and difficult to control for some of these biases in flu vaccine studies, especially if you don't know what they are. I thought it was very elegant that they showed a standard analysis, adjusting for all these covariates, and got typically what you see in the literature."
He said it was a little surprising that the IV analysis indicated that vaccination was linked to higher all-cause mortality outside the flu season, which suggests that the analysis "went too far one way." "Ideally you'd like that [odds ratio] to be close to one," he said.
The bottom line for Kelley is that the study again shows the need for better flu vaccines. "This is a second really detailed study that shows the vaccine doesn't work very well to prevent mortality and has limited impact on hospitalizations, just highlighting the need for a better vaccine to prevent flu in this population, given that they have 90% of the mortality," he commented.
Brookhart and McGrath commented that research is needed to determine if vaccines other than the standard formulations, such as the high-dose vaccine now available in the United States, would provide greater benefits in older people.
Wong K, Campitelli MA, Stukel TA, et al. Estimating influenza vaccine effectiveness in community-dwelling elderly patients using the instrumental variable analysis method. Arch Intern Med 2011; early online publication Feb 27 [Abstract]
Brookhart MA, McGrath L. The influenza vaccine in elderly persons. (Commentary) Arch Intern Med 2011; early online publication Feb 27 [Excerpt]
Oct 25, 2011, CIDRAP News story "Strict meta-analysis raises questions about flu vaccine efficacy"