Viewpoint: When trust is gone, show your work

Trust wooden puzzle pieces in man's hands

Andrii Yalanskyi / iStock

The nation’s highest-ranking health official is expanding an effort that he claims is reexamining the safety of vaccines. Americans should insist on seeing exactly how that work is done.

Health and Human Services (HHS) Secretary Robert F. Kennedy Jr. has described an expansive research agenda spanning the National Institutes of Health (NIH), Centers for Disease Control and Prevention (CDC), Food and Drug Administration (FDA), and other federal agencies. He says researchers are examining vaccine timing and sequencing, aluminum adjuvants, vaccination during pregnancy, autoimmune disease, neurodevelopmental disorders and autism. Federal health data, including Medicaid records, the Vaccine Safety Datalink and FDA surveillance systems, are being marshaled for these efforts, according to the secretary.

It cannot be overstated just how poorly positioned Kennedy is to mount a credible effort to examine or improve vaccine safety. He spent years as one of the nation’s most prominent anti-vaccine advocates, and he has a long history of making inaccurate or misleading claims about vaccine testing, the preservative thimerosal, the MMR vaccine, vaccine adverse-event reporting and autism.

In a September speech to Children's Health Defense, the organization he once led, Kennedy again suggested that vaccines could be responsible for a range of chronic diseases and said federal researchers were examining vaccines and autism, even though decades of research involving millions of children have found no credible evidence that vaccines cause autism.

This week, Kennedy announced additional initiatives: an NIH clinic to study and care for people who believe they have experienced vaccine injuries, along with a proposal to reimburse clinicians for reporting suspected injuries.

Credibility must be earned

Rigorously studying vaccine safety is vital. We should constantly evaluate the benefits and risks of vaccines, investigate potential safety signals and improve surveillance systems.

But scientific findings no longer deserve credibility simply because they come from the federal government. Credibility must be earned through rigorous methods, transparency, and independent scrutiny. 

To date, the secretary and his agency have failed that test.

The question is not whether federal health agencies should study vaccine safety. They should.

The question is whether these studies will be designed to discover what the evidence shows—or to produce evidence supporting conclusions that have already been reached.

Scientific findings no longer deserve credibility simply because they come from the federal government.

What, then, should these efforts look like? 

Before analyses begin, there should be transparent, competitive processes to award contracts for the work. Successful researchers should publicly post research questions, hypotheses, protocols, study designs, outcomes, and statistical analysis plans. They should disclose how populations are selected and how vaccinated and unvaccinated groups are defined. If human subjects are involved, there should be informed consent for participation, and an Institutional Review Board should review and approve plans. 

Researchers should explain how they will account for confounding factors and multiple comparisons. Changes to protocols should be documented. Data and analytic code should be made available to independent researchers to the fullest extent privacy laws allow. And HHS should commit in advance to posting results regardless of whether they support the hypotheses being tested.

In short: show the work.

Sound findings should withstand careful scrutiny

That principle guides the Vaccine Integrity Project. For our systematic evidence reviews, protocols are prospectively registered in PROSPERO, an international registry of systematic reviews. We describe how evidence will be identified, selected and evaluated before we know what that evidence will show. Multiple researchers independently review studies and assess their quality. We make our methods and evidence sources publicly available so others can scrutinize our conclusions, and the results are published in peer-reviewed medical journals.

This was how the federal government previously conducted its vaccine-related research, including during President Donald Trump's first administration, when, for example, CDC strengthened the process used to develop federal vaccine recommendations by adopting a formal Evidence to Recommendation framework. The agency publicly described the questions being considered, the outcomes used to evaluate benefits and harms, the evidence reviewed and the judgments underlying recommendations. When ACIP evaluated a hepatitis B vaccine with a novel adjuvant in 2018, for example, it publicly identified serious adverse events and cardiovascular safety as critical outcomes and published the methodology and evidence used in its review.

HHS under Kennedy has fallen far short of that standard.

Of course, transparency does not guarantee that researchers will always reach the right answer. It does make it possible for other scientists to determine whether the methods were sound and the conclusions supported by the evidence.

Of course, transparency does not guarantee that researchers will always reach the right answer.

The stakes could hardly be clearer.

This week Pennsylvania announced that its measles outbreak had reached 1,078 cases—the first US outbreak to exceed 1,000 cases in 35 years. Nearly 200 people have been hospitalized, and five have died. The last time the United States experienced measles outbreaks on this scale was 1991, when Philadelphia suffered an outbreak of roughly 1,400 cases and New York City experienced more than 2,000 cases that year.

Those numbers are a reminder that vaccine policy is not an abstract ideological debate. Decisions about vaccines have consequences for children, families, and communities.

At a moment when confidence in federal health agencies continues to decline, they cannot simply ask Americans to trust them. And given Kennedy's long record of making inaccurate claims about vaccines, no one—journalists, clinicians, scientists, and the public—should accept findings emerging from these new initiatives at face value.

That’s not the same thing as dismissing the findings simply because of their source, but they must demand the evidence.

If these studies are scientifically rigorous, their methods should withstand independent scrutiny. If the conclusions are supported by the data, other qualified researchers should be able to examine the work and reach the same conclusions.

Science earns credibility not by asking for trust, but by making verification possible.

For every vaccine study now being undertaken by HHS, that must be the standard.

This week's top reads