Public health officials respond to critique of BioWatch

Disclosure: The University of Minnesota's Center for Infectious Disease Research and Policy, which publishes CIDRAP News, has produced training programs and materials for the BioWatch program. The news team has no involvement in the center's BioWatch work.

Aug 17, 2012 (CIDRAP News) – In the wake of a newspaper investigation that questioned the value of the federal BioWatch program for detecting dangerous airborne pathogens, some public health officials familiar with the program acknowledge that it's far from perfect, but they say it's not time to scrap it.

In July the Los Angeles Times published a lengthy investigative article that essentially suggested that BioWatch, a secretive program that aims to detect bioterrorist attacks by monitoring the air in more than 30 large cities across the country, is more trouble than it's worth.

The story, by David Willman, said the program has triggered 56 disruptive "false alarms"—detections of pathogens that didn't represent a real danger. Moreover, it said the system probably would miss a real attack unless the air concentration of the pathogen were very high. For those reasons, local and state health officials just don't trust the program, the story suggested.

Interviews with a few officials who have worked in the program or are knowledgeable about it indicate that there's a good deal of truth in the story's account of the program's shortcomings. But they argue for improving the program, not cancelling it.

"From a local perspective, the information and issues raised in the article are not new to us. These are things we've heard consistently over the years from local health departments," said Jack Herrman, senior advisor and chief of public health preparedness at the National Association of County and City Health Officials (NACCHO), in an interview.

"However, I think it's important that we don't kind of bash the system in the absence of any other technology that's better," he said. "More important, I think we should focus on what we need to develop in human and technological resources that can provide us with timely and accurate data that can detect the presence of potentially life-threatening pathogens."

Similarly, Paul Biedrzycki, MPH, MBA, director of disease control and environmental health for the Milwaukee Health Department, said BioWatch is a "pioneering" system that has drawbacks. "I can understand why [some public health officials] don't think it's useful because of the sensitivity and specificity limitations," he said. "But it does add to the common operating picture. It is the only biosurveillance program I'm aware of that has some dimension of standardization in terms of collection and analysis of data in the country."

The BioWatch program, managed by the Department of Homeland Security (DHS) in cooperation with local health departments, operates air-monitoring stations with filters that are collected daily and tested at state and local health labs to see if they have trapped certain pathogens, such as Bacillus anthracis (anthrax) spores. It takes from 10 to 36 hours to detect a pathogen with the current equipment, according to DHS congressional testimony that was cited in a recent article in Advances in Preventive Medicine.

DHS is currently testing "Generation 3" air-monitoring equipment that would automate the sampling and testing, with the aim of reducing detection time to 4 hours, according to the article.

A long list of concerns
Here are the major points made in the Times investigative article:

  • BioWatch had 56 "false alarms" through 2008, and more followed. These occurred in a number of cities and threatened to disrupt the 2008 Democratic National Convention, the 2004 and 2008 Super Bowls, and the 2006 National League baseball playoffs. Many of these were caused by the tularemia bacterium (Francisella tularensis).
  • BioWatch could easily miss a bioterrorist attack, because perpetrators could disperse pathogens at concentrations too low to be detected by the equipment yet high enough to sicken thousands of people, according to scientists who spoke on condition of anonymity. In particular, testing of prototype Generation 3 units showed they could detect agents only if exposed to "extremely high concentrations" of pathogens.
  • Because of the "false alarms" and other drawbacks, state and local public health officials have shown no confidence in the program, as evidenced by the fact that they have never ordered evacuations or distributed emergency medicines in response to a pathogen detection. One state official called the program "a colossal waste of money."
  • Before BioWatch was established in 2003, the Pentagon had assembled a panel of prominent scientists to gather baseline data on the prevalence of certain pathogens in nature, with the goal of preventing false alarms. But in the rush to launch BioWatch, the group was disbanded. "The failure to conduct that work has hobbled the system ever since, particularly in regard to tularemia, which has been involved in nearly all of BioWatch's false alarms," the story said.

Some of these points more or less echo the conclusions of a report issued by the Institute of Medicine (IOM) and the National Research Council (NRC) in 2010. Regarding the chance of detecting a bioterrorist attack, the report said, "BioWatch has the potential to provide a more timely alert than the public health and health care systems under certain specific conditions: if a large-scale aerosol attack occurs where BioWatch is deployed, if an air sampler lies in the path of the release, and if the pathogen used is one of those included in the BioWatch laboratory assays."

The IOM-NRC report also said the current BioWatch system needs better testing to establish its effectiveness and better collaboration with public health systems. Also, commenting on the Generation 3 equipment, the report said, "The proposed enhancements of the BioWatch system will be possible only if significant scientific and technical hurdles are overcome."

DHS rejects 'false alarms' allegation
DHS officials, for their part, reject the notion that BioWatch has caused false alarms. They explain that many of the agents the system is designed to detect occur in nature, so when one of them is found, experts must decide if it represents an emergency or not.

In a recent blog post, Alexander Garza, MD, MPH, chief medical officer for DHS, said BioWatch has detected naturally occurring pathogens 37 times in 7 million tests. "The detection of commonly occurring environmental agents is not a 'false-positive,'" he wrote.

Garza likened the BioWatch system to a home smoke detector, which goes off for both burnt toast and a major fire. "These tools alone cannot and do not declare that a biological attack has occurred," he said. "Experts must interpret the data and quickly make tough, logical decisions about the reality of the threat."

In a reply to questions from CIDRAP News, DHS officials said the detections of naturally occurring organisms showed that the system was working and capable of detecting a bioterrorist attack. They said the detections also served to test and strengthen the program's decision-making and response procedures.

DHS officials also asserted that, contrary to the Times' claims, state and local officials have shown confidence in BioWatch. They said they have received positive comments about the way the program brings together public health, first responders, and other government officials and have had good feedback about the BioWatch exercise program in particular.

Probability of detection
Biedrzycki acknowledged that BioWatch could fail to detect a bioterrorist attack, given the system's limited geographic coverage and the large number of variables involved, such as the type of agent, the agent's longevity, weather, and monitoring locations.

"Some events may be under the radar for those reasons, the number of variables and the complexity of interactions among them," he said. "On the other hand, there are some things that could be picked up, if there were sufficient concentration. I don't think the system by any account is useless."

Biedrzycki agreed with DHS officials that the pathogen detections—known in DHS-speak as "BioWatch actionable results," or BARs—have been instructive. He said he has been involved in at least one BAR, but couldn't give any details about it.

"All of the protocols have been enhanced as a result of what the article referred to as false alarms," he said. "They have stimulated discussion, identified gaps, and improved protocols that I think will be extremely helpful."

He added that the BARs have contributed to the scientific data on what organisms are found in the environment, commenting, "That may not have been the primary intent of the system, but it's certainly helping us understand what's there."

Early missteps caused dissatisfaction
In Biedrzycki's view, some of the dissatisfaction expressed by state and local public health officials in the Times story stems from the early days of BioWatch, when, as he sees it, DHS officials failed to consult and collaborate adequately with state and local officials.

"I totally understand the frustration of some of my colleagues, who are still smarting from the fact that the system was rolled out without adequate consultation of local public health, and with the understanding that public health would be responsible for consequence management," he said.

"Some of the remarks [quoted in the Times] may reflect some of that lack of trust building and collaboration," Biedrzycki said. "That has changed significantly because public health is now invited through our professional organizations to the table far more routinely in the last 5 years than in the previous 5 years."

Some public health officials' discomfort with BioWatch also may be related to a culture clash between the public health world and the law enforcement and security realm, according to Biedrzycki.

"Public health typically hasn't been part of that culture, of law enforcement or national security and the intelligence community," he said. "This is new territory, and I think we don't fully understand how to operate within that culture.

"It's very difficult for us, coming from a very transparent, open, trust-building relationship with many of our clients, going into a less open environment in terms of information sharing. I can understand those criticisms, but in reality I think the trend is for public health to be integrated with the intelligence community."

A Houston perspective
In Houston, which has had several BARs of its own, David Persse, MD, the city's public health authority, said his experience with BioWatch has been "mostly positive," though he has been aware of criticism of the program from public health officials elsewhere.

"I have sensed dissatisfaction from other metro areas across the country, but here in Houston we're just not upset about it," Persse said.

"It's an early warning system and I think people need to keep in mind what it is," he said. "I think the critics are frustrated that when we get some information, that it isn't all the information that we'd really like to have. . . . I know it's not all the information we'd like to have, but the alternative is to not find out till tomorrow, when it's really too late. It's part of the solution, not the entire solution."

Persse said he got involved in BioWatch in 2006 and so missed the early stages of the federal-state-local collaboration on it. In the early days, detections sparked overreactions by program officials, but by the time he got involved, responses were more measured, he said.

He said the cooperation between federal, state, and local levels now is good. "When we've had issues we've had to take to higher levels, people at the federal level have been very responsive."

Persse said he didn't see anything factually inaccurate in the Times article, but he disagreed with many of the opinions expressed by those who were interviewed.

He agreed with the assertion that a real bioterrrorist attack might be missed by BioWatch. "Am I absolutely sure that it would detect it? No," he said. But he added that if someone dispersed B anthracis in the air and the system did catch it, "we'd probably know 3 days before people started showing up in the ER, so it would give us some time to mobilize our resources and start our response."

At the same time, Persse voiced concern about the Generation 3 monitoring equipment (for which DHS is considering whether to award a long-term contract). "With what we have now (Generation 2 equipment), you get a number and you can try and put it in context with what's going on in the community. In Houston we've had a number of BARs to interpret that," he said.

But Generation 3 equipment doesn't provide a number, only a positive or negative signal, he explained, adding, "We're going to get more continuous monitoring and results, which is a plus, but the downside is I won't be able to interpret whether it's an attack or an environmental contaminant."

Persse's bottom line on BioWatch is that he's willing to live with its limitations.

"Everything in life is a tradeoff," he said. "The tradeoff with BioWatch is you get some information early, but you may not get everything you want. . . . We haven't invented exactly the right kind of screwdriver for this, so we have to use the screwdrivers we have."

An independent review suggested
Biedrzycki and NACCHO's Herrman take seriously the doubts about the Generation 3 automated detectors described in the Times article. They both suggested that the technology should be reviewed by independent experts.

If the effectiveness of the technology really is in question, said Herrman, "We should have an independent review of the system by qualified professionals to ensure that the funding that's being devoted to it is actually going to produce a reliable technology."

Biedryzcki said he wasn't familiar with the details of the Generation 3 testing, but he said, "I think an independent review or evaluation of the system would maybe shed more light on what the cost-effectiveness and the gaps of the system are."

In a similar vein, the 2010 IOM-NRC report recommended that DHS and the Department of Health and Human Services (HHS) set up an independent panel of experts to provide advice on all BioWatch technical and operational issues. The report said the panel should include both technical experts and state and local health officials experienced in working with BioWatch.

In other observations, Herrman also echoed the point made in the Times article about the need for studies to measure the background levels of naturally occurring pathogens

"We need baseline studies of the presence of natural pathogens, and technology that takes those baseline things into account and [helps] determine when they reach a level where there's a public health implication," he commented.

DHS cites review processes
DHS officials, commenting on the suggestions about external reviews of BioWatch, said the program already has several review mechanisms in place.

Those include an advisory committee composed of local and state representatives who use quarterly conference calls to review issues and practices in the light of pathogen detections. Also, DHS holds an annual workshop where officials from all the BioWatch jurisdictions meet to discuss needs and best practices, officials said.

They said lessons learned from BARs are also reviewed by the Federal BioWatch Working Group, which consists of officials from many partner agencies, including the Centers for Disease Control and Prevention, Department of Defense, FBI, Secret Service, HHS, and Environmental Protection Agency.

See also:

Jul 7 Times report on BioWatch

IOM page for the 2010 report BioWatch and Public Health Surveillance
 

Short summary of the 2010 IOM-NRC report
 

Alexander Garza's blog response to Times article

Jun 18 CIDRAP News item about the cost of upgrading BioWatch equipment
 

Jul 20 Times report on congressmen pressing DHS for information on BioWatch

This week's top reads