Controversy over data in hydroxychloroquine COVID-19 study grows

White pills in shape of question mark
White pills in shape of question mark

Tatiana / iStock

The editors of The Lancet yesterday issued a statement acknowledging the criticism of a recent study that concluded that the antimalaria drugs hydroxychloroquine and chloroquine did not benefit COVID-19 patients and were associated with higher risk of death and serious heart rhythm complications.

The statement comes in response to a letter signed by more than 100 scientists and clinicians worldwide that raised questions about the data behind the large observational study.

The Lancet said in its statement that important scientific questions had been raised about the data reported in the paper, and that an independent audit of the data commissioned by the authors is ongoing.

The study, published in The Lancet on May 22, compared outcomes in patients treated with hydroxychloroquine and chloroquine (with or without macrolide antibiotics) with those who received neither drug, and was the largest to date on the use of hydroxychloroquine and chloroquine in COVID-19 patients. But in the days following publication, several scientists took to social media to express skepticism about the study and the data on which it was based, and that criticism has continued.

In an open letter to Lancet Editor Richard Horton late last week, critics raised concerns about the methodology of the study, asked that a group convened by the World Health Organization (WHO) perform an independent validation of the analysis, and requested that the journal make peer review comments on the study openly available.

Concerns about bias and confounding

Among the concerns laid out in the letter are that there was inadequate adjustment for known and measured confounders, such as severity of illness.

That's been a common criticism of several of the observational studies on use of hydroxychloroquine and chloroquine for COVID-19: that many of the patients treated with the drugs in these studies have been sicker. Experts say that the inability to fully account for more severe disease among patients treated with the drug skews the results.

"Because it is an observational study, we know that bias and confounding can really affect the results," said Ruanne Barnabas, MBChB, DPhil, a physician and associate professor of global health at the University of Washington who signed the letter. "My concern about it being observational was that it [hydroxychloroquine] was used compassionately for people who were more sick and likely to do worse anyway…so we would not be able to assess the impact using the study design."

That criticism is one of the main reasons why there is a consensus among experts that randomized controlled trials (RCTs)—the gold standard for evaluating whether a drug is truly safe and effective against a disease—are needed to determine whether the drugs can help COVID-19 patients.

The letter also argued that the authors of the study have not released their code or data, that no ethics review was performed, and that some of the mean daily doses of hydroxychloroquine given to patients in the study are higher than US Food and Drug Administration recommendations. And it raised specific questions about some of the data points.

In particular, the critics note that data from Australia are not compatible with country reports on COVID-19 cases, with too many cases for just five hospitals and more in-hospital deaths than had occurred in the entire country during the study period. In addition, data from Africa indicated that 25% of COVID-19 cases and 40% of deaths on the continent occurred in Surgisphere-associated hospitals, a claim that the letter said seems unlikely.

In a correction published in The Lancet on May 30, the authors of the study provided revised numbers on participants from Asia and Australia, and said that a hospital self-designated as belonging to the Australasia continental designation should have been assigned to the Asian continental designation. But there were no changes to the findings of the paper.

Data origins, validity questioned

In addition, the critics raised red flags about the origins of the data used in the study. The analysis of 96,032 patients from 671 hospitals on six continents used data from the Surgical Outcomes Collaborative, a database that collects de-identified patient information from electronic health records, supply chain databases, and financial records. The database is owned by Surgisphere, a company founded by study co-author Sapan Desai, MD, PhD.

"There was no mention of the countries or hospitals that contributed to the data source and no acknowledgments of their contributions," the letter states. "A request to the authors for information on the contributing centers was denied."

"You can understand that individual patients don't want to share their data, but at a minimum, knowing which hospitals contributed would be important," Barnabas said, noting that in most observational studies, you know the hospitals where the data came from, and investigators from those hospitals are often co-authors.

"Although an independent audit of the provenance and validity of the data has been commissioned by the authors not affiliated with Surgisphere and is ongoing, with results expected very shortly, we are issuing an Expression of Concern to alert readers to the fact that serious scientific questions have been brought to our attention," The Lancet said.

An emailed statement sent to CIDRAP News on behalf of lead study author Mandeep Mehra, MD, medical director of the Brigham and Women's Hospital Heart and Vascular Center, said the goal of the independent third-party auditor is to verify the source data and assess the accuracy of the database and the authors' findings.

"Upon completion of the reviews, the auditor will simultaneously provide its findings directly to the editors of the journal and to the co-authors, independent of Surgisphere," the statement said. "I eagerly await word from the independent audits, the results of which will inform any further action."

In a statement on its website, Surgisphere said the company's data use agreements with hospitals prevent it from sharing customer names.

"Our strong privacy standards are a major reason that hospitals trust Surgisphere and we have been able to collect data from over 1,200 institutions across 46 countries," the company said. "While our data use agreements with these institutions prevents us from sharing patient level data or customer names, we are able to complete appropriate analyses and share aggregate findings to the wider scientific community."

The company also said it stands behind the integrity of its studies, scientific researchers, clinical partners, and data analysts.

Surgisphere data were also used in another COVID-19 study, published in the New England Journal of Medicine on May 1, that has come into question. NEJM Editor-in-Chief Eric Rubin, MD, PhD, yesterday issued an Expression of Concern about that study, saying the journal has asked the authors to provide evidence that the data are reliable.     

Study results, media coverage affecting clinical trials

Another issue raised in the letter from the 100-plus scientists and clinicians is that the findings of The Lancet study, and subsequent media attention, have caused concern among those currently participating in RCTs. Barnabas, who is the principal investigator in an RCT looking at whether hydroxychloroquine can prevent illness in those who've been exposed to people with a confirmed or pending COVID-19 diagnosis, said the findings have affected recruitment.

"Certainly our trial has seen a decrease in recruitment and enrollment with every news story that comes out," Barnabas said. "When you talk to participants and you present all the information, they understand the impact of observational studies and they are willing to participate, but then a family member or a friend will encourage them not to participate, and we have had people withdraw."

Barnabas said an independent data safety and monitoring board has reviewed and analyzed all the unblinded safety data from the trial, along with data from other ongoing RCTs studying the use of hydroxychloroquine in COVID-19 patients, and recommended continuing the study. "They had no safety concerns," she said.

The WHO, meanwhile, after announcing last week that it was pausing recruitment to the hydroxychloroquine arm of its SOLIDARITY trial, said today that its data safety monitoring board had not found a safety signal, and that the study will resume.

Barnabas said data from RCTs are "absolutely essential" for determining whether hydroxychloroquine can help prevent or treat COVID-19.

"We definitely need those trials to go forward," she said. "And we don't need just one trial, we need several trials, in different populations, asking the question in slightly different ways, so that we can understand whether there is a role here for hydroxychloroquine."

This week's top reads