Months before The Wall Street Journal accused the blood-testing startup Theranos of being a massive fraud, a researcher at Stanford University published a paper pointing out that despite the hype around the company, it had not published any peer-reviewed research, dubbing its efforts “stealth research.” But a new study by the same researcher and colleagues shows that far from being isolated to Theranos, stealth research remains a widespread problem among highly valued startups.
The study, which has been accepted for publication in the European Journal of Clinical Investigation and was led by Stanford professor of medicine Dr. John Ioannidis, found that of 425 papers published on PubMed by unicorns, only 34 – including two reviews – were highly cited. Of 413 papers published by exited unicorns, only 47 – including nine reviews – were highly cited.
The study included 18 current unicorns involved in digital health, artificial intelligence, personal genomics or messenger RNA therapeutics. The 29 that had exited were mostly in biotechnology and pharmaceuticals.
In a phone interview, Ioannidis said there are legitimate reasons for peer-reviewed scientific literature often not keeping up with unicorns’ valuations. “Probably, peer-reviewed publication is not very high on their priority list, and one could understand they have other priorities,” he said. “But at the same time, without having that scrutiny through the wider scientific community, I think what they’re trying to create is based on thin air sometimes. And investors would not be able to tell whether what is being proposed and what is being touted is real and has real potential or is likely to fail.”
In addition to the dearth of peer-reviewed or highly cited research, Ioannidis and the other researchers – from New York University and Babes-Bolyai University in Romania – found that only 18 of the unicorn companies had founders who were highly cited scientists. And in 10, there were no highly cited scientists among the leadership or boards, while only four had information about scientific advisory boards available.
Unlike in the tech sector, where big promises about “disruption” are common, peer-reviewed research is essential in healthcare in order to maintain a threshold of transparency, accountability and credibility in the scientific community, the researchers wrote.
Indeed, earlier this week, Israeli biotech startup Accelerated Evolution Biotechnologies, or AEBi, encountered scrutiny when it became the subject of a glowing article in The Jerusalem Post in which its founders claimed their drug candidate – which had only been tested on mice and could not be published due to insufficient funding – would be a “cure for cancer.”
Investors cannot necessarily be relied upon to act as a check on companies, either. “Even for biopharma or device-oriented startups, investors may have MD or PhD degrees, but this does not necessarily secure that they would be perfectly capable of understanding the science behind what is being proposed,” Ioannidis said. Biomedical science is making tremendous progress, but is overspecialized, he said, such that even as editor-in-chief of a journal he usually depends on people specialized in particular fields to review most papers that come in.
Thus, peer review enables the whole scientific community to appraise research, and may have been able to catch Theranos before it became a darling of Silicon Valley. “If they had been open to peer review in the scientific community, these problems would have been clear early on,” Ioannidis said.
Photo: Justin Sullivan, Getty Images