to BioTechniques free email alert service to receive content updates.
Initiative Tackles Scientific Study Validation

Melissa Lee Phillips

A new initiative seeks to help researchers establish reproducibility prior to publication of their experiments.

Scientists looking to verify their research results now have an option besides redoing everything themselves. The start-up Science Exchange, along with open-access publisher Public Library of Science (PLoS) and open data repository Figshare, have launched the Reproducibility Initiative, a program to help scientific researchers validate their findings.

Science Exchange co-founders Ryan Abbott, Elizabeth Iorns (CEO), and Dan Knox. Source: Science Exchange

The impetus behind the program came from Science Exchange founders, Elizabeth Iorns, Dan Knox, and Ryan Abbott. Established in August 2011, the company allows academic researchers to search for services and expertise not available at their own institution, but after its launch, Iorns and colleagues noticed that it was being used for something unexpected.

"We found that industry, in particular, was starting to use our platform to do independent replications of interesting results that they were interested in licensing from academic labs," she said. "Because they were so concerned about this issue of lack of reproducibility, they were looking for a way to get the results independently reproduced in a cost-effective way."

To respond to this need—not only for industry groups but for academic labs as well as funding agencies—Science Exchange brought on an advisory board of scientists interested in the issue of reproducibility. When a research group submits their study to the Reproducibility Initiative, the board will match the study with someone from the existing network of experimental service providers. This provider, which remains anonymous to the original researchers, receives a fee to replicate the original experiments as closely as possible and submit their results to the initiative.

If their studies are validated, the researchers receive a "certificate of reproducibility" and the option to upload their primary data to Figshare. They also can publish their validation in a special upcoming issue of the journal PLoS ONE devoted to reproducibility. If the studies are not validated, the researchers still have the option of publishing in PLoS ONE but are not required to do so. "We don't want to put punitive pressure on people who want to take part," said Iorns.

The cost of validation—the outside facility's fees plus a 5% cut paid to Science Exchange—will likely be about 10% of the price of the original experiments, estimated Iorns, and will be covered by the researchers' own grants. But that’s something she hopes to change. "We're working really hard with funding agencies to get them to fund not only original exploratory studies but also to fund validations," she said. "But we haven't gotten that pinned down yet."

In light of several recent studies showing that attempts to reproduce laboratory studies can be astonishingly unsuccessful, "it really makes sense for the funding agencies to pay," said Iorns. "If so much of the research that they're funding is not reproducible, that's a huge waste of money."

As for individual researchers, Iorns hopes that they'll be motivated by potential rewards of an additional publication and the certificate verifying reproducibility, which "will be looked on favorably by industry partners and by future funders."

Keywords:  reproducibility