Is diversity the key to reproducibility?

Promising laboratory results often don’t make it to the clinic. A new study suggests diversity could be the answer.

Preclinical research is traditionally performed in a single-laboratory under highly standardized conditions, as this should ensure confidence in results. However, a team of researchers from the Universities of Bern (Switzerland) and Edinburgh (UK) has suggested that the opposite could be true – diversity could be important to ensuring reproducibility.

“This entails the risk that the study results may only be valid under the specific conditions of the test laboratory, which may explain the poor reproducibility of preclinical animal research,” the authors noted.

Using computer simulations based on 440 preclinical studies covering animal models of stroke, heart attack and breast cancer, the group compared reproducibility between single- and multi-laboratory studies. Studies with higher diversity – such as those from multiple laboratories, albeit with as few as two locations – were more consistent, and thus more reproducible.

They stressed that the results did not necessarily result in a need for multiple participating laboratories – the greatest gain in improvement was the jump from one to two laboratories.

Furthermore, they demonstrated that simply increasing the sample size in a single laboratory made the situation worse. Taking steps to reduce bias was also not sufficient to guarantee reproducibility.

“Standardization is often promoted also for ethical reasons because standardization reduces variation in experimental results, and therefore fewer animals are needed per experiment to achieve a desired level of statistical power,” the authors explained.

“However, our findings show that reducing animals per experiment through standardization may be short sighted because it means trading animals against the external validity and reproducibility of experimental results. Poor external validity and poor reproducibility question the benefit of the research in the harm–benefit analysis of animal experiments, which could mean that although fewer animals may be used in a standardized experiment, they may be wasted for inconclusive research.”