A video game designed to compare open and closed peer-review systems shows that transparency between authors and reviewers increases the accuracy of the process, according to researchers from John Hopkins Bloomberg School of Public Health (JHBSPH).
To examine the differences between open and closed review systems, researchers divided almost 60 participants from laboratories at the university into two groups, authors and reviewers. Then the authors answered GRE questions in an online game, while reviewers either accepted or rejected the answers submitted by their peers. Some of the authors and reviewers engaged in open peer-review system in which the authors knew who reviewed their problem and the reviewers knew who submitted the answers; others operated in a closed peer-review system in which the author did not know the reviewers (1).
“I support the idea of open peer review, and I like the idea of the peer review process being more cooperative. It’s currently being engendered by our anonymous peer review system,” said lead researcher Jeffrey Leek, assistant professor of biostatistics at JHBSPH.
A recent rejection that offended Leek led him to analyze and better understand the process. So he recruited two JHBSPH colleagues: postdoctoral fellow Margaret Taub who develops methods for statistical genetics and theoretical nuclear physicist Fernando Pineda who develops algorithms, tools, and models to characterize biological systems.
When Leek approached Pineda, he immediately recognized that the easiest and fastest way to develop it would be in the form of an online application. In about ten weeks, the team designed the application and developed specifications for the study, using the Amazon Elastic Compute Cloud (EC2) to host the results of their experiment.
“Perhaps, we could capture the essence of the problem,” said Pineda. “It’s a caricature of what’s happening in the real world, but it captures some of the sociology that forms a community.”
Finally, they were ready to start playing games. To begin, the participants were seated in an open room and given a link to log into the game. For 40 minutes, the participants were asked to answer questions and review their peers solutions. At first, some groups were nervous answering questions, but the group dynamic eventually became more interactive, especially in the open peer-review groups.
In the end, Leek and colleagues hope to perform follow-up studies on peer-review models, if they can find the necessary funding. “If we could capture the attention of any funding agency or journal, we would consider doing more sophisticated studies,” said Pineda.
- J.T. Leek, Taub M.A., and Pineda F.J., 2011. Cooperation between referees and authors increases peer review accuracy. PLoS ONE 6: e26895. doi:10.1371/journal.pone.0026895
- K.H. Desai, Tan C.S. Leek J.T., Maier R.V., Tompkins R.G., et al. 2011. Dissecting inflammatory complications in critically injured patients by within-patient gene expression changes: A longitudinal clinical genomics study. PLoS Med 8: e1001093. doi:10.1371/journal.pmed.1001093