The reproducibility and replicability of research is a cornerstone of the scientific method (National Academies 2019). In order for research results—at least of a quantitative nature—to be valid, it must be sufficiently well-described so that it is replicable by an independent team of researchers (Jasny et al. 2011). Accordingly, replication studies—a re-evaluation, re-confirmation, or extension of an original study (Quadri and Rosen 2019)—are an important scientific activity in the sciences at large. It is the failure of many seminal scientific results to be reproduced that has led to the so-called “replication crisis” in fields such as psychology and medicine (Ioannidis 2005), opening the door for a “credibility revolution” (Vazire 2018). While those fields were the initial flashpoints for this crisis, the understanding that the underlying methodological issues affect a broad swath of the natural and social sciences is increasingly widespread (Baker 2016).
In other words, there are clear benefits for data visualization to take heed of these developments in other fields and embark on a robust replication effort of older results. In fact, Haroz and Kosara (2018) suggest that we could succeed in “skipping the replication crisis in visualization” altogether by addressing common threats to validity in visualization papers; their paper outlines several solutions.
Despite these benefits, replication is still relatively rare in the human-computer interaction field in general (Greenberg and Buxton 2008), and data visualization in particular (Quadri and Rosen 2019). A study by Hornbæk et al. (2014) showed that a mere 3% of a total of 891 papers from four HCI publication venues were reinvestigations to confirm, expand, or generalize older results, and many of these were not originally intended as replication studies. Similarly, while Quadri and Rosen (2019) note that authors are increasingly including experimental data with their papers to encourage replication and point to several existing replication efforts, such as the RepliCHI and EuroRVVV3 (EuroVis Workshop on Reproducibility, Verification, and Validation in Visualization) workshops, they echo the sentiment that replication studies are still too rare in the visualization and HCI fields.
Part of the reason may be that our field is very focused on novelty. Another may be that some HCI results are qualitative in nature and not intended to be replicable. Many years ago, Greenberg and Buxton (2008) noted that for the HCI field “the problem is that replications are not highly valued”, that replication papers “are difficult to publish”, and “are rarely considered a strong result.” Although this attitude has historically been equally true for the visualization field, times are changing. Recently, visualization researchers have highlighted many successful strategies for publishing replication studies: for example, rather than pursuing strict direct replication, it has been fruitful to frame partial or conceptual replications as a platform that can be built upon to yield novel contributions through expanding the scope or findings of the evaluation, or to specialize the contribution to apply the results to a specific domain (Quadri and Rosen 2019).
VIS 2024 OPCs feel that publishing replication studies of many kinds will be valuable for the visualization field. We encourage the VIS community to make a long-term and deliberate effort to both perform replication activities and to value them as valid contributions to the field. For this year, with the approval of the VIS Steering Committee (VSC), we will do this by both publishing this blog post as well as giving explicit instructions to reviewers on how to review replication studies. Next year, we also plan to discuss the value of replication studies in the call for papers.
Of course, changing the collective minds of the entire reviewer pool can take time and there is no doubt risk involved in being a guinea pig for methodological innovation. Nevertheless, we hope to see some brave authors submitting replication studies to VIS this year, and we wish those authors the best of luck!
References
- Monya Baker. (2016) 1,500 scientists lift the lid on reproducibility. Nature (News Feature), 533(7604):452–454
- Saul Greenberg, Bill Buxton. (2008) Usability Evaluation Considered Harmful (Some of the Time). In Proc. ACM Conference on Human Factors in Computing Systems (CHI) 2008, pp. 111–120
- Kasper Hornbæk, Søren S. Sander, Javier Bargas-Avila, Jakob Grue Simonsen. (2014) Is Once Enough? On the Extent and Content of Replications in Human-Computer Interaction. In Proc. ACM Conference on Human Factors in Computing Systems (CHI) 2014, pp. 3523–3532
- John P. A. Ioannidis. (2005) Why most published research findings are false. PLOS Medicine. 2(8):e124.
- Barbara R. Jasny, Gilbert Chin, Lisa Chong, Sacha Vignieri. (2011) Again, and Again, and Again…. Science 334(6060):1225–1225
- Robert Kosara, Steve Haroz. (2018) Skipping the Replication Crisis in Visualization: Threats to Study Validity and How to Address Them: Position Paper . In Proc. IEEE VIS Workshop on Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) 2018, pp. 102–107
- National Academies of Sciences, Engineering, and Medicine. (2019) Reproducibility and Replicability in Science. Washington, D.C., USA.
- Ghulam Jilani Quadri, Paul Rosen. (2019) You Can’t Publish Replication Studies (and How to Anyways). In Proc. IEEE VIS Workshop on Vis X Vision, 2019. arXiv:1908.08893
- Simine Vazire. (2018). Implications of the Credibility Revolution for Productivity, Creativity, and Progress . Perspectives on Psychological Science, 13(4), 411-417