At VIS last year, we in the Open Practices Committee led a town hall where we discussed the long-term Open Science goals for our community. These were:
- Research should be accessible to everyone. Financial means and privileged access should not limit anyone’s ability to participate in and learn from visualization research.
- Research should be transparent, scrutinizable, and trustworthy. Authors should be as transparent as possible about their research processes and results. Increased transparency can help reviewers and readers judge for themselves whether the research conducted was plausible and whether the results are reliable.
- Research should be replicable and/or transferable. We believe that our community should support knowledge transfer between teams and that research results should stand up to scrutiny by future researchers.
For VIS 2024, we set objectives to further encourage our community to submit supplemental materials for review. We provided guidelines to assist in the submission of supplemental materials as well as guidelines for reviewing these materials. We have also continued to encourage authors to post preprints on approved servers and to provide clearer guidance for the community through the VIS 2024 Open Practices pages. To further increase our transparency, we provided an updated VGTC journal template allowing the sections “Supplemental Material”, “Figure Credits” and “Acknowledgements” on the reference-only pages.
Accepted authors (congratulations, by the way!) may also have noticed some additional questions in the camera-ready submission form, asking for information about their open practices for their work in terms of posting preprints, preregistration, and supplemental material. As our community continues to move towards more Open Science, we also asked accepted authors to reflect and consider if their work exhibited exemplary or noteworthy practices around documentation, reproducibility, or transparency. Thank you to those of you who provided all of this extra information.
So, what is the state of Open Practices for VIS 2024? How do we compare to last year, and what more can we do in the coming years?
Full Papers
Let’s have a look first at the full papers. Of the 124 accepted full papers to VIS 2024, authors reported the following Open Science practices:
Preprints
For VIS 2023, we asked authors about preprints. Self-reported data showed that about half of accepted authors posted their preprints on arXiv, just over 6% to OSF, 8% to another source, and around 35% not at all. How does this year compare?
This year, we see a slight improvement in the posting of preprints, up to 83 (67%) of submissions. This is a 2% improvement over last year. Furthermore, we see that all reported preprints have been posted to a free and open repository (ArXiv, OSF, or HAL). This is a step in the right direction.
Open Practices recommends uploading a preprint of your work to a free and open-access repository to make it more findable and accessible. Our Open Access Preprint Guide and FAQ provides more information and several tutorials to help make your preprint more available. It is never too late to do this!
Supplemental material
115 (93%) of accepted submissions used at least one supplemental material field, whether PCS, external, or both PCS and through an external link.
While not all papers necessarily have relevant supplemental material (for instance, position papers), for nearly all paper contributions, having supplemental material (like experimental data, code bases, or even just additional proofs and analyses) helps the work to be scrutinizable, so we view this high percentage as a step in the right direction. We do, however, recommend uploads to both (1) PCS for review as well as (2) a free, reliable, and long-term archive, in alignment with VIS’s long-term Open Science goals. This redundancy allows both a static copy of record associated with the entry in the IEEE digital library, as well as an open and more easily scrutinized or re-analyzed version of the work.
Besides PCS, where did accepted submissions upload their supplementary material?
We see that 80 (around 65%) of accepted submissions uploaded their supplementary materials to an external source in addition to or instead of PCS, with the majority using OSF. For further guidance on where supplementary material should be upload, please see our FAQ: Where should I upload supplementary material?
Preregistration
We also asked about study preregistration, another new question for this year. Only eight (12.5%) submissions out of the 124 full papers reported preregistration of their studies, seven of which did so on OSF Registries and one on AsPredicted.
This is a clear area where our community has room to improve. VIS Open Practices still needs formal recommendations for study preregistration, and we acknowledge that this may not be less relevant for certain contributions. However, for work involving empirical studies, preregistering an analysis plan helps us as researchers to clearly separate hypothesis generation (postdiction) from hypothesis confirmation (prediction), thus leveraging the benefits and maintaining awareness of the limitations of statistical inference while improving the study’s reproducibility and replicability [1]. OSF provides an excellent set of resources for preregistering your project. For future iterations of VIS, the Open Practices committee plans to examine more deeply where and how we support the community in preregistering relevant studies. We look forward to conversations and feedback on this process with all of you.
Short Papers
Now, how does the short papers track look? Of the 66 accepted short papers, authors reported the following on Open Practices:
Preprints
39, or around 59% of accepted short papers, posted a preprint of their work. This is lower than what we see in the full paper submissions and we would like to see this number improve in the coming years.
Of those submissions that posted preprints, most used ArXiv, a few OSF, and a few used custom domains, mainly their university hosting services.
Supplementary material
Of the 66 accepted short papers, 54 (around 82%) included supplementary material in some form in their submission, whether through PCS directly and/or through upload to an external repository. The majority of authors, as we can see, did both.
And where did authors upload their supplementary material externally?
Here, we see more variation than for the full paper submission. GitHub and OSF are similarly used, and some authors also used Kaggle, Google Drive, and Observable to host their supplementary material. As for the full papers, we recommend archiving supplementary materials to a free, reliable, long-term archive—GitHub links can change or disappear.
Self-nomination for exemplary open practices at VIS 2024
Finally, we asked accepted authors to consider self-nominating if they feel their practices around documentation, reproducibility, and/or transparency are noteworthy.
Of the 190 accepted submissions across both full and short paper tracks, 37 self-nominated their work for exemplary open practices. This is further split between 27 for the full paper and 10 for the short paper track.
Most self-nominations, whether full or short paper track, indicated provision of code, data, thorough documentation, and/or other study instruments (e.g., cleaned transcripts, codebook, study protocols) for reproducibility and/or replicability. Several also noted study preregistration and registering materials under Creative Commons licenses for re-use. As a community, we hope that soon, such practices will become the norm for all submissions.
We are grateful to those authors who self-nominated and provided further details on their practices. We are still discussing how to move forward with this information and plan to reach out to some or all self-nominees in the coming months for thoughts and feedback.
Thank you, authors!
Thanks again to all authors who provided this information. We are grateful for your time and contributions and look forward to ways that we as a community can (and should!) continue to make our science more transparent and scrutinized. As always, we welcome thoughts and feedback from the community and look forward to seeing you in Florida in just over one month.
References
[1] Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606.