In this edition of the Road to VIS 2024, we want to tackle the topic of reviewing and reviewer fatigue. We want to shed light on the invisible labor that our hard-working reviewers do for every paper submitted to the conference. Our goal is to enlighten and encourage, not to reproach. But yes, if this post motivates you to go to PCS right now and volunteer to review papers for VIS 2024, so much the better. (We mean it—do it now!)
The truth of the matter is that every paper you submit to IEEE VIS comes at a cost: the unpaid labor of at least three (four in earlier years) reviewers who have volunteered to evaluate your work. In addition, work is done by the Area Papers Chairs (APCs) who spend time finding suitable primary and secondary reviewers to handle your submission, and then tries to make a decision about your paper based on the reviews, not to mention us Overall Paper Chairs (OPCs) who facilitate this accept/reject decision-making process at a global level. Taken together, there are probably dozens of hours invested in each paper you submit.
This is all to say that while it may seem like submitting a paper is a little like playing the lottery for free—after all, you don’t have to pay anything to enter—there is a very real cost in human labor for every paper submitted to the conference. At the same time, many of us may admit to at one point or another saying something along the lines of “let’s submit this paper even if it’s not entirely ready—at least we’ll get good feedback.” Yes, there is often good feedback for IEEE VIS—our reviews tend to be of remarkably high quality—but the primary goal is to select high-quality papers with the potential for being accepted and nurture them to that point. An additional outcome of the review process is to provide good feedback to authors on the strengths and weaknesses of the work that is rejected, but you should not treat the IEEE VIS review process as a free editing or research critique service. Of course, it’s difficult to predict which papers will shine and which will be deemed premature—many of us have been surprised by our favorites being rejected and our long-shots accepted, so we don’t wish to discourage you from submitting work that you personally consider above the bar for publication! Rather, this post is intended to raise awareness that if we treat all of these volunteer efforts as unpaid labor that we can leverage for our own good, the system may not be sustainable.
Since reviewing is not free, it is important that authors devote time to it to keep the scales balanced. Given that three reviewers handle each paper you submit, you and your co-authors should plan to review at least three other papers in return (split between all co-authors). If you do not review your share, you are essentially benefiting from other people’s labor without contributing your own. This approach is not equitable, and in the long run it could lead to the collapse of the peer review system.
Some conferences are already seeing indications that the system is beginning to fail, particularly since the one-two punch that started with the pandemic, and continues to this day: most venues experienced a significant increase in submissions, even as more review requests were declined. VIS has not been spared—in past years, we heard tales of PC members having to ask half a dozen or more potential external reviewers before someone finally accepted the request to review a specific paper. Reviewer fatigue is real, and the only antidote is that every author pulls their own weight. ACM CHI, for example, now requires that all submissions list at least three co-authors who commit to review papers in return. With VIS, we are taking the slightly more gentle approach of strongly encouraging this level of commitment, although not yet formally requiring it.
As always in life, there are exceptions to the hard and fast rule of reviewing at least as many papers as your own paper receives. For example, junior students are often exempted from reviewing, mainly because they don’t possess the necessary experience and judgment to take on this task. Even senior authors may be excused if they have significant service commitments elsewhere. For example, none of the OPCs and APCs this year will be reviewing VIS papers because our time is already committed. Still, our philosophy is that if you take the time to author research for a conference, you should also make time to review for that conference. Please keep this principle in mind as external review requests start to appear in early April.