With the final acceptance decisions behind us, the official VIS 2024 review process is coming to a close. We felt that it was worth taking a moment now to look back at this most recent milestone in the review process: the decision making. In fact, doing so can be quite educational; not so much for the vast majority of papers where the process went right, but rather for the very few where it went wrong (even very wrong).
Because IEEE VIS publishes its papers in a special issue of the IEEE Transactions on Visualization and Computer Graphics (TVCG) journal, the VIS review process has real teeth. TVCG stipulates two full review rounds: an initial review round (starting April 1) followed by a second round for minor revisions (starting July 1). Papers that proceed to the second round are said to be “conditionally accepted”, but make no mistake: the second round is a real review round. The primary reviewer, after discussing with the full review panel, is supposed to list required revisions in the first round review. To address these, authors get almost a full month, which is consistent with TVCG (3 months for major revisions, 1 month for minor). If these required revisions are not addressed to the satisfaction of the primary reviewer, we have no qualms about rejecting conditionally accepted papers in the second round.
For VIS 2024, we conditionally accepted a total of 129 papers out of 557 submitted papers in the first round, yielding a provisional acceptance rate of 23.2%. Even if there is no explicit target acceptance rate, this is a low number; for comparison, it was 25.8% in 2023. While we regret the low rate, our decision-making was strictly based on scientific merit and not pursuit of an arbitrary acceptance rate. Following the process outlined in our last blog post on “From Reviews to Decisions”, the conditional accepts were determined through careful discussions between the reviewers, PC members, APCs, and OPCs. We shared the first-round notifications on June 6, 2024. Beyond the 129 conditional accepts, we also recommended an additional 29 as “TVCG fast-track”; submissions that showed promise but which we deemed would need a major revision requiring more than a brief month to bring over the finish line.
What followed in that interim of three weeks in June was no doubt feverish activity across visualization labs all over the world as authors of conditionally accepted papers raced to address all the required revisions. On July 1, all 129 papers were resubmitted to the second round. At this point, the second round reviewing commenced, with the primaries (and sometimes secondaries) checking each paper, then the APCs, and finally the OPCs.
That process ended earlier this summer, and we are happy to say that the majority of authors really did do their due diligence by carefully addressing all the required revisions. A total of 124 of the 129 papers that were conditionally accepted from the first round were finally accepted at the end of the second round. For a small number of these 124 papers, there were minor issues that the OPCs, APCs, and program committee members flagged and discussed. In the end, all of these issues could be handled within the confines of the regular review process. Congratulations to the authors of these 124 papers—we very much look forward to seeing your work being presented in Florida this October!
But what about the remaining 5 papers that were rejected in the second round? We feel there are valuable lessons here to share with the community while keeping the narrative high-level enough to protect the identities of the authors, reviewers, and PC members involved.
For one of the five rejected papers, the primary flagged a situation where the authors had failed to address a critical requirement: framing the novelty of the contributions of their work with respect to previous work. Moreover, they removed almost 20 references in their second round revision. When looking closer, we found that the authors had addressed a comment about reorganizing the related work by shrinking it in half and cutting many of the original references. Importantly, several of the missing relevant literature that the reviewers asked the paper to cite had not been added to the new version of the paper (even if the authors claimed they had done so in their revision report). We OPCs ended up spending a significant amount of time checking that this was not just a case of the authors trimming the fat off the paper, but came to the same conclusion as the primary. The changes to the related work did not sufficiently address the required revision. The paper was rejected.
For a second rejected paper, the new revision was submitted to the second review round with exactly 10 pages of content and 2 pages of references. As VIS authors will know, we only allow papers with up to 9 pages of content and up to 2 pages of references. Nevertheless, the authors—some senior ones among them—resubmitted the paper with a full page beyond the limit. The paper was rejected.
You might argue for both of the above cases that these are situations where the OPCs could have engaged the author team in a conversation to address the issues rather than rejecting outright. Unfortunately, there simply isn’t sufficient time in the schedule for authors to shrink their paper down by a full page, or address novelty concerns that were not handled during revisions. And besides, there is a fairness issue; why should some authors get special treatment? What if everyone did this?
The final three rejected papers share a common theme: a severe conflict of interest violation. During the final checks on the list of accepted papers, it was discovered that a paper had been managed by a primary reviewer who was the former Ph.D. student of the paper’s senior author. As we dug deeper, we found two more conditionally accepted papers involving that senior author that had been handled by the same primary reviewer (a member of the IEEE VIS 2024 program committee).
Your own masters, Ph.D., and postdoc advisors are permanent conflicts in the IEEE VIS community (and in most scientific communities). This means that unlike for your paper co-authors or grant co-investigators, where conflicts disappear after three years, your conflict of interest with your former advisor (and, inversely, with your former students) never goes away. This information is widely known in the VIS community; it is enshrined in our paper submission guidelines with a link to the IEEE VGTC Ethics Guidelines, and it was recently discussed in our Road to VIS 2024 blog post on “Handling Conflicts”. We tell all our PC members to be mindful of conflicts, both when they are recruited as well as when they bid on papers and recruit reviewers.
In other words, knowingly reviewing papers written by your own Ph.D. advisor is a very serious breach of the integrity of the IEEE VIS review process.
PC members see the identity of all authors in PCS (our submission reviewing system) even if a paper is anonymized. This particular primary reviewer had been given clear instructions, did not declare a conflict with their advisor, and failed to tell us that they had been assigned three of their advisor’s papers as primary reviewer—for four months!
As far as we can tell, nothing like this has ever happened before in the 30-year history of the VIS conference. We thus had no procedures or tool support to spot this specific problem. Furthermore, with a total of more than 1,900 unique authors submitting papers to VIS 2024, it is virtually impossible for the OPCs and APCs to be fully aware of all conflicts between them. Instead, we rely on our PC members, reviewers, and authors to carefully self-declare all these conflicts in advance of the review assignment process. Note that we don’t expect perfection: after the assignments are released, there is a chance for reviewers to let us know if a previously undetected conflict appears. There were several situations when this happened, causing us OPCs to swap papers between PC members when people discovered a conflict not previously declared.
Unfortunately, the fact that this conflict did happen irrevocably compromised the review process for all three of these papers. Primary reviewers have a significant impact on a paper’s fate by leading the discussion and summarizing the other reviews. There is no easy way to disentangle this impact. Perhaps if the problem had been spotted in the first round, we could conceivably have replaced the primary. At this point, near the very end, it is impossible to know whether these papers would have even made it to the second round with an unconflicted primary. There is also no time to run the review process all over again given our tight schedule. As a result, all three papers were rejected, but with a special offer from TVCG for the papers to be immediately resubmitted to the journal while retaining the two unconflicted reviewers and only replacing the primary.
The fallout from this incident is significant. Most serious is the impact on the graduate students whose papers were rejected due to factors beyond their own control. Second, the senior author (the Ph.D. advisor of the primary reviewer at fault) has three of their conditionally accepted papers rejected, again due to no fault of their own. And third, the primary reviewer who committed this breach of review integrity and professional ethics has opened themselves up to an academic misconduct investigation.
The greater lesson from this incident is to take conflicts of interest very seriously. Make sure that you carefully declare not only your co-authors and co-investigators in PCS, but also your masters, Ph.D., and postdoc advisors. If you have ever advised students, you should also declare your own current and former students as permanent conflicts! Vigilance about your own conflicts is one good strategy to protect yourself from situations like this in the future.
There is damage to the VIS review process as well. Our goal when addressing the incident was to protect the scientific integrity of VIS (and TVCG), but there can be no perfect solution to this kind of difficult problem. We OPCs and APCs will be reviewing our procedures to ensure that this problem will not happen again. We will be looking into ways to improve the PCS paper submission and reviewing system to spot problems of this nature. And we will do our best to educate the community about the importance of conflicts. This blog post is part of that latter effort.