Review Instructions

The success of the IEEE VIS 2021 papers program relies on a knowledgeable, experienced, and ethical community of reviewers. IEEE VIS 2021 employs a two-phase, (author-optional) double-blind review process that follows the review process for IEEE TVCG, the journal in which IEEE VIS full papers are published. All papers are reviewed by at least 4 reviewers, at least 2 of which are PC members and at least 2 of which are external members of the community. All reviewers are bound to the ethics guidelines of the IEEE VGTC.

This page provides detailed instructions for both PC reviewers and external reviewers.

Contents

PC Reviewing Instructions

Keywords Update

By March 20

You will need to update your expertise in PCS with the keywords adopted since 2019. Please read through the summary of the keywords, which includes instructions for reviewers on how to select for expertise. If you do not enter your areas of expertise you will have all matching scores set to -1 during the bidding phase which will greatly impact suggestions of interesting papers for you to bid on. The matching scores are calculated manually by OPCs once before bidding starts and if you are late you will have to wait for them to do this step again at a later point in time. Therefore, it is important that you enter you expertise before the abstract deadline. Please note that the Area Curation Committee (ACC) also continuously collects information on missing keywords, so be sure to enter keywords you think are missing in the respective checkboxes.

  1. Log onto the PCS system.

  2. Go to the Reviews tab.

  3. Click on “Provide areas of expertise”.

  4. Select “VGTC” for the Society, and “VIS 2021” for the Conference/Journal.

  5. Select your keywords.

Bidding

March 22 thru March 31

We cannot overemphasize the importance of bidding: proper and timely bidding is crucial for the successful paper assignment. Please complete your bidding by March 31, 2021.

  1. Log onto the PCS system.

  2. Go to the Reviews tab.

  3. Click on the “Review (as a committee member)” action for the “VIS 2021 Papers” category.

  4. Click on the “Conflicts” link.

  5. Click the checkbox for each author with whom you are in conflict. The checkbox will flash green to indicate that your conflict has been recorded. There is no submit button on this page.

    You can type an institution name in the “Search” box in the upper-right corner to show people from a particular institution. For guidance on reviewer conflicts-of-interest please see the IEEE VGTC ethics guidelines for reviewers.

To bid papers, continue to follow these steps:

  1. Click on “VIS 2021 Papers” under the Reviews tab to return to your main reviewing page.

  2. Click on the “Bidding” link.

  3. For bidding, you will see four options for each submission: want, willing, reluctant, and conflict. The conflicted papers should be pre-selected as conflicts, based on steps 4-5 above: these paeprs will show “author conflict” instead of the radio buttons.

    You can select an option by clicking the appropriate radio button beside each submission. The radio button will flash green to indicate that your bid has been recorded. There is no submit button on this page.

    Please select “want” for papers that match your areas of expertise; we expect that you mark at least 50 papers as want or willing. Please select “reluctant” for any papers that you feel are far from your expertise. Note that, if you do not select any option, the system will record your preference as “willing” by default.

Useful Tips for Bidding

As a unified conference, the number of submissions drastically increased. It may be too time consuming to read through all the abstracts carefully. You may first sort the list by the ‘score’ column, which estimates the match between your expertise and the paper keywords.

Another way to find papers that match your expertise and interest is to use (1) the search box (which searches in all the fields) and (2) the option to see papers by area or keyword. You can find additional information about bidding strategies on the IEEE VIS blog.

Assigning External Reviewers

April 14 thru April 21

Both primaries and secondaries are to recruit ONE external reviewer each. Make certain that this reviewer is NOT from the same institution/group as any of the other reviewers. For the single-blind submissions, the assigned reviewers should double check conflicts of interest as well.

PCS has an option to suggest reviewers to you according to a match score. This list of suggested reviewers and their interests may be an aid to jog your memory, but note that it must be used with caution. Do not blindly rely on these suggestions without checking carefully that the suggested reviewers are indeed a good fit. Consider the mix of expertise and experience across all four reviewers, including varied career stages, gender diversity, practitioners and researchers, industry and academia, etc. We aim for a good mix of perspectives.

Please do not hesitate to add new, qualified reviewers that are not already in the PCS database using the “register new person” mechanism. We are always looking for the top experts in a specific topic, and extending the database will benefit others as well.

Follow up with the external reviewer and ensure that they accept the invitation. If they decline or do not respond in a timely manner, please find a replacement as soon as possible.

Do not ask other PC, APC, or OPC members to be external reviewers. The full PC list can be found on the conference website. (Note that they are also marked as “cmte” on the review assignment screen in PCS.)

First Round Reviewing

April 14 thru May 14

During this phase, both primaries and secondaries:

  • Write a regular review of their assigned papers.

Additionally, the primaries:

  • Contact the APCs at an early stage if they need copies of any related work that they cannot access (e.g., EuroVis submissions). The paper chairs can also communicate directly with authors if there are other inquiries. The sooner this engagement occurs, the better, so please get in touch.
  • Remind all external reviewers to complete their reviews on time as the review deadline approaches. The OPCs will send one reminder as well, a week before the deadline.

As an IEEE VIS PC member, you are in a position to directly control the quality of the VIS papers program. Make sure that you stand behind the work you recommend for acceptance! Most importantly, this means that you should only accept high-quality papers that will contribute significantly to the VIS research community.

However, you should also be on the lookout for bold, creative, and potentially transformative work that may significantly extend the frontier of visualization research. While such submissions should be held to the same quality standard as all other submissions, it is expected that they may not be able to exhaustively (or perfectly) validate all of their contributions. Such papers often become easy targets for tertiary reviewers, so it is your job as a primary or secondary reviewer to use your experience and perspective to recognize them before they are summarily rejected. We do not want the conference to consist of work with the fewest flaws; we need to weigh the possible significance of the work and the degree of ambition alongside the extent to which it is described, compared, and validated.

If you have any questions in this regard about a particular paper, please either flag this in the confidential section of your review or email the paper chairs directly. Your insight and ability to look beyond the surface is exactly the kind of guidance the paper chairs need from an experienced PC.

Please see Additional Reviewing Guidance for further considerations.

Discussion

May 15 thru May 21

Primaries:

  • As soon as possible, initiate discussion with the other reviewers in order to reach a consensus about whether the paper should be conditionally accepted or rejected for publication.
  • Ensure that all reviewers contribute to the discussion about the paper. Consider emailing external reviewers directly if they are not responsive.
  • By the end of the discussion period, write a summary review with a recommendation of conditional acceptance or rejection based on the reviews and the discussion. This summary review MUST explicitly spell out what the conditions are for turning a conditionally accepted paper into an accepted one. The authors need to know what is required of them as well as what comments they should consider but that aren’t required to address. If you and the secondary reviewer cannot agree on a recommendation for a submission, please document the discussion in the confidential portion of the summary review. If views have changed or consensus cannot be achieved, please let us know why in the confidential comments. Summary reviews are due on May 21.

Secondaries:

  • Discuss with the primary reviewer to agree on a recommendation in the first review cycle.

Please keep the discussion anonymous by referring to other reviewers as R1, R2, etc., and remember to not sign your name in the discussions.

First Round Decisions

May 22 thru June 13

Please be available electronically for discussion with the paper chairs about the papers, your recommendation, and the review process. This is particularly important if any of your review processes are contentious or controversial. If you will not be available to field our questions, please let the paper chairs know in advance.

Shepherding

June 14 thru July 1

At the beginning of June, the paper chairs will send out conditional acceptance and rejection notifications. For some conditionally accepted papers the summary review indicates that there is either a longer set of things to change or a few that will require more careful work. For these papers the paper chairs will encourage authors to engage with the primary and secondary reviewers early in the revision period to clarify that their proposed edits/changes will be sufficient.

The paper chairs will reach out to primaries and secondaries on shepherded papers to ensure their availability during this phase.

Primaries: As primary of a paper undergoing shepherding, you agree to communicate anonymously via PCS with the authors to help clarify questions about revision comments, to give feedback on proposed changes, and to provide an informal review of a revised paper draft during the revision phase of the review process. It is, however, the responsibility of the authors to initiate communication and to ask for feedback with sufficient time for the primary to respond, ie. it is not expected that a primary will be able to give feedback on a paper draft a day before the revision deadline. Additionally, primaries are not expected to engage in copy-editing of the paper, but instead to offer high-level feedback on revisions.

All shepherded papers must meet the same review expectations as other conditionally accepted papers, and failure to do so by the authors should result in the paper ultimately being rejected.

In communicating with authors, be careful not to send email with your signature attached in order to maintain your anonymity, and only correspond through the PCS system.

Second Round Reviewing

July 2 thru July 14

Primaries: For each of the papers for which you are the primary reviewer and that has received a conditional accept decision, please:

  1. READ THE SUMMARY OF CHANGES AND CHANGED PAPER and consider whether the authors adequately addressed the required changes from your summary review. For most cases, deciding whether a revised version of a paper is acceptable is a decision that the primary can make on their own. If you have any doubt, feel free to initiate a discussion with the secondary. If you think that there remain any minor issues that authors can fix simply, PCS also allows you to communicate anonymously with the authors. If you do this, please let the paper chairs know so we can re-open the paper for author edits. In communicating with authors, be careful not to send email with your signature attached, to maintain your anonymity. Finally, if applicable please also check revisions to supplementary material.

  2. WRITE A RESPONSE: First select a final decision for the paper in your review form for the paper. The options are “Accept” and “Unsure”. It is relatively rare that papers are rejected at this stage, so if you find yourself selecting the “Unsure” rating, please communicate with the APCs as soon as possible. Next, add text in the “Second round comments (public)” text box indicating that you have read the changed document and include any other responses you may have. Please note that you can still make requests for cosmetic fixes (e.g., typos, etc) for the camera-ready copy. However, you should not be requesting new substantive changes. Also fill in the form items on the acceptability of the supplementary material, and recommend a publicity image if applicable.

External Reviewing Instructions

Keywords Update

By March 31

You will need to update your expertise in PCS with the keywords adopted since 2019. Please read through the summary of the keywords, which includes instructions for reviewers on how to indicate their levels of expertise. Make sure to update your areas of expertise by the deadline above so that the program committee can take them into account during the reviewer assignment period.

  1. Log onto the PCS system.

  2. Go to the Reviews tab.

  3. Click on “Provide areas of expertise”.

  4. Select “VGTC” for the Society, and “VIS 2021” for the Conference/Journal.

  5. Select your keywords.

Reviewing

Apr 14 thru May 14

High-quality reviews are critical to the quality of the conference program. Please familiarize yourself with the reviewing ethics and guidelines described in Additional Reviewing Guidance.

Discussion

May 15 thru May 21

Following the completion of the reviewing phase, the primary reviewer will initiate a discussion with all reviewers of a paper through the PCS system. During this time you are to engage with the other reviewers to come to a consensus decision on whether to conditionally accept or reject the paper. Please keep the discussion anonymous by referring to other reviewers as R1, R2, etc., and remember to not sign your name in the review or the discussions.

Additional Reviewing Guidance

Paper Quality

As an IEEE VIS reviewer, you are in a position to directly influence the quality of the VIS papers program. Make sure that you stand behind the work you recommend for acceptance! Most importantly, this means that you should only accept high-quality papers that will contribute significantly to the VIS research community. You should, however, also be on the lookout for bold, creative, and potentially transformative work that may significantly extend the frontier of visualization research. While such submissions should be held to the same quality standard as all other submissions, it is expected that they may not be able to exhaustively (or perfectly) validate all of their contributions. We do not want the conference to consist of work with the fewest flaws; we need to weigh the possible significance of the work and the degree of ambition alongside the extent to which it is described, compared, and validated.

Reviewing Ethics

All reviewers should make sure to follow the basic principles outlined below. As a primary or secondary, feel free to remind external reviewers of these as well as any additional resources you find helpful.

  • Be timely: the conference has tight deadlines and late reviews are unfair to submitting authors as they reduce the time for discussion for this work.
  • Protect ideas: the content of submissions should be handled as confidential information and should not be disclosed to close research colleagues or any wider audience.
  • Avoid conflicts of interest: these include being directly involved in the work, co-author of one of the authors within the past 3 years, member of the same institution of one of the authors (or have been in the past year), supervisor/advisor of one of the authors, familial or emotional tie to one of the authors, or any other factor that would result in a biased review of the submitted work. If you discover a conflict of interest after the reviewing phase begins please contact the paper chairs immediately.
  • Be specific and constructive: offer specific critiques that are addressable by the authors, point out concrete issues to fix in an encouraging manner, and substantiate your comments with evidence and references.
  • Find reasons to accept papers: have an open-mind or at least disclose your biases. It is easier to find reasons to reject papers than to point out the contributions that they bring to the community. Emphasize positive aspects to help surface impactful research to the community.
  • Be tactful and polite: emotional rants or sarcastic comments have no place in a professional review. Ask yourself if you could read your review aloud in front of an audience including the authors.

Additionally, all reviewers are bound to the ethics guidelines of the IEEE VGTC. Please make sure you are familiar with these guidelines.

What is a minor revision?

All IEEE VIS acceptances are conditional after the first round of reviewing, after which authors will be asked to perform a minor revision on the paper. While a select few papers do not require any changes at all, this is very rare. In other words, reviewing in the first round becomes one of deciding which papers will be acceptable for publication after a minor revision. No paper is perfect; rather, your job is to identify the work that will have a significant contribution to visualization after the authors get a period of four weeks to address reviewer comments.

So, what is a minor revision? We rely on you to draw on your own experience to make this judgment. Don’t be close-minded, skeptical, or suspicious; be trusting and open to new ideas. Don’t expect perfectionism, reward pragmatism.

Here is an incomplete list of example changes suitable for a minor revision. You cannot require a specific result, but ask for details or another example, like

  • Add missing related work (unless the missing work invalidates the contribution(s)).
  • Reinforce arguments or improve reasoning (unless the central argument is fundamentally flawed).
  • Adapt the exposition of the contribution and scope to better fit the research area (unless there is no conceivable connection to relevant topics in the area).
  • Improve language, spelling, and grammar (unless poor language precludes understanding).
  • Change poor visual design choices (unless they are central to the contribution).
  • Correct issues with statistical tests (unless you suspect the new results will be radically different).
  • Add or improve illustrations or images (unless this requires non-trivial algorithm changes or additional research).

There are many ways to fix an imperfect paper, and countless others are possible. Keep an open mind. Avoid prescribing a solution: explain the problem and allow the authors to choose how to address it.

Managing Resubmission Materials

As we all know, successful publication in a premier venue such as VIS sometimes takes time. It can be frustrating for authors to resubmit a rejected paper, only to have a new set of reviewers reject a paper for precisely the opposite reasons from the original rejection. To add some memory to the process, we now encourage authors to submit resubmission materials to their papers, including past versions, past reviews, and a cover letter with the authors’ response and a list of changes made.

Providing resubmission materials is OPTIONAL and at the authors’ discretion. Even if you have reviewed a paper before, you should not penalize that paper if the authors chose not to provide resubmission materials (or vice versa).

As with all supplemental materials, each individual reviewer decides how to treat resubmission materials. However, the paper chairs urge you to at least take a look at any resubmission material provided and write your reviews based on a full picture of the paper.

Assigning Scores, Requesting Revisions

Remember that the numeric scores are shorthand for a recommendation. Please check the descriptions for the numbers on the reviewing form and carefully justify your recommendation in your review. Short, superficial reviews do not help the paper chairs in making their decisions.

Note that submitting a neutral/borderline rating of 3 is acceptable, but not encouraged. If you do assign a 3, please specify in the revisions what must be completed within the (short) conference review cycle for the paper to become an ‘Accept’.