Review Instructions

Updated 6 April 2020

The success of the IEEE VIS 2020 papers program relies on a knowledgeable, experienced, and ethical community of reviewers. IEEE VIS 2020 employs a two-phase, (author-optional) double-blind review process that follows the review process for IEEE TVCG, the journal in which most IEEE VIS full papers are published. All papers are reviewed by at least 4 reviewers, at least 2 of which are IPC members and at least 2 of which are external members of the community. All reviewers are bound to the ethics guidelines of the IEEE VGTC.

This page provides detailed instructions for both IPC reviewers and external reviewers.

Contents

Note: Dates are subject to change due to our coronavirus response.

IPC Reviewing Instructions

Keywords Update

By April 20

At VIS 2019, the VEC and the V-I-S Steering Committees adopted a proposal by the reVISe committee to replace the current PCS keywords with a new set of keywords to improve the reviewing process. You will need to update your expertise in PCS with these new keywords. Please read through the summary of the new keywords, which includes instructions for reviewers on how to select for expertise.

  1. Log onto the PCS system.

  2. Go to the Reviews tab.

  3. Click on “Provide areas of expertise”.

  4. Select “VGTC” for the Society, and “VIS 2020” for the Conference/Journal.

  5. Select your keywords.

Bidding

April 21 thru April 30

If you have not yet updated your keywords then see the Keywords Update phase above.

  1. Log onto the PCS system.

  2. Go to the Reviews tab.

  3. Click on “Review (as a committee member)” beside “VAST/InfoVis/SciVis”.

  4. Click on “Conflicts”.

  5. Click the checkbox beside each author with whom you are in conflict. The checkbox will flash green to indicate that your conflict has been recorded. There is no submit button on this page. You can type an institution name in the “Search” box in the upper-right corner to show people from a particular institution. For guidance on reviewer conflicts-of-interest please see the IEEE VGTC ethics guidelines for reviewers.

  6. Click on the VAST/InfoVis/SciVis tab under the Reviews tab to return to your main reviewing page.

  7. Click on “Bidding”.

  8. For bidding, you will see four options for each submission: want, willing, reluctant, and conflict. The conflicted papers should be pre-selected as conflicts, based on steps 4-5 above. Please select “want” for papers that match your areas of expertise; typically, we expect that you mark 10 to 20 papers as “want”. If you do not click an option, the system will record your preference as “willing”. Please select “reluctant” for any papers that you feel are far from your expertise. You may find it easiest to first sort the list by the ‘score’ column, which estimates the match between your expertise and the paper keywords. Click the appropriate radio button beside each submission. The radio button will flash green to indicate that your bid has been recorded. There is no submit button on this page. Submissions that have authors with whom you have declared a conflict will show “author conflict” in place of the radio buttons.

Assigning External Reviewers

May 4 thru May 10

Both primaries and secondaries are to recruit ONE external reviewer each. Make certain that this reviewer is NOT from the same institution/group as any of the other reviewers.

PCS has an option to suggest reviewers to you according to a match score. This list of suggested reviewers and their interests may be an aid to jog your memory, but note that it must be used with caution. Do not blindly rely on these suggestions without checking carefully that the suggested reviewers are indeed a good fit. Consider the mix of expertise and experience across all four reviewers, including junior vs. senior, male vs. female, practitioner vs. researcher, industry vs. academia, etc. We aim for a good mix of perspectives.

Please do not hesitate to add new, qualified reviewers that are not already in the PCS database using the “register new person” mechanism. We are always looking for the top experts in a specific topic, and extending the database will benefit others as well.

Follow up with the external reviewer and ensure that they accept the invitation. If they decline or do not respond in a timely manner, please find a replacement as soon as possible.

Do not ask other IPC members from any of the three tracks (VAST, InfoVis, SciVis) to be external reviewers. The full IPC list can be found on the conference website. (Note that they are also marked as “cmte” on the review assignment screen in PCS.)

First Round Reviewing

May 4 thru June 7

During this phase, both primaries and secondaries:

  • Write a regular review of their assigned papers.

Additionally, the primaries:

  • Contact the paper chairs at an early stage if they need copies of any related work that they cannot access (e.g., EuroVis submissions). The paper chairs can also communicate directly with authors if there are other inquiries. The sooner this engagement occurs, the better, so please get in touch.
  • Remind all external reviewers to complete their reviews on time as the review deadline approaches. The paper chairs will send one reminder as well, a week before the deadline.

As an IEEE VIS IPC member, you are in a position to directly control the quality of the VIS papers program. Make sure that you stand behind the work you recommend for acceptance! Most importantly, this means that you should only accept high-quality papers that will contribute significantly to the VIS research community.

However, you should also be on the lookout for bold, creative, and potentially transformative work that may significantly extend the frontier of visualization research. While such submissions should be held to the same quality standard as all other submissions, it is expected that they may not be able to exhaustively (or perfectly) validate all of their contributions. Such papers often become easy targets for tertiary reviewers, so it is your job as a primary or secondary reviewer to use your experience and perspective to recognize them before they are summarily rejected. We do not want the conference to consist of work with the fewest flaws; we need to weigh the possible significance of the work and the degree of ambition alongside the extent to which it is described, compared, and validated.

If you have any questions in this regard about a particular paper, please either flag this in the confidential section of your review or email the paper chairs directly. Your insight and ability to look beyond the surface is exactly the kind of guidance the paper chairs need from an experienced IPC.

Please see Additional Reviewing Guidance for further considerations.

Discussion

June 8 thru June 14

Primaries:

  • As soon as possible, initiate discussion with the other reviewers in order to reach a consensus about whether the paper should be conditionally accepted or rejected for publication.
  • Ensure that all reviewers contribute to the discussion about the paper. Consider emailing external reviewers directly if they are not responsive.
  • By the end of the discussion period, write a summary review with a recommendation of conditional acceptance or rejection based on the reviews and the discussion. This summary review MUST explicitly spell out what the conditions are for turning a conditionally accepted paper into an accepted one. The authors need to know what is required of them as well as what comments they should consider but that aren’t required to address. If you and the secondary reviewer cannot agree on a recommendation for a submission, please document the discussion in the confidential portion of the summary review. If views have changed or consensus cannot be achieved, please let us know why in the confidential comments. Summary reviews are due on June 14.

Secondaries:

  • Discuss with the primary reviewer to agree on a recommendation in the first review cycle.

Please keep the discussion anonymous by referring to other reviewers as R1, R2, etc, and remember to not sign your name in the discussions.

First Round Decisions

June 15 thru June 29

Please be available electronically for discussion with the paper chairs about the papers, your recommendation, and the review process. This is particularly important if any of your review processes are contentious or controversial. If you will not be available to field our questions, please let the paper chairs know in advance.

Shepherding

July 7 thru July 31

At the beginning of July, the paper chairs will send out conditional acceptance and rejection notifications. For some conditionally accepted papers the summary review indicates that there is either a longer set of things to change or a few that will require more careful work. For these papers the paper chairs will encourage authors to engage with the primary reviewer early in the revision period to clarify that their proposed edits/changes will be sufficient.

The paper chairs will reach out to primaries on shepherded papers to ensure their availability during this phase.

Primaries: As primary of a paper undergoing shepherding, you agree to communicate anonymously via PCS with the authors to help clarify questions about revision comments, to give feedback on proposed changes, and to provide an informal review of a revised paper draft during the revision phase of the review process. It is, however, the responsibility of the authors to initiate communication and to ask for feedback with sufficient time for the primary to respond, ie. it is not expected that a primary will be able to give feedback on a paper draft a day before the revision deadline. Additionally, primaries are not expected to engage in copy-editing of the paper, but instead to offer high-level feedback on revisions.

All shepherded papers must meet the same review expectations as other conditionally accepted papers, and failure to do so by the authors should result in the paper ultimately being rejected.

In communicating with authors, be careful not to send email with your signature attached in order to maintain your anonymity, and only correspond through the PCS system.

Second Round Reviewing

August 1 thru August 7

Primaries: For each of the papers for which you are the primary reviewer and that has received a conditional accept decision, please:

  1. READ THE SUMMARY OF CHANGES AND CHANGED PAPER and consider whether the authors adequately addressed the required changes from your summary review. For most cases, deciding whether a revised version of a paper is acceptable is a decision that the primary can make on their own. If you have any doubt, feel free to initiate a discussion with the secondary. If you think that there remain any minor issues that authors can fix simply, PCS also allows you to communicate anonymously with the authors. If you do this, please let the paper chairs know so we can re-open the paper for author edits. In communicating with authors, be careful not to send email with your signature attached, to maintain your anonymity. Finally, if applicable please also check revisions to supplementary material.

  2. WRITE A RESPONSE: First select a final decision for the paper in your review form for the paper. The options are “Accept” and “Unsure”. It is relatively rare that papers are rejected at this stage, so if you find yourself selecting the “Unsure” rating, please communicate with the paper chairs as soon as possible. Next, add text in the “Second round comments (public)” text box indicating that you have read the changed document and include any other responses you may have. Please note that you can still make requests for cosmetic fixes (e.g., typos, etc) for the camera-ready copy. However, you should not be requesting new substantive changes. Also fill in the form items on the acceptability of the supplementary material, and recommend a publicity image if applicable.

External Reviewing Instructions

Keywords Update

By May 3

At VIS 2019, the VEC and the V-I-S Steering Committees adopted a proposal by the reVISe committee to replace the current PCS keywords with a new set of keywords to improve the reviewing process. You will need to update your expertise in PCS with these new keywords. Please read through the summary of the new keywords, which includes instructions for reviewers on how to select for expertise.

  1. Log onto the PCS system.

  2. Go to the Reviews tab.

  3. Click on “Provide areas of expertise”.

  4. Select “VGTC” for the Society, and “VIS 2020” for the Conference/Journal.

  5. Select your keywords.

Reviewing

May 4 thru June 7

High-quality reviews are critical to the quality of the conference program. Please familiarize yourself with the reviewing ethics and guidelines described in Additional Reviewing Guidance.

Discussion

June 8 thru June 14

Following the completion of the reviewing phase, the primary reviewer will initiate a discussion with all reviewers of a paper through the PCS system. During this time you are to engage with the other reviewers to come to a consensus decision on whether to conditionally accept or reject the paper. Please keep the discussion anonymous by referring to other reviewers as R1, R2, etc, and remember to not sign your name in the discussions.

Additional Reviewing Guidance

Paper Quality

As an IEEE VIS reviewer, you are in a position to directly control the quality of the VIS papers program. Make sure that you stand behind the work you recommend for acceptance! Most importantly, this means that you should only accept high-quality papers that will contribute significantly to the VIS research community. You should, however, also be on the lookout for bold, creative, and potentially transformative work that may significantly extend the frontier of visualization research. While such submissions should be held to the same quality standard as all other submissions, it is expected that they may not be able to exhaustively (or perfectly) validate all of their contributions. We do not want the conference to consist of work with the fewest flaws; we need to weigh the possible significance of the work and the degree of ambition alongside the extent to which it is described, compared, and validated.

Reviewing Ethics

All reviewers should make sure to follow the basic principles outlined below. As a primary or secondary, feel free to remind external reviewers of these as well as any additional resources you find helpful.

  • Be timely: the conference has tight deadlines and late reviews are unfair to submitting authors as they reduce the time for discussion for this work.
  • Protect ideas: the content of submissions should be handled as confidential information and should not be disclosed to close research colleagues or any wider audience.
  • Avoid conflicts of interest: these include being directly involved in the work, co-author of one of the authors within the past 3 years, member of the same institution of one of the authors (or have been in the past year), supervisor/advisor of one of the authors, familial or emotional tie to one of the authors, or any other factor that would result in a biased review of the submitted work. If you discover a conflict of interest after the reviewing phase begins please contact the paper chairs immediately.
  • Be specific and constructive: offer specific critiques that are addressable by the authors, point out concrete issues to fix in an encouraging manner, and substantiate your comments with evidence and references.
  • Find reasons to accept papers: have an open-mind or at least disclose your biases. It is easier to find reasons to reject papers than to point out the contributions that they bring to the community. Emphasize positive aspects to help surface impactful research to the community.
  • Be tactful and polite: emotional rants or sarcastic comments have no place in a professional review. Ask yourself if you could read your review aloud in front of an audience including the authors.

Additionally, all reviewers are bound to the ethics guidelines of the IEEE VGTC. Please make sure you are familiar with these guidelines.

What is a minor revision?

All IEEE VIS acceptances are conditional after the first round of reviewing, after which authors will be asked to perform a minor revision on the paper. While a select few papers do not require any changes at all, this is very rare. In other words, reviewing in the first round becomes one of deciding which papers will be acceptable for publication after a minor revision. No paper is perfect; rather, your job is to identify the work that will have a significant contribution to visualization after the authors get a period of four weeks to address reviewer comments.

So, what is a minor revision? We rely on you to draw on your own experience to make this judgment. Don’t be close-minded, skeptical, or suspicious; be trusting and open to new ideas. Don’t expect perfectionism, reward pragmatism.

Here is an incomplete list of example changes suitable for a minor revision:

  • Add missing related work (unless the missing work invalidates the contribution).
  • Reinforce arguments or improve reasoning (unless the central argument is fundamentally flawed).
  • Adapt the contribution and scope to better fit the research area (unless there is no conceivable connection to relevant topics in the area).
  • Improve language, spelling, and grammar (unless poor language precludes understanding).
  • Change poor visual design choices (unless they are central to the contribution).
  • Correct issues with statistical tests (unless you suspect the new results will be radically different).
  • Add or improve illustrations or images (unless this requires non-trivial algorithm changes or additional research).

There are many ways to fix an imperfect paper, and countless others are possible. Keep an open mind.

Managing Resubmission Materials

As we all know, successful publication in a premier venue such as VIS sometimes takes time. It can be frustrating for authors to resubmit a rejected paper, only to have a new set of reviewers reject a paper for precisely the opposite reasons from the original rejection. To add some memory to the process, we now encourage authors to submit resubmission materials to their papers, including past versions, past reviews, and a cover letter with the authors’ response and a list of changes made.

Providing resubmission materials is OPTIONAL and at the authors’ discretion. Even if you have reviewed a paper before, you should not penalize that paper if the authors chose not to provide resubmission materials (or vice versa).

As with all supplemental materials, each individual reviewer decides how to treat resubmission materials. However, the paper chairs urge you to at least take a look at any resubmission material provided and write your reviews based on a full picture of the paper.

Assigning Scores, Requesting Revisions

Remember that the numeric scores are shorthand for a recommendation. Please check the descriptions for the numbers on the reviewing form and carefully justify your recommendation in your review. Short reviews do not help the paper chairs in making their decisions.

Note that submitting a neutral/borderline rating of 3 is acceptable, but not encouraged. If you do assign a 3, please specify in the revisions what must be completed within the (short) conference review cycle for the paper to become an ‘Accept’.