The success of the IEEE VIS papers program relies on a knowledgeable, experienced, and ethical community of reviewers. IEEE VIS employs a two-phase, (author-optional) double-blind review process that follows the review process for IEEE TVCG, the journal in which IEEE VIS full papers are published. Submissions are reviewed by at least 4 reviewers, at least 2 of which are PC members and at least 2 of which are external members of the community. All reviewers are bound to the ethics guidelines of the IEEE VGTC.
This page provides detailed instructions for both PC reviewers and external reviewers.
Contents
PC Reviewing Instructions
Keywords Update
By March 20
You will need to update your expertise in PCS if you have not already done so after the 2020 rollout of the new keywords. Please read through the summary of the keywords, which includes instructions for reviewers on how to select for expertise. If you do not enter your areas of expertise you will have all matching scores set to -1 during the bidding phase which will greatly impact suggestions of interesting papers for you to bid on. The matching scores are calculated manually by the OPCs once before bidding starts and if you are late you will have to wait for them to do this step again at a later point in time. Therefore, it is important that you enter you expertise before the abstract deadline. Please note that the Area Curation Committee (ACC) also continuously collects information on missing keywords, so be sure to enter keywords you think are missing in the respective checkboxes.
-
Log onto the PCS system.
-
Go to the Reviews tab.
-
Click on “Provide areas of expertise”.
-
Select “VGTC” for the Society, and “VIS 2023” for the Conference/Journal.
-
Select your keywords.
Bidding
March 22 thru March 31
We cannot overemphasize the importance of bidding: proper and timely bidding is crucial for the successful paper assignment. Please complete your bidding by March 31. If you do not carry out your bidding work thoroughly and promptly, you will be removed from the PC.
-
Log onto the PCS system.
-
Go to the Reviews tab.
-
Click on the “Review (as a committee member)” action for the “VIS 2023 Papers” category.
-
Click on the “Conflicts” link.
-
Click the checkbox for each author with whom you are in conflict. The checkbox will flash green to indicate that your conflict has been recorded. There is no submit button on this page.
You can type an institution name in the “Search” box in the upper-right corner to show people from a particular institution. For guidance on reviewer conflicts-of-interest please see the IEEE VGTC ethics guidelines for reviewers.
To bid papers, continue to follow these steps:
-
Click on “VIS 2023 Papers” under the Reviews tab to return to your main reviewing page.
-
Click on the “Bidding” link.
-
For bidding, you will see four options for each submission: want, willing, reluctant, and conflict. The conflicted papers should be pre-selected as conflicts, based on steps 4-5 above: these papers will show “author conflict” instead of the radio buttons.
You can select an option by clicking the appropriate radio button beside each submission. The radio button will flash green to indicate that your bid has been recorded. There is no submit button on this page.
Please select “want” for papers that match your areas of expertise; we expect that you mark at least 50 papers as want or willing. Please select “reluctant” for any papers that you feel are far from your expertise. Note that, if you do not select any option, the system will record your preference as “willing” by default.
Useful Tips for Bidding
As a unified conference, the number of submissions drastically increased. It may be too time consuming to read through all the abstracts carefully. You may first sort the list by the ‘score’ column, which estimates the match between your expertise and the paper keywords.
Another way to find papers that match your expertise and interest is to use (1) the search box (which searches in all the fields) and (2) the option to see papers by area or keyword. You can find additional information about bidding strategies on the IEEE VIS blog.
Assigning External Reviewers
April 8 thru April 14
Both primaries and secondaries are to recruit ONE external reviewer each. Make certain that this reviewer is NOT from the same institution/group as any of the other reviewers. For the single-blind submissions, the assigned PC members should double check conflicts of interest as well.
Consider the mix of expertise and experience across all four reviewers, including varied career stages, gender diversity, practitioners and researchers, industry and academia, etc. We aim for a good mix of perspectives.
PCS has an option to suggest reviewers to you according to a match score. This list of suggested reviewers and their interests may be an aid to jog your memory, but note that it must be used with caution. Do not blindly rely on these suggestions without checking carefully that the suggested reviewers are indeed a good fit, since expertise is self-reported and has not been vetted in any way. Moreover, the Volunteered Reviews number in PCS may not be up to date, as many people in the PCS database may not have updated their willingness numbers but may indeed be willing to accept requests; it is reasonable to ignore the automatic warning to secure their permission in advance before assigning, and simply use PCS itself to make the request. Please do not hesitate to add new, qualified reviewers that are not already in the PCS database using the “register new person” mechanism. We are always looking for the top experts in a specific topic, and extending the database will benefit others as well.
New in 2023: Batch release of initial external requests. The first round of invitations to external reviewers will be released in a single batch on April 14, to encourage externals to make immediate decisions for which requests to accept. After that, review requests will be send out immediately. PC members should assemble a list of multiple suitable externals for each paper as backup possibilities, in addition to your initial choice, so that you can send another request immediately if your initial request(s) are declined. PC members should actively check that invited external reviewers accept the invitation. If they decline or do not respond in a timely manner (i.e., within a few days of the request being sent), please send a request to the next person on your list as soon as possible.
Do not ask other PC, APC, or OPC members to be external reviewers. The full Program Committee listing can be found on the conference website PC page. (Note that they are also marked as “cmte” on the review assignment screen in PCS.) The Area Papers Chairs and Overall Papers Chairs are listed on the conference website Organizers page. In addition, do not ask the EiC (Editor in Chief) or A-EiCs (Associate Editor in Chiefs) of IEEE TVCG to be external reviewers as they also oversee the review process and should remain neutral. For 2023, TVCG EiC is Han-Wei Shen, and visualization-related A-EiCs are Shixia Liu, Ross Maciejewski, and Daniel Weiskopf.
First Round Reviewing
April 14 thru May 8
During this phase, both primaries and secondaries:
- Write a regular review of their assigned papers.
Additionally, the primaries:
-
Contact the APCs at an early stage if reviewers need copies of any related work that they cannot access (e.g., EuroVis submissions). The APCs can also communicate directly with authors if there are other inquiries. The sooner this engagement occurs, the better: please get in touch with the APCs sooner rather than later, if needed.
-
Remind all external reviewers to complete their reviews on time as the review deadline approaches. The OPCs will send one reminder as well, a week before the deadline.
As an IEEE VIS PC member, you are in a position to directly control the quality of the VIS papers program. Make sure that you stand behind the work you recommend for acceptance! Most importantly, this means that you should only accept high-quality papers that will contribute significantly to the VIS research community.
However, you should also be on the lookout for bold, creative, and potentially transformative work that may significantly extend the frontier of visualization research. While such submissions should be held to the same quality standard as all other submissions, it is expected that they may not be able to exhaustively (or perfectly) validate all of their contributions. Such papers often become easy targets for tertiary reviewers, so it is your job as a primary or secondary reviewer to use your experience and perspective to recognize them before they are summarily rejected. We do not want the conference to consist of work with the fewest flaws; we need to weigh the possible significance of the work and the degree of ambition alongside the extent to which it is described, compared, and validated.
If you have any questions in this regard about a particular paper, please either flag this in the confidential section of your review or write to the chair(s) directly. Your insight and ability to look beyond the surface is exactly the kind of guidance the paper chairs need from an experienced PC member.
Please see Additional Reviewing Guidance for further considerations.
Discussion
May 9 thru May 15
Primaries:
- As soon as possible, initiate discussion with the other reviewers to discuss whether the paper should be conditionally accepted or rejected for publication.
- Ensure that all reviewers contribute to the discussion about the paper. Consider contacting external reviewers directly if they are not responsive.
- By the end of the discussion period, write a summary review with a recommendation of conditional acceptance or rejection based on the reviews and the discussion. This summary review MUST explicitly spell out what the conditions are for turning a conditionally accepted paper into an accepted one. The authors need to know what is required of them as well as what comments they should consider but that aren’t required to address. If you and the secondary reviewer cannot agree on a recommendation for a submission, please document the discussion in the confidential portion of the summary review. If views have changed or consensus cannot be achieved, please let the papers chairs know the reason in the confidential comments. Summary reviews are due on May 15.
Secondaries:
- Actively participate in the discussion of whether the paper should be conditionally accepted or rejected for publication.
- Discuss with the primary reviewer to agree on a recommendation in the first review cycle.
Please keep the discussion anonymous by referring to other reviewers as R1, R2, etc., and remember to not sign your name in the discussions.
First Round Decisions
May 16 thru June 6
Please be available electronically for discussion with the paper chairs about the papers, your recommendation, and the review process. This is particularly important if any of your review processes are contentious or controversial. If you will not be available to field our questions, please let the paper chairs know in advance.
Second Round Reviewing
July 2 thru July 14
Primaries: For each of the papers for which you are the primary reviewer and that has received a conditional accept decision, please:
-
READ THE SUMMARY OF CHANGES AND REVISED PAPER, and consider whether the authors adequately addressed the required changes from your summary review. For most cases, deciding whether a revised version of a paper is acceptable is a decision that the primary can make on their own. If you have any doubt, feel free to initiate a discussion with the secondary. If you think that there remain any minor issues that authors can fix simply, PCS also allows you to communicate anonymously with the authors. If you do this, please let the paper chairs know so we can re-open the paper for author edits. In communicating with authors, be careful not to send email with your signature attached, to maintain your anonymity. Finally, if applicable please also check revisions to supplementary material.
-
WRITE A RESPONSE. First select a final decision for the paper in your review form for the paper. The options are “Accept” and “Unsure”. It is relatively rare that papers are rejected at this stage, so if you find yourself selecting the “Unsure” rating, please communicate with the APCs as soon as possible. Next, add text in the “Second round comments (public)” text box indicating that you have read the changed document and include any other responses you may have. Please note that you can still make requests for cosmetic fixes (e.g., typos, etc) for the camera-ready copy. However, you should not be requesting new substantive changes. Also fill in the form items on the acceptability of the supplementary material, and recommend a publicity image if applicable.
External Reviewing Instructions
Keywords Update
By March 31
You will need to update your expertise in PCS if you have not already done so after the 2020 rollout of the new keywords. Please read through the summary of the keywords, which includes instructions for reviewers on how to indicate their levels of expertise. Make sure to update your areas of expertise by the deadline above so that the program committee can take them into account during the reviewer assignment period.
-
Log onto the PCS system.
-
Go to the Reviews tab.
-
Click on “Provide areas of expertise”.
-
Select “VGTC” for the Society, and “VIS 2023” for the Conference/Journal.
-
Select your keywords.
Reviewing
Apr 14 thru May 8
High-quality reviews are critical to the quality of the conference program. Please familiarize yourself with the reviewing ethics and guidelines described in Additional Reviewing Guidance.
Discussion
May 9 thru May 15
Following the completion of the reviewing phase, the primary reviewer will initiate a discussion with all reviewers of a paper through the PCS system. During this time you are to engage with the other reviewers to come to a consensus decision on whether to conditionally accept or reject the paper. Please keep the discussion anonymous by referring to other reviewers as R1, R2, etc., and remember to not sign your name in the review or the discussions.
Additional Reviewing Guidance
Paper Quality
As an IEEE VIS reviewer, you are in a position to directly influence the quality of the VIS papers program. Make sure that you stand behind the work you recommend for acceptance! Most importantly, this means that you should only accept high-quality papers that will contribute significantly to the VIS research community. You should, however, also be on the lookout for bold, creative, and potentially transformative work that may significantly extend the frontier of visualization research. While such submissions should be held to the same quality standard as all other submissions, it is expected that they may not be able to exhaustively (or perfectly) validate all of their contributions. We do not want the conference to consist of work with the fewest flaws; we need to weigh the possible significance of the work and the degree of ambition alongside the extent to which it is described, compared, and validated.
Supplemental Materials
Authors may include additional material with their submission or on external repositories such as OSF.io. Suplemental materials could include, but are not limited to, data tables, code for generating results or analyses, videos, working demos, or appendices with additional results or findings. It is up to the judgment of individual reviewer to determine to what extent this supplemental material is discussed in their review, and to what level of detail they analyze the provided supplements or appendices. However, the reviewers should feel free to raise potential issues with the supplemental material in their review and request changes or additions, especially in the case where additional material is needed to verify the claims made in the paper.
Reviewing Ethics
All reviewers should make sure to follow the basic principles outlined below. As a primary or secondary, feel free to remind external reviewers of these as well as any additional resources you find helpful.
- Be timely: the conference has tight deadlines and late reviews are unfair to submitting authors as they reduce the time for discussion of their work.
- Protect ideas: the content of submissions should be handled as confidential information and should not be disclosed to close research colleagues or any wider audience.
- Avoid conflicts of interest: these include being directly involved in the work, co-author of one of the authors within the past 3 years, member of the same institution of one of the authors (or have been in the past year), supervisor/advisor of one of the authors, familial or emotional tie to one of the authors, or any other factor that would result in a biased review of the submitted work. If you discover a conflict of interest after the reviewing phase begins please contact the paper chairs immediately.
- Be specific and constructive: offer specific critiques that are addressable by the authors, point out concrete issues to fix in an encouraging manner, and substantiate your comments with evidence and references.
- See both pros and cons and find reasons to accept papers: have an open-mind or at least disclose your biases. Note that reviewing is not only about finding problems, but very much also about identifying the strengths of a submission. It is easier to find reasons to reject papers than to point out the contributions that they bring to the community. Make sure that you also comment on all pros that you identify and emphasize positive aspects to help surface impactful research to the community.
- Be tactful and polite: emotional rants or sarcastic comments have no place in a professional review. Never forget that you review the presentation of some particular research work, NOT the person who wrote it; reconsider whenever you are about to include a personal remark in your review. Ask yourself if you could read your review aloud in front of an audience including the authors.
Additionally, all reviewers are bound to the ethics guidelines of the IEEE VGTC. Please make sure you are familiar with these guidelines.
What is a Minor Revision?
IEEE VIS acceptances are conditional after the first round of reviewing, after which authors will be asked to perform a minor revision on the paper. While a few papers may not require any changes at all, this is usually very rare. In other words, reviewing in the first round becomes one of deciding which papers will be acceptable for publication after a minor revision. No paper is perfect; rather, your job is to identify the work that will have a significant contribution to visualization after the authors get a period of four weeks to address reviewer comments.
So, what is a minor revision? We rely on you to draw on your own experience to make this judgment. Don’t be close-minded, skeptical, or suspicious; be trusting and open to new ideas. Don’t expect perfectionism, reward pragmatism.
Here is an incomplete list of example changes suitable for a minor revision. You cannot require a specific result, but ask for details or another example, like
- Add missing related work (unless the missing work invalidates the contribution(s)).
- Reinforce arguments or improve reasoning (unless the central argument is fundamentally flawed).
- Adapt the exposition of the contribution and scope to better fit the research area (unless there is no conceivable connection to relevant topics in the area).
- Improve language, spelling, and grammar (unless poor language precludes understanding).
- Change poor visual design choices (unless they are central to the contribution).
- Correct issues with statistical tests (unless you suspect the new results will be radically different).
- Add or improve illustrations or images (unless this requires non-trivial algorithm changes or additional research).
There are many ways to fix an imperfect paper. Keep an open mind. Avoid prescribing a solution: explain the problem and allow the authors to choose how to address it.
Managing Resubmission Materials
As we all know, successful publication in a premier venue such as VIS sometimes takes time. It can be frustrating for authors to resubmit a rejected paper, only to have a new set of reviewers reject a paper for precisely the opposite reasons from the original rejection. To add some memory to the process, we now encourage authors to submit resubmission materials to their papers, including past versions, past reviews, and a cover letter with the authors’ response and a list of changes made.
Providing resubmission materials is OPTIONAL and at the authors’ discretion. Even if you have reviewed a paper before, you should not penalize that paper if the authors chose not to provide resubmission materials (or vice versa).
As with all supplemental materials, each individual reviewer decides how to treat resubmission materials. However, the paper chairs urge you to at least take a look at any resubmission material provided and write your reviews based on a full picture of the paper.
Assigning Scores, Requesting Revisions
Remember that the numeric scores are shorthand for a recommendation. Please check the descriptions for the numbers on the reviewing form and carefully justify your recommendation in your review. Short, superficial reviews do not help the paper chairs in making their decisions.
Note that submitting a neutral/borderline rating of 3 is acceptable, but not encouraged. If you do assign a 3, please specify in the revisions what must be completed within the (short) conference review cycle for the paper to become an ‘Accept’.