Hello VIS community,
Since IEEE VIS 2020 we use a new set of keywords in several submission processes. The purpose of this blog post is to explain how the keywords matter to program committee members and to offer a little help in choosing keywords. A first blog post already covered how keywords matter for paper authors.
Read this text if you have been selected as a PC member for IEEE VIS. You should be interested to get a good set of paper suggested to you that you are interested in and maybe even excited about reviewing.
Expertise Rating
Your first contact with the new keywords will be the expertise rating interface.
Your keyword selection interface will look like this:
The page will explain in more detail what the categories “expert”, “competent”, “limited”, and “none” mean. You will recognize the new keywords used at VIS by the additional explanations added as a second line under each keyword.
Rating your expertise in this interface will be important later for the bidding interface. Do it as truthfully as possible. Paper chairs might also peak at your expertises when they need to make changes to paper assignments.
Note that setting your expertise to “expert” does not mean that you will by default or automatically be assigned papers related to the keyword. This is particularly important to remember for the large domain keywords shown above. Of course nobody can be an expert in all visualization applications to “Social Science, Education, Humanities, Journalism, …” for example. However, if you are interested in seeing recommendations in the bidding interface for education papers and you consider yourself an expert in visualization for education, then select “expert” for this keyword as those papers will show high in your list. You do not need to bid on the journalism, intelligence analysis, etc. papers that will also show up there - remember that the numbers of papers that have chosen this particular keyword is likely manageable.
Bidding
The second time you’ll be confronted with keywords will be during bidding. We already described bidding in Part 1 of the keywords series. If you’ve never participated in it, take a look at this post first.
In the new area model you will have a lot more papers in your list, therefore you will not be able to look at all abstracts. Keywords will help you to filter the list to papers of interest through a) the matching score calculated between your expertise and the keywords selected by paper authors and b) through a search/filter interface on the bidding page.
Paper Assignment
You will not be part of the paper assignment process but it is worth explaining how both expertise rating and bidding influence which papers you will be assigned.
Paper-to-PC member assignment is a multi-step process at IEEE VIS:
- The initial matching of PC members to papers will be done primarily based on bids.
- Then paper chairs will check this initial assignment. If it seems sensible, they will keep it and otherwise adjust it. They will try their best to make sure that you receive papers that you wanted to or were willing to read, that match your expertise, while balancing conflicts of interest between assigned PC members and paper authors.
Step 1 on the process, however, is automatic and based on your bids - and NOT on automatic keyword matching. This is a change in the process from some previous years. Therefore for you the purpose of keywords is to help you navigate and filter through the large set of papers you will have to bid on.
What strategies should I use for rating my expertise?
Be truthful in your selection of expertise ratings. You want to bid on papers that you know something about and selecting the right keywords as “expert” will help here. You have been selected to be on the PC because the paper chairs believe you are an expert in some area of VIS. This means that you should rate yourself expert on several keywords; less than 3 is probably too few and more than 8-10 is probably too many. Note that you can still rate yourself “expert” if you are an expert in one of the subareas of a larger keyword: “Social Science, Education, Humanities, Journalism, …”, for example.
What strategies should I use for bidding?
In the new area model you will have up to 500 papers in your list and you will not be able to look at all abstracts. Therefore start by setting all papers to “reluctant”, then start organizing your list. Sort your interface by the match score, use the search box to find papers with keywords that interest you, use the filter mechanism to only look at papers with keywords you rated yourself “expert” on, and look at papers submitted to areas that match your expertise best. Also look for papers where authors made mistakes in selecting keywords - in particular those with a matching score of -1. Some hidden gems might be residing there.
- Select all papers you would be interested in reviewing to “want”. The more the better. You are much more likely to receive a paper from this list than from any other.
- Select all papers you could review because you have the expertise to “willing”. Many more would be better.
- Leave everything else as reluctant.
- You should have bid around 40 papers together in the “want” and “willing” category.
Keep in mind that the fewer papers you select the more likely you will receive one from your “reluctant” pile - and this pile will be huge and random. So do take bidding seriously and select as many papers as want or willing as you can. If you get a “reluctant” paper assigned paper chairs will look at the assignment to make sure that you could do the job but you’ll make their life significantly easier if you selected many papers that they could consider to give you instead, should a fit indeed be bad.