A public letter to our prospective Area Chairs – 6 key changes

Dear invited Area Chairs:

[TL;DR: actually you need to read the whole post.  So no summary! ]

To increase transparency of the process, we are making this an open letter, which we hope you (and perhaps the general public) will comment on to improve the reviewing process.

First, thank you for considering the charge of being an area chair (AC) for ACL 2017!

We are writing this letter to you to inform you of several important changes to the ACL 2017 organization that will affect how you recruit reviewers and manage the deadlines for the conference.  Since these changes could potentially affect whether you decide to take up the decision to be an AC, we decided to write this post first.

Major highlights of the changes to the ACL 2017 reviewing process:

  1. The deadline for short and long papers is joint.  This affects the number of submissions and also the coordination and reviewing load.
  2. The initial review period is shortened to two weeks (13-27 Feb 2017).  This affects how responsible reviewers need to be to ensure that the reviews are completed in time.
  3. Authors will have a textbox in the author response form to talk directly to ACs, only if needed.  This will also affect ACs’ workflow during the author response period.
  4. We will use fairly broad areas and allow reviewers to review for multiple areas.  This hopes to address the problem of “gaming” the selection of areas in the review process.  However, this increases the coordination load among ACs assigned to manage their area.
  5. We are planning to use the Toronto system (used in NIPS) for assisting with assignment of papers to reviewers.  This affects the workload of reviewers, as they will need to provide details on their past scientific work that will help build their reviewing profile to best match them to papers of their interest and qualification.
  6. There will be a meta AC per area to serve as a single point of contact for making recommendations for the area.  This AC will be a senior member of the community that accepts this extra responsibility for organization and coordination matters.  All ACs are, by default, assumed to have equal share in the scientific decisions on all matters for their respective areas.

Let’s look at each of these changes one by one, in more detail.

  1. Joint deadlines.  With our community and the number of scientific submissions growing, we believe that ACL will attract a record number of submissions.  We believe this is more so as in 2017, ACL will act as a both NAACL and ACL together, being the rotation where ACL is back in the Americas.  This will put a burden on finding enough high quality reviewers to execute the review process faithfully for each submission.With staggered deadlines that have been the feature of ACL in recent years, this stretches out the review period and encourages more paper submissions.  In contrast, we chose a joint deadline to try to lessen the workload per reviewer and have the reviews over a shorter time span.  We believe that the joint deadline will not have much impact in lessening the number of lower quality submissions, but it may affect short paper submission numbers.  This change will affect the demands on your time in coordinating reviews more carefully.  We see the importance of emphasizing this need to be communicated carefully during the reviewer recruiting process.
  2. Shorter review period.  After observing reviewing in many venues, our conclusion is that the length of the reviewing period doesn’t greatly matter for many reviewers — reviews tend to be completed very close to the deadline.  A shorter reviewing period allows for more time for the other activities, especially meta reviews and considering author responses.We understand that reviewers do schedule their availability such that they can complete the reviews early, especially if they are not available close to the deadline, and that such reviewers would benefit from a longer review period.  However, we believe this to be the minority of cases, and we firmly believe that the responsible reviewer needs to be involved in the dialogue that ensues from the initial review — the author response, the discussion with peer reviewers and the ACs, all the way until notification (i.e., from 6 Feb-30 Feb Mar).
  3. Direct to AC text box on author response form.  This is a new pilot that we are conducting for ACL 2017, in which the author response form will have a text box for the authors to write about their concerns about any misunderstandings of the reviewers.  This idea originated from the problem where deserving works were rejected by reviewers who may have misread or misunderstood the paper and who might not be receptive to the author response mechanism.  We will try to ensure that this facility is only used in the case where the authors feel that the reviewers’ comments are out of line, and need to take the dialogue directly to ACs.
  4. Broad Areas.  We have heard reports where the review quality in different areas can differ drastically, such that the authors feel the need to strategically select reviewing areas.  Occasionally this can happen when having many, small areas that have fined-grained differences that may not matter for most submissions.  In such cases, the decision of which area to assign a paper to can be difficult and cause bias. By having larger, broader areas, we hope to level the review process to ensure a more uniform review quality.Of course, this comes with a cost — areas will have to be monitored with more area chairs, which increases the coordination load among ACs to get their work done on schedule and to resolve conflicts within a reasonable timeframe.  Again, some prescheduling and coordination work ahead of the reviewing period will be important to make this work, and the devotion of all ACs of an area to agree and adopt the schedule.
  5. Toronto system.  We are planning to adopt the Toronto system for helping to assign reviewers to papers.  The system, used in the Annual Conference of the Neural Information Processing Systems (NIPS), suggests reviewers based on an analysis of their published works (i.e., language model with LDA).  You as part of the AC team can decide which recommendations to approve of, if any.  It will substitute the current system where the matching is done on keywords alone.   Based on feedback from the NIPS community, it has enhanced their reviewing process by providing a better fit of papers to reviewers, increasing the overall conference quality.  To familiarize yourself with the system, please read the documentation on it:http://www.cs.toronto.edu/~zemel/documents/tpms.pdf
    However, it does require work to use.  Aside from work on our side to ensure it can be integrated in a timely fashion for ACL 2017, it requires reviewers to provide a webpage where the reviewer’s publications can be found, downloaded and analyzed.  This affects the reviewers’ workload, as they will have to do some work to provide the necessary information for the recommendations to be of use.  The good news is that the work is done one time only and when the system is used again in future ACL conferences, the work for any repeat reviewer is negligible (if any).  This information needs to be carefully communicated to prospective reviewers.

    This is a major change that we hope will continue to feature in ACL conferences and a pilot experiment backed by the ACL Executive board.  It may not influence your workload much, but is important to disclose to you.  Its usage is not finalized yet, because it depends on the technical team at Softconf to ready the system for integration.  We plan to discuss the system in more detail in subsequent posts when the system is close to ready for use.

  6. Meta AC.  With the expanded pool of ACs needed to handle the expected workload for ACL 2017, we will be appointing an initial meta AC for each area as well.  This will be a senior member of the community that will be the main point of contact for making recommendations for the area, and will organize a discussion for papers among other ACs.  The ACs for an area may decide to change the meta AC to another AC in the area — it’s not binding.
    We assume that all ACs in an area carry the same weight in the scientific aspects of the program.  To be clear, the meta AC role is for contact and organization only.  We do not mean for a meta AC to be assumed as a head AC, whose role might be construed to mean moderating fellow AC decisions.  As PC co-chairs, we will take that responsibility upon ourselves.

There are more details that are important for ACs but these are the major changes that we think will affect whether you accept the charges of being an AC.  The above changes show that ACs will have a much more involved role in ACL 2017 in deciding the technical program, and much added responsibility.  Most importantly, we need all ACs to be responsive during the critical time frame of 6 Feb – 30 Feb Mar.

If you can agree to these responsibilities, and dedicate time during this period to be able to join us as an area chair, we will gladly accept your help in making ACL 2017 a success.  However, we completely understand if you cannot make the time commitment that will be needed for the expanded AC role that we are asking for.  If you agree to these terms, please do respond to our personal invitation that will be emailed to you in the next few days.

We have also created a Google calendar containing the important pertinent dates.  This is a global calendar for all ACs and will be updated when changes to events and deadlines occur.  Where possible, please ensure you have access to the live calendar to keep apprised of relevant ACL 2017 AC events.  Here’s the address:

https://calendar.google.com/calendar/ical/arq0ig9b7dvhvpnv1n93bbluv0%40group.calendar.google.com/public/basic.ics

We look forward to your favorable decision!

– The PC chairs

18 thoughts on “A public letter to our prospective Area Chairs – 6 key changes

  1. I’m not a prospective area chair, but chances are I will be a reviewer for ACL for many years to come… I find all these changes very positive, except for adopting the Toronto system. My experience at NIPS has been that this system favours assigning me papers based on my PAST interests, rather than the current ones. This has made me a lukewarm reviewer of papers in areas I might have expertise in, but I’m not currently interested in. Also, I wonder if this mechanism favours the development of very specialized sub-communities, instead of encouraging cross-pollination across fields of CL… My two cents, of course.

    Liked by 1 person

    1. Hi Marco, thanks for the great thoughts on the subject! We hope others will comment as well (here, or on Facebook, Twitter). I believe the Toronto system can be set to discard older papers that you are no longer interested in, but that does require some work on the part of reviewers.

      In either case, the Toronto system will likely be used to score papers to prospective reviewers, but the ultimate assignment responsibility will still lie with the Area Chairs — the system is there just to help give an alternative viewpoint aside from bids alone.

      As for cross-pollination, we do welcome that, and reviewers can express their interests with their bids (I myself like to read papers outside my field for this reason too). The problem that we hope to address is to better find expert reviewers who can appreciate the contribution and critique appropriately. Many of us have had that “what was the reviewer thinking?” question when reading reviews that don’t seem to align to our perspective of our own work.

      We do appreciate the time that it takes to provide feedback and are happy to receive more. The more feedback the better the discussion for all and a more well informed ACL 2017 reviewing team we’ll have aboard!

      Like

  2. I find the changes very positive, and like the open discussion format. I’m posting my comments to the invitation that I got (together with Regina’s reply):

    – will the Toronto system be added to Softconf (or will we switch to the NIPS system?) (I’d hope the former)

    We will add Toronto system ranking to the Softconf system

    – can reviewers upload a simple bibtex for it?

    We are currently exploring different options.

    – do you know already who will be the senior AC in our area?

    we will decide once we hear from all the people we sent AC invitations.

    Like

    1. We plan to make a separate post about the Toronto system very soon, but the gist is that reviewers will need to create an account with the system and provide a URL where PDFs of their scholarly works can be found. The system can pull down, analyze and build a topic model from the papers. Alternatively, you can add papers one at a time, by giving an accessible URL or uploading each individually.

      With the system’s account, you may review which papers are found by the system, and exclude papers found by the system in batch mode.

      You can find out more about the system here:

      (This is an outdated link; use the new link instead http://papermatching.cs.toronto.edu/webapp/profileBrowser/about_us/)
      http://torontopapermatching.org/webapp/profileBrowser/login/

      Liked by 1 person

  3. Hey ACL 2017 organizers!

    First of all, hats off for the changes you’re introducing!

    Have you considered asking the reviewers to list conflict domains? The guessing game of “Should I click COI or not?” is getting a bit tedious these days. 🙂

    Thanks!

    Like

      1. Hey, by “conflict domains” I meant, e.g., a list of (URLs of) current and past affiliations, similar to whwt they do for NIPS, AISTATS, etc.

        Liked by 2 people

      2. Hi Zeljko:

        We’re told by START’s Rich Gerber that this is already a functionality in START. You can go to your user profile and add the domains that correspond to your prior affiliations. These will be then used to mark domains for which you have conflicts when bidding for papers.

        Hope that helps!

        Like

      3. Dear Zeljko,

        We will check with Softconf people whether the current reviewing system supports this functionality.

        Thanks for bringing it to our attention.

        Liked by 1 person

  4. Dear organizers,

    thanks for putting a lot of thought into the reviewing schedule and processes!

    The one thing I’m worried about is shortening the reviewing period. I’m definitely one of that minority of people who prefer to stagger their reviewing rather than finish it all just before the deadline. With a reviewing period of just two weeks, I have two options: Either ask for a drastically reduced workload (which increases the number of reviewers the ACs have to find and herd) or submit superficial reviews. Reading the submissions carefully and giving constructive feedback is very important to me, and I’m worried that we’re encouraging the opposite by requiring reviewers to finish five reviews within two weeks (at a time where many of us are wrapping up semesters).

    I can imagine the ACs will profit from more time to make acceptance decisions, especially with so many expected submissions, but in my experience, discussions among the reviewers are finished very quickly and I don’t think we need more time there.

    Like

    1. Dear Ulrike,

      We are planning to assign each reviewer up to 4 papers. I hope that for most of us two weeks should be sufficient to review four papers. Based on my past experience as an area chair, most reviewers open
      their assigned papers in the last few days before the deadline. Therefore, this change will have minimal impact on those reviewers. But I see your concern.

      We will take a chance this year with this new change and report the outcomes to the community.

      Like

  5. Many interesting ideas. Congrats to program chairs for trying some innovative schemes, and all the best for their success. I also agree that most reviewers finish assignments at the last minute, but some don’t. So, the 2-week strategy might work ok (at least, statistically speaking 🙂 ).

    I am concerned about just 1 day for area chairs to finalize reviewer assignments. 2-3 days would have been nicer, since ACs are often in different time zones and coordination can be tricky.

    Finally, my biggest challenge in doing NLP decisions (especially compared to human computation, AI planning and other AI fields that I am a part of) is the lack of reviewer participation after the reviews are in. Very few reviewers seem to respond to author rebuttals and even fewer are willing to give their first assessments a serious rethought. This makes it difficult to get a discussion going, and many area chairs end up resorting to taking executive decisions overruling the original reviewers based on (often) a quick glance of the paper. This can lead to author dissatisfaction.

    We should try to devise some mechanisms to encourage reviewers to participate in discussion. At the least, we should explicitly explain to reviewers that their work does not end at submitting reviews, they are also responsible for ensuring that a healthy debate and discussion happens and a consensus (whenever possible) is reached. As a mechanism, we could insist that each reviewer *has to* fill a new section called “Response to Author Rebuttal”, which has enough details and, when necessary, summarizes the discussion for the benefit of the authors. This will result in more transparency and likely better author experience, as well as overall better decisions.

    All the best for the conference!

    Like

    1. Hi Mausam: Thanks for your comments. Certainly we want to give sufficient time for ACs to set the right reviewers on to each paper. We hope ACs will be proactive and make partial determinations before bids are finalized. We may consider revising the schedule to give a bit more time.

      We hope that between bidding and the Toronto system’s usage (if that materializes from its hardware issues) will make the assignment job a bit easier. We’ll likely also try to follow the practice of (by default; reviewers could then opt to change their area) assigning reviewers strictly to one area so it is easier to manage whom can be assigned.

      Yes, we are most concerned about the dialogue process in the reviewing cycle. In the schedule, there is more time for discussion (and as many have noted, less time for the initial review) and we hope to change our reviewers’ mindset that there needs to be more involvement in the review process than just the initial review alone. We need your input on how this change might come about too!

      Like

  6. Hi, Regina, Min-Yen, Chris,

    Thanks for putting so much thought into this important process! One issue that has made reviewing for ACL so much better than most conferences (definitely better than NIPS) has been the ability of ACs to *easily* change papers’ assignments manually, so they can make sure that they personally know the reviewers of each paper. I think that this is essential if we want to guarantee high quality and timely reviews; I hope that this will stay.

    Liked by 1 person

    1. Hi Dan:

      Definitely we would like to keep that feature of the paper routing process. Regina and I are working on these parameters within START now, actually, so hopefully it’ll be seen by the area chairs in the system soon.
      Thanks so much for this important comment!

      Like

  7. Min/Regina – Are authors going to be given any guidance as to what makes a good short paper? I’ve just finished reviewing short papers for EACL, and none of the authors seems to have a clue as to what a short paper should look like. All I’ve seen are long papers from which useful detail has been cut out, to reduce their size to four pages (plus references).

    If we’re going to solicit short papers (and I’m happy that there is only a single call), we should provide some guidance (if of course, we have some agreement on what a good short paper should look like).

    Best, B.

    Like

Comments are closed.