We are delighted to have Xavier and Kevin, co-chairs of the recent EMNLP 2016, to write a post detailing key difficulties that they faced in this past year’s conference organization. If you have a burning issue with how we (or ACL in general) puts together its scientific program, please contact us so we can consider a guest post from you. Now, on to our regularly scheduled program…
– Regina and Min
Thanks to Regina and Min-Yen for letting us participate in this great blog. We’re the program chairs for EMNLP 2016. Regina and Min-Yen asked us to comment about the issue of “Reject before review”, which will hopefully be helpful for authors.
At EMNLP 2016 we had to reject 46 papers without review, out of 1,185 initial submissions, which represents nearly a 4%. And let us stress that we had to, and we are sorry for it because we are aware of the hard work behind each submission. We are somewhat used to reject papers with reviews, based on their content, but we were not expecting having to reject so many papers before review, based on the form.
There are three main categories of violations to the instructions: non-anonymous submissions, over-length submissions, and violations to the official style files. The first category is rare, easy to validate, and is often correlated with formats which are completely different from the official ones. The other two categories are much more frequent than we thought. For over-length papers, we observed a continuum of extra content: one or two lines are probably caused by a last-minute-edit mistake, for one extra paragraph the authors perhaps thought we would be flexible, for a full extra column, page or figure, the authors did not take the instructions seriously. Formatting issues are the most problematic, because they include the cases of intentionally hacking the styles to fit more content, and in general the program committee does not have designated experts who can judge if the layout and formatting of a document comply with the instructions. Without technical guidance, it is very difficult to distinguish between good and bad cases. If submissions that have clear violations are kept, even if the violations are mild, the reviewers will inevitably be distracted by them.
Why were we strict about rejecting papers that violated the formatting instructions? Because we wanted to be fair to the authors who have followed the instructions. How would you feel as an author if you’ve spent precious time trimming your submission and sacrificing useful results and explanations due to space, and then discover later that your paper is being ranked against other papers that did not adhere to the same rules? That would not feel like a fair comparison, right?
Over the last years, the number of submissions in conferences increased, while the reviewing periods remain very tight. The role of program chairs is to ensure a high-quality and smooth review process, and this means we had to focus on the 96% of papers which were correctly formatted. It would have been too time-consuming to keep the remaining 4%. That would have been extra burden for the reviewers and area chairs. And this is why we had to make use of our right to reject papers before review.
If you are an author, please read carefully the submission instructions, and in particular always use the official style files for that conference (the ACL style files keep changing in subtle ways for every conference). Instruct your younger co-authors about the importance of this. Even if in the past the program committees were more permissive about certain hacks, the volume of submissions today does not give any room for distractions.
And, if certain instructions cause you a problem, you should contact the program chairs way before the submission deadline. Notably, at EMNLP 2016 we did not provide a template for MS Word, but we got many submissions that used the ACL 2013 MS Word template, which had significant differences that affect the amount of content. Whether to distribute or not templates for MS Word, and perhaps other word processors, is a matter of having enough volunteers in the organizing team who can deal with the extra work.
On another front, let us comment on a common pattern that directly challenges the double-blind nature of our review process. Note we do not want to start a debate about double blind reviewing (see, for example, http://hunch.net/?p=2656). These days, many people post their submission on arXiv.org after the submission deadline. Some authors even start posting on Facebook and Twitter about the importance of their newest findings. In most cases there are clear mentions that the paper is under review at a conference. The multiple submission policy of our conferences, including ACL 2017, says that “Authors must state in the online submission form the name of the workshop or preprint server and title of the non-archival version.”, which does not strictly rule out posting it after the submission. And many people do, which by design triggers notifications and email alerts throughout the network, precisely during the weeks where the paper reaches the reviewers and therefore the anonymity of the paper is specially relevant. The current policy says that “Reviewers are free to do what they like with this information”. At this point, we should perhaps recommend reviewers that if they really want to have an unbiased opinion of the papers they review, they should not check social media and they should unsubscribe from the daily arXiv email updates, just for a few weeks. Alternatively, authors could keep their submitted papers anonymous at all levels, just for a few weeks. Which one makes more sense?
To summarize: “Reject before review” is something nobody wants. It is extra work for program chairs, and it is a disappointing result to the authors. Just adhere to the instructions. Don’t take your chances. Everyone will be happier.
Xavier Carreras and Kevin Duh
EMNLP 2016 Program Chairs