The goal of this post is to share with you the outcome of our experiment on
area chair crowdsourcing experiment we conducted this year. There were
several reasons for trying the new model. First, the recent report from the
ACL exec about serious diversity issues in our community was an eye opener.
I believe that many of the concerning points in that report were related to
the standard practice of nominating people with whom we are personally
familiar. While there is nothing wrong with this concept per se, it leads to a
coterie of a privileged few that all know each other. AC is selected
this way tend to be very responsive, since there is a personal relationship at stake. But it is overly limited in its reach in finding qualified and passionate area chairs. I have to admit that this was my guiding principle for selecting ACs for EMNP 2011, which I co-chaired. As our community grows, this practice introduces significant selection bias. Second, as a
community we are blessed with many people who will be excellent ACs. The
only thing that was missing is the formal means for bringing these
candidates to the attention of the program chairs. While emailing the program committee is one solution, in practice very few people exercise it. In fact, we only received
two emails with AC nominations before we issued the call. These among other
reasons motivated us to start our experiment.
I am happy to report that the results exceeded our expectations. We received
192 nominations. From this strong pool, we selected 44 (22%) candidates, which
constitute 72% of our current AC chairs. In addition, Min and I nominated additional area chairs, depending on the need of specific areas. Our main criteria for selecting area chairs was technical excellence and reviewing experience. We also aimed to take into
account geographical and gender diversity, degree of seniority, academic vs
industry background, and prior AC experience. It was a complex optimization
problem to solve, but we are very satisfied with the results. The attached
tables illustrate the distribution of ACs according to these different axes.
Thank you all for participating in this experiment! We are glad with outcomes, but let’s
wait until the show is over to judge its impact on the scientific program. We would like to thank all the community members who put the nominations, and to express our gratitude to area chairs in advance for the hard work that comes in coordinating the scientific program!