ArXiv and double-blind reviewing revisited

jn-160410Joakim Nivre
Uppsala University
President of the ACL

It is with great interest that I have followed the discussion in connection with ACL Vice-President Marti Hearst’s post last week on the conflict between pre-acceptance arXiv publishing and double-blind reviewing. Just to recap, most people agree that the former has a number of advantages such as fast and easy access, “time-stamping” of results and ideas, and an open feedback and revision process. At the same time, posting papers on arXiv prior to acceptance for publication clearly undermines double-blind reviewing, which in a number of studies has been shown to promote fairness and prevent bias in the selection of papers.

I draw two conclusions from the discussion so far. The first is that there is no simple or obvious way out of the dilemma, which just confirms what I have thought for a long time. The second is that opinions are quite divided in the community, something that was perhaps less clear to me previously. Some people consider double-blind reviewing worth saving at almost any cost because of the proven advantages. Some people feel just as strongly about the merits of fast and open dissemination of research results using arXiv. And some people are actively looking for compromises to save the best of both worlds.  

The ACL Executive has discussed this issue on and off since I joined the committee two years ago, and it has become clear that we need to do something about it. We need both a long-term and a short-term strategy. We need to work long-term because it is a hard problem where a viable solution is unlikely to be found quickly, and because it is part of a larger complex of issues having to do with how we want our models of publishing and reviewing to evolve in the future. In addition, being a relatively small community, we probably need to work in synergy with neighboring communities to find a viable solution, especially if we want to impose changes to existing systems, and this will also take time.  

At the same time, we need to find a workable short-term solution. Because another conclusion I draw from the recent discussion is that the policy currently adopted by TACL and most of our conferences is not really working. According to this policy, a paper can be submitted for double-blind review despite being posted (non-anonymously) on arXiv as long as the authors declare this. This may have been an appropriate policy when arXiv was a marginal phenomenon, but to most people today it appears either confusing, ineffective, or simply unfair (because we keep rejecting papers whose anonymity has been compromised in other ways). And a policy that lacks support from the community is not going to work in the long run.  

So what is the ACL Executive planning to do about this? At our recent winter meeting, we decided first of all to gather more information by organizing a membership poll. The details are still to be worked out, but the poll is likely to contain questions about your current practice – to what extent you post papers on arXiv prior to acceptance, for example – as well as about your views on possible alternative systems and policies – such as whether you are willing to sacrifice pre-acceptance publication to save double-blind review or vice versa. While you are waiting for this poll, you may want to check out an earlier poll organized by Kyunghyun Cho:
(http://www.kyunghyuncho.me/home/blog/toarxivornottoarxiv).

We also have plans to organize a special session at ACL 2017 in Vancouver to discuss this and related issues. Except for our regular business meeting, we normally do not have meta-sessions at our conference, but we feel that this is important enough to the community to set aside a bit of time somewhere in the program. Again, the details are still to be worked out, but one of the things to discuss will surely be a new and better policy that can be put in place for next year’s conferences. If you have ideas about how this session should be organized or would like to contribute, please get in touch with me or any of the other members of the ACL Executive – or simply make a comment on this post!

Based on the input that we gather from the poll and the meta-session, and hopefully with a better short-term policy in place, we should be in a better position to map out a long-term strategy towards a viable model for publishing and reviewing in our field. Deciding how to deal with the conflicting pressures of arXiv publishing and double-blind review is only one of many strategic decisions that we have to face, but it is clearly a very important one and we look forward to getting your input on this. Because a long-term strategy can only be successful if it is well anchored in the community. 

15 thoughts on “ArXiv and double-blind reviewing revisited

  1. Would it be possible to ask arXiv to add an option to keep a paper hidden until a given date ? It would be registered and made available just after the answer date of the targeted conference.

    Like

    1. This is an interesting idea, brought up also by Jason Eisner in response to Marti’s previous post. I have no idea how feasible it is, but it seems worth looking into.

      Like

  2. This has probably been suggested before. What if arXiv had an option to “hide” authorship of submitted paper but would reveal the authorship “on request” from anyone? So if someone casually visits / reads the paper they won’t see the authors’ details but if someone explicitly requested for it to cite the paper then they would immediately see the authors’ details.

    And of course the authors would have the control to make the authorship visible to all whenever they choose to. Would that at least partially solve this situation?

    Like

    1. I think I was proposing something different from the previous comment. I am wondering if arXiv could support a “soft anonymization” state for papers during which the authorship details won’t be visible by default BUT anyone can press a button on the site to immediately reveal the authorship.

      Reviewers could then be instructed to not reveal the authorship for papers they are reviewing even if they encounter them on arXiv feeds. On the other hand, other researchers who may want to cite the arXiv paper could use the reveal option.

      This obviously doesn’t take care of the scenario in which authors publicize their work outside of arXiv on social media.

      Just my 2c.

      Like

  3. There are at least two issues with the huge increase in use of arXiV and only one of them is to do with double blind review. The other and more pernicious is the citation of arXiV postings before and increasingly in the absence of any peer review at all. Peer review, though flawed, is critical to ensure the validity of our science.

    Liked by 1 person

  4. First off, thanks for the thoughtful and considerate discussion!

    I am much in favour of double blind reviewing, but I do see that publishing things on arXiv can speed up matters.
    The discussion so far has focussed on the problem of truly blind reviewing when the reviewer can be expected to stumble on the names of the authors just by searching for background work.
    Next to this discussion, I would like to raise a suggestion to somewhat discourage publishing non-peer-reviewed online in the first place.

    How about we only allow arXiv papers to be cited within one year of their being published on arXiv?

    If the paper is published in a peer-reviewed venue in the meanwhile, fine; it can be cited forever as always.
    If not, its “window-of-citability” closes after one year.

    This way, research might still benefit from fast publication cycles. But if the non-peer-reviewed work does not make it in the end, it can only gain so many citations in a year, and you can never get a high h-index by only having non-peer-reviewed papers.

    I do realise, btw, that this doesn’t in any way deal with anonymity issue.
    And I would like to add my +1 to the idea of arXiv supporting some form of anonymizing, regardless of this one-year-citation idea above.

    Like

  5. Hi Joakim,

    another consideration that I would like to see discussed (I’m probably not going to be in Vancouver) is: to what extent do ArXiv papers count as part of the canon. Or to put it the other way around: Is not citing a relevant paper that’s on arXiv a crime that’s punishable with a low “awareness of the literature” score?

    My own opinion is pretty clear: if it’s not peer-reviewed and published, it shouldn’t be required to cite.

    My justification is that I think handling things differently would lead to a very volatile “first past the post” situation: everyone would constantly have to scour arXiv for results by other groups, people would have to change their stories at the last time to avoid collisions with arXiv posts that were published yesterday, and so on. In short, it would introduce a huge element of stress into a field that’s already moving very much into this direction — at I think that’s a very bad thing.

    I realize that this puts into question the whole purpose of the arXiv business though…

    At any rate, what we need is guidelines one way or another, because right now different authors and reviewers treat this question very differently and I know that papers were rejected because of mismatches between expectations..

    Liked by 1 person

    1. This is exactly one of my big concerns regarding arXiv (I mentioned it in other discussion):

      – Reviewers are starting to do exactly this: “you haven’t cited arXiv paper this-and-this”, which I find unfair, especially if the given arXiv paper was submitted only a few days before the conference deadline.

      – Somebody mentioned the constant change “so that no version of a paper is ever final” as one of the great features of arXiv.
      How that is supposed to work if citing arXiv papers becomes obligatory?

      Say, I have some idea. Then, in addition to the conference and journal papers, I have to check arXiv papers, too. I find one related paper on arXiv, but my ideas are still novel, great.
      So I work on this, write an article, prepare to submit it to a conference, and then, one day before the deadline, I have to check
      a) not only if in the meanwhile some new arXiv related papers have appeared (very stressful),
      b) but also if the previous paper has changed (horribly stressful).

      Like

  6. Public availability is distinct from anonymity. Both are good for sound science.
    Since arXiv is a preprint service, such papers should be anonymous until peer-reviewed acceptance.
    Anyone wishing to contact the authors could do so either privately via a Craigslist-style email alias, or publicly via GitXiv.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s