NAACL-HLT 2021 Instructions for Reviewers

Thank you for reviewing for NAACL-HLT 2021!

In order to ensure the quality of reviews, we would like to share with you the following instructions for reviewing papers. Please read these instructions carefully before you start working on your reviews.

Reviewers were assigned 3-4 papers on average. Since we had a joint long and short paper deadline, please be mindful that you may have a combination of long and short papers to review.

Confidentiality

Please note that the content of any submission to the NAACL-HLT 2021 conference, any discussion of such submissions as well as the names of participants in such discussions are confidential.

The Review Form

We adapted the review form from ACL 2020 and EMNLP 2020. The review form consists of four main sections.

1. In-Depth Review: This section is for you to give your overall assessment of the paper and to provide evidence to support your opinions. There are 3 subsections:

  • The core review: This is the most important part. It should include your view of the main contributions that the paper intended to make and how well it succeeds at making these contributions. From your point of view, what are the significant strong and weak parts of the paper and the work it describes? This could be a 2 paragraph (or longer) essay and/or bullet points. Remember to describe how the work advances the state of knowledge in computational linguistics and/or highlights why it fails to make a sufficient contribution.

  • Reasons to accept: please briefly summarize from your core review the main reasons why this paper should be accepted for the conference, and how the NLP community would benefit from it. You may refer back to your review to provide more context and details.

  • Reasons to reject: please briefly summarize the main reasons that this paper cannot be published and presented in its current form. What are the parts that would need to be improved in order to advance the state of knowledge?

2. Questions and Additional Feedback for the Author(s): Since we will have an author response process, for questions you would like the author(s) to respond to during the response period, please include them here. This is also the place for you to give suggestions to the authors to help them improve the paper for the final version (or a future submission).

3. Overall Recommendation: This section includes your 5 subsections, covering reproducibility checklist, ethics review and justification, overall recommendation, reviewer confidence and best paper award recommendation and justification.

  • Reproducibility: Following EMNLP 2020 reviews with some updates, this year we introduced a reproducibility checklist in an effort to increase reproducibility of the research work in NLP (please see NAACL-HLT 2020 call for papers for reproducibility criteria). In the review form, please answer the following two questions:

    • “How do you rate the paper’s reproducibility? Will members of the ACL community be able to reproduce or verify the results in this paper?” Scores of 1-5 are used to assess this aspect. The detailed explanation for each point level is provided in the review form. N/A may be used for papers that do not include empirical results.

    • “Are the authors’ answers to the Reproducibility Checklist useful for evaluating the submission?”. Three choices are provided for this question (very useful, somewhat useful, not useful). Note that this question is for us to collect feedback regarding the usefulness of the reproducibility checklist, and is not about evaluating the paper itself.

  • Ethics Review: This year, authors were allowed extra space after the 8th page (or the 4th page for short papers) for a broader impact statement or other discussion of ethics. While an ethical considerations section is not required, papers working with sensitive data or on sensitive tasks that do not discuss these issues will not be accepted. Hence, we would like you to flag papers for ethical concerns, and these papers will be further reviewed by an ethics committee. Please read the ethics FAQ for more guidance on some problems to look out for and key concerns to consider relative to the code of ethics.

  • Overall recommendation: Here you are asked to synthesize the above and come up with your own recommendation for the paper.

    • Like ACL 2020, we have used a 5 point scale with a half point increments. The detailed explanation for each point level is provided in the review form. These numbers are just a concise way of expressing your overall opinion and relative importance of the factors mentioned above.

    • Different from ACL 2020, we are allowing a rating of 3 (ambivalent). Please try to take a stand on whether the paper is above or below the borderline, e.g., by selecting 2.5 or 3.5. However, as much as we would like you to do that, if you think this is indeed a borderline paper or you are not able to decide, you should use 3.

    • Decisions will be made not just on the scores and certainly not on average scores, but will also take into account the whole review, reviewer discussion and Area Chair meta-reviews and recommendations. However it is important to align your recommendation with the reasoning given above, so that authors will be able to understand the motivation for the recommendations and how decisions were arrived at.

  • Reviewer confidence: This section should be used to inform the committee and authors how confident you are about your recommendation, taking into account your own expertise and familiarity with this area and the paper’s contents.

  • Recommendations for awards: This part is for nominating papers for the best paper award. NAACL-HLT 2021 will have best paper awards in the following categories: Best Long Paper, Best Short Paper, and Best Theme Paper. Please be open minded and feel free to nominate good quality papers even though they may not be the typical types. These can be a survey paper, an opinion paper, a paper about resources and datasets, a paper for low resource language, an analysis paper, etc. A committee will evaluate best paper candidates, and we would like to have a wide variety of paper types in the candidate pool, not just vanilla empirical research papers. In addition to the best long paper award and best short paper award, we will give several outstanding paper awards. Note that unlike in previous conferences, if you recommend a paper for an award, the authors will see your comments.

4. Confidential Information: Your answers to questions in this section will not be shared with the authors. Here we ask you about confidential comments to the area chairs and/or PC chairs.

5. Author response: There will be an author response period. It is important for you to check whether author responses have cleared up your questions or misunderstandings. This may influence your overall recommendation and the core review. If that’s the case, please update your recommendation and review accordingly (and state in your review any new decisions you made so the Area Chairs are aware).

Supplementary Materials

Supplementary materials are allowed as a stand-alone document uploaded as an additional file. Supplementary materials are, as the name suggests, supplementary, and you have no obligation to read them. You should treat them like other citations in submissions that may be helpful in understanding background or details beyond the scope of the paper itself.

However, as noted above, given the new requirement for reproducibility, authors may provide additional information about their datasets and experiments in Appendix, and attach a zip file with resources such as code and data. Please take some time to check those, if applicable. Uploading data and/or code alongside paper submissions is preferred over supplying a hyperlink. The latter could violate double-blind review practices.

Secondary Reviewers

As in most previous NLP conferences, you are allowed to solicit help from others. However, when it comes to writing the final review and giving the final scores, we expect you to take the secondary reviewer’s review and rewrite it using your own words and adjust the scores when you see fit. Essentially, the final review should reflect your own opinions about the paper, and you need to be able to justify the opinions you present in the final review.

Format of Submissions

The program chairs and area chairs have tried to identify submissions that violated our formatting guidelines and have desk-rejected those submissions. Therefore, you do not need to worry about formatting issues with the submissions assigned to you. However, if some formatting issues were missed and you think the paper violates the format guidelines, please contact your area chairs or PCs. Otherwise, the paper format shouldn’t be used to down weight evaluation of the paper.

Important Dates

Your reviews are due Tuesday, Jan 12, 2021 (11:59pm anywhere on Earth). Please note that there is a reviewer discussion period from January 26 to 30 after the author response. Your duties are listed below. Do not leave reviewing to the last minute!

  • December 18, 2020 – January 12, 2021: Review Period
  • January 20 - 25: Author response period
  • January 26 – 30: Reviewer discussion period

Review Advice

Please read our post on writing high quality reviews that we copied from EMNLP 2020. The post will be published before the author rebuttal period so that the authors are aware of the guidelines and can reference them in their rebuttal. ACs will be instructed to flag poor reviews, ask reviewers to revise their reviews or provide objective reasons to justify your positions.