Language technologies and their applications are an integral and critical part of our daily lives. The development of many of these technologies trace their roots to academic and industrial research laboratories where researchers invented a plethora of algorithms, benchmarked them against shared datasets and perfected the performance of these algorithms to provide plausible solutions to real-world applications. While a controlled laboratory setting is vital for a deeper scientific understanding of the language problem and the impact of algorithmic design choices on the performance of a technology, transitioning the technology to real-world industrial strength applications raises a different, yet challenging, set of technical issues.
We invite submissions describing innovations and implementations in all areas of speech and natural language processing technologies and systems that are relevant to industrial applications. The primary focus of this track is on papers that advance the understanding of, and demonstrate the effective handling of, practical issues related to the deployment of language processing technologies in non-trivial real-world systems. By “non-trivial real-world system” we mean an application that is deployed for real-world use, i.e. outside controlled environments such as a laboratories, classrooms or experimental crowd-sourced setups, and that uses natural language processing (including speech technology), even if not state of the art in terms of research. There is no requirement that the system be made by a for-profit company, but the users of the system must be outside of the NLP research community.
This track provides an opportunity to highlight the key insights and new research challenges that arise from real world implementations. Topics include, but are not limited to, the following (in alphabetical order):
- Continuous maintenance and improvement of deployed systems
- Design of application-relevant datasets
- Design of non-standard evaluation metrics
- Engineering challenges encountered while implementing at scale
- Ethical issues specific to deployed systems, including but not limited to privacy issues and bias
- Case-study papers on specific experiences (e.g. lessons learned, best practices, etc.)
- Handling unexpected inputs
- Monitoring and maintenance
- Novel practical solutions to known problems
- Novel previously unsolved problems
- Offline and online system evaluation methodologies
- Operational processes and workflows
- Practical considerations (e.g. cost, processing time)
- System combination and hybridization
In addition, opinion/vision papers and papers highlighting interesting negative results related to real-world applications are also welcome.
Submissions must clearly identify one of the following three areas they fall into:
- Deployed: Must describe a system that solves a non-trivial real-world problem. The focus may include describing the problem related to actual use cases, its significance (against opportunity size, value proposition, and ideal end state), design/formulation of methods, tradeoff design decision for solutions, deployment challenges, and lessons learned.
- Emerging: Must describe the development of a system that solves a non-trivial real-world problem (it need not be deployed or even close, but there needs to be evidence that this development is intended for real-world deployment). Papers that describe enabling infrastructure for large-scale deployment of natural language processing techniques also fall in this category.
- Discovery: Must include results obtained from NLP applications in real world scenarios that result in actionable insights. These discoveries should reveal promising directions in their application areas, leading to further system or societal enhancements. For example, an actionable discovery from an analysis of call center transcripts may reveal that certain language choices negatively impact customer experience, leading to better training of service representatives and improved customer experience.
Evaluation and decision criteria
Submissions will be reviewed in a double-blind manner and assessed based on their novelty, technical quality, potential impact, and clarity. Submissions in the industry track should emphasize real-world implementations of natural language processing systems, the development of such systems, or provide insights based on real-world datasets with obvious industry impact. For papers that rely heavily on empirical evaluations, the experimental methods and results should be clear, well executed, and repeatable (though the data may be proprietary).
Submission: Authors are invited to submit original, full-length (6 page) industry track papers that are not previously published, accepted to be published, or under consideration for publication in any other forum. Manuscripts should be submitted electronically, in PDF format and formatted using the official NAACL-HLT 2021 style templates:
Papers cannot exceed 6 pages in length (excluding ethical considerations and references) and should be submitted through the NAACL-HLT 2021 industry track online submission system (link coming soon). Appendices are not allowed for the industry track. Submissions of identical or closely related work to multiple NAACL-HLT tracks will be treated as duplicate submissions, which is not allowed for the industry track.
Extra space for ethical considerations: Extra space is allowed after the 6th page for an ethics/broader impact statement. At submission time, this means that if you need extra space for the ethical considerations section, it should be placed after the conclusion so that it is possible to rapidly check that the rest of the paper still fits in 6 pages. See below for a more detailed discussion of ethical issues.
Final version: Accepted papers will be given one additional page of content (up to 7 pages; ethical considerations, acknowledgements and references do not count against this limit) so that reviewers’ comments can be taken into account. Previous presentations of the work (e.g. preprints on arXiv.org) should be indicated in a footnote that should be excluded from the review submission, but included in the final version of papers appearing in the NAACL-HLT 2021 proceedings.
Presentation requirement for accepted papers: Industry track papers will be presented orally or as posters to be determined by the program committee. The decisions as to which papers will be presented orally and which ones as poster presentations will be based on the nature rather than the quality of the work. There will be no distinction in the proceedings between industry papers presented orally and those presented as posters. All accepted papers must be presented at the conference to appear in the proceedings. The 2021 Industry Track will run in parallel with the Research Track. At least one author of each accepted paper must register for NAACL-HLT 2021 by the early registration deadline.
Anonymity Period: The anonymity period for the NAACL-HLT 2021 Industrial Track is from December 18, 2020 to March 17, 2021. You may not make a non-anonymized version of your paper available online to the general community (for example, via a preprint server) during the anonymity period. You may not update the non-anonymized version during the anonymity period, and we ask you not to advertise it on social media or take other actions that would further compromise double-blind reviewing during the anonymity period.
Authors are required to honor the ethical code set out in the ACM Code of Ethics. The consideration of the ethical impact of our research, use of data, and potential applications of our work has always been an important consideration, and as artificial intelligence is becoming more mainstream, these issues are increasingly pertinent. We ask that all authors read the code, and ensure that their work is conformant to this code. We reserve the right to reject papers on ethical grounds, where the authors are judged to have operated counter to the code of ethics, or have inadequately addressed legitimate ethical concerns with their work.
As mentioned above, authors will be allowed extra space after the 6th page for a broader impact statement or other discussion of ethics. The NAACL review form will include a section addressing these issues and papers flagged for ethical concerns by reviewers will be further reviewed by an ethics committee. Note that an ethical considerations section is not required, but papers working with sensitive data or on sensitive tasks that do not discuss these issues will not be accepted. Conversely, the mere inclusion of an ethical considerations section does not guarantee acceptance. In addition to acceptance or rejection, papers may receive a conditional acceptance recommendation. Camera-ready versions of papers designated as conditional accept will be re-reviewed by the ethics committee to determine whether the concerns have been adequately addressed. Please read the ethics FAQ for more guidance on some problems to look out for and key concerns to consider relative to the code of ethics.
Industry Track Co-Chairs:
- Young-Bum Kim (Amazon)
- Yunyao Li (IBM Research)
- Owen Rambow (Stony Brook University)
General chair: Kristina Toutanova (Google)
|Start of the Anonymity Period||Friday||December 18, 2020|
|Paper Submission Deadline||Monday||January 18, 2021|
|Acceptance Notification||Wednesday||March 17, 2021|
|Final Version Submission Deadline||Sunday||April 18, 2021|
|Final notification for papers requiring ethics re-review||Friday||May 7, 2021|
All deadlines are 11.59 pm UTC -12h (“anywhere on Earth”).
Frequently Asked Questions
Is the industry track only for participants from industry?
No, the industry track welcomes participants from the entire ACL community. Researchers working on real-world applications that match the industry track call for papers are invited to submit papers. Everyone is welcome to attend industry track sessions.
What do you mean by real-world applications?
We are looking for applications that are deployed (or expected to be deployed) for real-world use, i.e. outside controlled environments such as laboratories, classrooms or experimental crowd-sourced setups.
Can students also submit papers to industry track?
Yes! If your work matches the industry track call for papers, consider submitting a paper to the industry track.
I work in the industry. Can I still submit my paper to the research track?
Absolutely! There are no changes to the main conference submissions. The industry track offers a forum to submit papers describing aspects of real-world applications that may differ in focus from the research track reviewing criteria.
Will the papers in the industry track be published in the proceedings?
Yes, industry track papers will be published as a separate volume of the proceedings. For example, see the NAACL-HLT 2019 proceedings.
How do I decide whether to submit to the research track or the industry track?
Papers describing key lessons learned and challenges pertaining to real-world deployment of NLP and speech technologies are best suited for industry track. Authors are advised to review the call for papers for both tracks and submit to the track that best matches your work. The list of topics and reviewing criteria may be helpful. You can also reach out to the track chairs if you need help deciding.