Reproducibility Checklist

This is a copy of the reproducibility checklist that the authors will be asked to answer during paper submission. See the Reproducibility Criteria section in the Call for Papers for more details.

---

For all reported experimental results:

• A clear description of the mathematical setting, algorithm, and/or model
• A link to a downloadable source code, with specification of all dependencies, including external libraries (recommended for camera ready, though welcome for initial submission)
• A description of computing infrastructure used
• The average runtime for each model or algorithm, or estimated energy cost
• The number of parameters in each model
• Corresponding validation performance for each reported test result
• A clear definition of the specific evaluation measure or statistics used to report results.

For all results involving multiple experiments, such as hyperparameter search:

• The exact number of training and evaluation runs
• The bounds for each hyperparameter
• The hyperparameter configurations for best-performing models
• The method of choosing hyperparameter values (e.g. manual tuning, uniform sampling, etc.) and the criterion used to select among them (e.g. accuracy)
• Summary statistics of the results (e.g. mean, variance, error bars, etc.)

For all datasets used:

• Relevant statistics such as number of examples and label distributions
• Details of train/validation/test splits
• An explanation of any data that were excluded, and all pre-processing steps
• For natural language data, the name of the language(s)
• A link to a downloadable version of the dataset or simulation environment
• For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control