NeurIPS 2020 Code Submission Policy

NeurIPS 2020 will use the following Code and Data Submission Policy.

 

1. The policy only applies to papers that contribute and present experiments with a) a new algorithm (or a modification to an existing algorithm) or b) a new dataset. That is, a paper is not covered by this policy if:

      a. The paper is not claiming the contribution of any novel algorithm and it is tested on existing datasets.

      b. The paper presents a new algorithm but only analyzes it theoretically (i.e., no experimental results are presented).

2. Code and data submission for papers covered by this policy is expected but not enforced.

3. The policy accepts a reimplementation by the authors that isn't the code originally run to produce the results reported in the paper (what is instead requested is the equivalent of an official implementation of the paper's contribution).

4. The policy accepts code that isn't “executable” as is as it has dependencies going beyond the algorithm itself and that cannot be released. Such dependencies would include: 

      a. Dataset that cannot be released (e.g., for privacy reasons).

      b. Specialized hardware that might not be commonly accessible (e.g., specialized accelerators or robotic platforms).

      c. Non-open sourced or non-free libraries, which do not include the algorithm that is claimed as the scientific contribution of the paper (e.g., paid-for mathematical programming solvers, commercial simulators, MATLAB).

         The authors will be asked to explain what dependencies are not released and why.

5. The policy expects code only for accepted papers, and only by the camera-ready deadline (October 22, 2020).

6. The policy strongly encourages authors reporting experiments on a new dataset to conform to the following rules:

  • A link to the dataset is provided in the paper (at submission time in anonymized format).
  • The dataset is deposited in a repository that ensures long term preservation of the data.
  • The dataset has a persistent identifier such as Digital Object Identifier or Compact Identifier.
  • The dataset adheres to Schema.org or DCAT metadata standards.
  • The license and/or any data access restrictions are described in the paper.
  • The paper includes a convincing justification of the special nature of the dataset that makes it impossible to conform to these suggestions.

 

After the camera-ready deadline, NeurIPS intends to measure the percentage of accepted papers for which code and data were not released, despite being covered by the policy.