NeurIPS 2021 BEETL Competition: Benchmarks for EEG Transfer Learning

Code and Tutorial

We release start kits for the two tasks at https://github.com/XiaoxiWei/NeurIPS_BEETL.

Registration and Leaderboard

We will run our competition registration and label files submission on Codalab: NeurIPS 2021 BEETL Competition.

Procedure

  • We will provide training data the true labels and baseline methods with a tutorial at the beginning of this competition.
  • The first part of the test sample will be released together with training data. Test sets are without true labels. This part is the leader board testing set.
  • After participants have the predictions of the leader board test samples, they can upload the TXT file with their predictions to our leader board website to see real-time ranking and scores of each task and data sets.
  • At the final stage of the competition, we will release the test samples for the final ranking.
  • Participants upload their predictions to our website together with their code and a 2-page document (in paper format with methods, analysis and discussion).
  • Their final rankings and scores will be released after the competition deadline.

Evaluation

There will be two phases of evaluation. The first phase is the leader board evaluation. After participants upload their predictions of the leader board test trials,the website will automatically compare the predictions with true labels, and update the rankings on our websites. The second phase consists in uploading predictions of the final test set to our website, then we will ask participants to email us their code (python are recommended but participants can use any programming language) and documents. We will write emails to ask for code and documents. We will test run their python code to generate labels and compare them with the labels they uploaded, please make sure the labels are reproducible using, for example, fixed random seeds. If the test run labels are not identical to the ones participants uploaded, we may consider cancelling the scores accordingly. In their document, we highly recommend they have a comparison of accuracy with and without their transfer learning approach; and both should present in their code if applicable. Please be AWARE that ranking in the first phase does NOT represent final rankings, we may use different data sets with different distributions in the second phase. Scores for final ranking will be only computed based on the second phase (final test phases for scoring). The final ranking will be the sum score of the two tasks.


If a participant only choose to submit in one Task and not the other, their Final Score will be calculated as follows: their score for the not submitted task will be set to the 0.99 * minimum score achieved by participants that actually submitting in that task.