NeurIPS 2021 BEETL Competition: Benchmarks for EEG Transfer Learning

Data

Note: We provide APIs to download the following data sets when they are released; please see our Code&Tutorial for data download. Proportion of training/testing data could be different from the discription/figures below, please refer to our tutorial for exact details of the training/testing data.

Physionet sleep dataset

For Task 1, the sleep stage decoding task, Physionet sleep data set is one of the ideal data sets. The sleep-edf is a public database that contains 197 whole-night sleep recordings with event markers annotated by experts. Sleep patterns consist of sleep stages W, R, 1, 2, 3, 4, M (Movement time) and '?' (not scored). The number of subjects is large enough for transfer learning algorithms to learn the diversity of distributions while the file size is proper for participants to download (8GB). This data set has a clustered distribution of participants of different ages (see Fig.3).

For Task 2, the motor imagery decoding task, we selected three public data sets from the MOABB database as sources for transfer learning training. There will be two data sets used as the leaderboard testing and final testing for scoring. MOABB is a framework for benchmarking BCI classification algorithms on publicly available data sets. It allows automatic download of referenced data sets from different task paradigms; currently, there are 15 motor imagery, 3 SSVEP and 5 P300 datasets. MOABB also takes care of the dataset preprocessing to ensure a fair and reproducible evaluation. We selected three public motor imagery data sets in the MOABB containing left-hand and right-hand motor imagery as sources:

Cho 2017 Motor Imagery Dataset

This is a public and from the Motor Imagery dataset in and is part of MOABB. This data set contains a BCI experiment for motor imagery movement (MI movement) of the left and right hands with 52 subjects (19 females, mean age ± SD age = 24.8 ± 3.86 years); EEG data were collected using 64 Ag/AgCl active electrodes. A 64-channel montage based on the international 10-10 system was used to record the EEG signals with 512 Hz sampling rates. The EEG device used in this experiment was the Biosemi ActiveTwo system. The BCI2000 system 3.0.2 was used to collect EEG data and present instructions (left-hand or right-hand MI). Subjects were asked to imagine the hand movement depending on the instruction given. Five or six runs were performed during the MI experiment. After each run, we calculated the classification accuracy over one run and gave the subject feedback to increase motivation. Between each run, a maximum 4-minute break was given depending on the subject’s demands.

BNCI 2014-001 Motor Imagery dataset

It is from data set IIa from the BCI competition 4 in and is also publically available as part of MOABB. This data set consists of EEG data from 9 subjects. The cue-based BCI paradigm consisted of four different motor imagery tasks, namely the imagination of movement of the left hand (class 1), right hand (class 2), both feet (class 3), and tongue (class 4). Two sessions on different days were recorded for each subject. Each session is comprised of 6 runs separated by short breaks. One run consists of 48 trials (12 for each of the four possible classes), yielding a total of 288 trials per session. Twenty-two electrodes were used to record the EEG; 250 Hz and bandpass-filtered between 0.5 Hz and 100 Hz. The sensitivity of the amplifier was set to 100 uV. An additional 50 Hz notch filter was enabled to suppress line noise. Left-hand and right-hand motor imagery of the data set will be used in this competition.

PhysionetMI dataset

Is a motor imagery dataset from the Physionet Motor Imagery dataset and is public and also part of MOABB. This data set consists of over 1500 one- and two-minute EEG recordings, obtained from 109 volunteers. Subjects performed different motor/imagery tasks while 64-channel EEG were recorded using the BCI2000 system (http://www.bci2000.org). Each subject performed 14 experimental runs: two one-minute baseline runs (one with eyes open, one with eyes closed), and three two-minute runs of each of the four following tasks: 1. A target appears on either the left or the right side of the screen. The subject opens and closes the corresponding fist until the target disappears. Then the subject relaxes. 2. A target appears on either the left or the right side of the screen. The subject imagines opening and closing the corresponding fist until the target disappears. Then the subject relaxes. 3. A target appears on either the top or the bottom of the screen. The subject opens and closes either both fists (if the target is on top) or both feet (if the target is on the bottom) until the target disappears. Then the subject relaxes. 4. A target appears on either the top or the bottom of the screen. The subject imagines opening and closing either both fists (if the target is on top) or both feet (if the target is on the bottom) until the target disappears. Then the subject relaxes.

Leaderboard test dataset

We will not inform the name of the data set to participants, details of the data set will be released according to the competition schedule.

Final test data set for ranking

We have collected a data set for the purpose of the competition which will be our test set for Task and that we will add afterwards to the MOABB database. In the Cybathlon 2020 BCI game (https://cybathlon.ethz.ch/en/event/disciplines/bci ), paralysed subjects attempt to control a virtual wheelchair on a winding race track to go straight, turn left, turn right and turn on the headlight. The dataset consists of eight sessions from six right-handed subjects (male), details of the data set will be released according to the competition schedule.

Training and test set

For the Physionet sleep data set (Sleep Cassette Study, around 150 sessions), we will provide subjects aged from 25-64 with full labels as resources and 5 subjects aged from 65 to 79 as example subject of this age group; we will test the performance of the algorithm on more subjects aged from 65 to 79. Similarly, we will provide 5 subjects aged from 80 to 95 as example subjects of this age group, we will test the performance of the algorithm on more subjects aged from 80 to 95. For the five motor imagery data sets above for task 2, we will align their sampling rates, window lengths and channels. We will provide 64 channels for all four data sets. For the Physionet MI, Cho2017 and BNCI data sets specifically, we will provide full data sets with labels as sources. For the leaderboard test data set, we do not inform participants what data set it is to avoid cheating. The the Cybathlon data set will be used for the final scoring of the second task. In both of the test data sets, we provide approximately 30% of data with labels per label per subject as examples for transfer learning. This represents a set of problem where we can have some calibration from new users before they use the decoding system. Please see our tutorial code in Code&Evaluation page for more information about competition data sets.