Zero Speech Challenges organization
J. Karadayi - M. Bernard - CoML team
March 09, 2020
4 editions so far...
www.zerospeech.com
ZS 2021
ZS 2020 = 2017 + 2019
Outline
(focus on ZRC 2020)
- participant registration
- participant submission
- submission / evaluation pipeline
- ideas for improvements
Participant registration
procedure at zerospeech.com/2020
1. send a mail to zerospeech2020@gmail.com
[manual reply with license agreement]
Limitations...
- too complex (misuse from participants)
- not centralized (gmail, codalab, github)
- require manual intervention from our side
2. accept the license agreement
[manually send the dataset password]
[manually add participant to google doc]
3. receive password and download the dataset
4. download/install the software from github
5. register on codalab
-
zerospeech2020-evaluate -
zerospeech2020-validate
Participant submission
Pretty simple but codalab...
- sometimes goes out of line
- no feedback during upload (may fail silently)
- not robust to connection interruption
- random bandwidth (shared with all challenges)
=> we need a robust solution for large files
(baseline.zip is 6.8 GB)
- upload the archive to codalab
- get notification of submission acceptance (or rejection)
- ... wait to May 01, 2020 ... leaderboard at zerospeech.com/2020
- participant build an archive "submission.zip"
- validate it using
zerospeech2020-validate
Submission pipeline
upload a submission to habilis and validate it
Limitation ...
about 4h per submission (without concurency)
Evaluation pipeline
3 evaluations
- machine (ABX and bitrate on embeddings)
- human (speech synthesis quality)
- is open source ?
A suite of little tools on habilis
- zs2020-status : eval status of each submission
- zs2020-eval-machine : run ABX & bitrate on a submission
- zs2020-eval-opensource : help us inspecting submissions
- zs2020-publish-leaderboard : upload zerospeech.com
+ human evaluation
Limitations
- codalab not adapted and misused (only for upload to habilis)
- registration too complex, need automation
Improvements
Implement a "user profile" on zerospeech.com
- sign-in / sign-up / personal information
- register to specific edition and accept licence agreement
- upload submissions / consult its status
web tech skills for implementation ?
self-hosted on habilis2 ? hosted on OVH ? where to compute evals ?