OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages

• This paper introduce OpenHands, a library where we take four key ideas from the NLP community for low-resource languages and apply them to sign languages for word-level recognition.
• It is a open-source library, all models and datasets in OpenHands with a hope that it makes research in sign languages more accessible.

Abstract:

Introduction:

• When compared against text and speech-based NLP research, the progress in AI research for sign languages is significantly lagging.
• We implement these ideas and release several datasets and models in an open-source library OpenHands with the following key contributions:

•  Standardizing on pose as the modality.
•  Standardized comparison of models across languages.
• Corpus for self-supervised training.
• Effectiveness of self-supervised training.x

Background and Related Work:

Models for ISLR:

Pretraining strategies

• Sequence based models
• Graph based models

• Masking-based pretraining
• Contrastive-learning based
• Predictive Coding

Datasets available in Openhands library:

Experiment Setup and Results:

• PyTorch Lightning to preprocessing data and for training pipelines
• Adam Optimizer
• For LSTM Batch size as 32 and initial learning rate (LR) as 0.005
• For BERT,  batch size  as 64, and LR of 0.0001.
• For ST-GCN and SL-GCN, batch size of 32 and LR of 0.001.
• Single NVIDIA Tesla V100 GPU

Comparision of each dataset with its state of the art model and with models available in OpenHands:

Comparision of Inference times across various models and CPUs:

Indian SL Corpus for Self-supervised pretraining

Explains process of collection and storing SL corpus:

Pretraining Startiges:

Effectiveness of pretraining startegies:

Evaluation on low-resource and crosslingual settings:

Conclusion:

•  Introduced 7 pose-based datasets and 4 different ISLR models across 6 sign languages as part of openhands library.
• Also released first large corpus of SL data for self-supervised pretraining.
• Among different pretraining strategies and found DPC as effective.
• Also showed that pretraining is effective both for in-language and crosslingual transfer.

Future Work:

• Evaluating alternative graph-based models.
• Efficiently sampling the data from the raw dataset such that the samples are diverse enough.
• Quantized inference for 2×-4× reduced latency
• Extending to CSLR.