Probabilistic Ensembles of Zero- and Few-Shot Learning Models for Emotion Classification

Angelo Basile

Guillermo Perez-Torro

Marc Franco-Salvador

angelo.basile@symanto.com

guillermo.perez@symanto.com

marc.franco@symanto.com

The angry wolf ate the happy boy

Building affective corpora is hard.

Can we approach at least the existing datasets using few or no annotated instances?

Research Question

Emotion Classification as Entailment

I loved the pizza!

This person expressed a feeling of pleasure.

This person feels sad.

Premise

Hypothesis

This person feels [...].

  • Entailment
  • Contradiction
  • Neutral

Natural Language Inference (NLI)

JOY

SADNESS

What NLI dataset?

What PLM?

What hypothesis?

???

ANLI, Fever, MNLI, XNLI?

BERT, BART, RoBERTa?

This person feels [...], This person is feeling [...]?

Our Proposal

Build many different models and infer the best possible label from their predictions.

MACE (Hovy et al., 2013)

The angry wolf ate the happy boy

Model 1

Model 2

...

Model N

JOY

SADNESS

SADNESS

...

Final Label

Experiments

Unify Emotion

(Bostan and Klinger, 2018)

  • Zero-shot models provide modest performance
  • A model of aggregation helps
  • Few-shot NLI models are almost as good as fully-supervised models

The benefits of a model of aggregation

  • confidence value for each instance
  • estimation of model's performance
  • usually better than majority voting
  • integrate labeled data, if available
  • merge rule-based and deep learning systems

(see Passonneau and Carpenter, 2014)