Descriptive modelling of utterance transformations in chains:

short-term linguistic evolution in a large-scale online experiment

Sébastien Lerique

IXXI, École Normale Supérieure de Lyon

HBES 2018, Amsterdam

Camille Roth

médialab, Science Po, Paris

Adamic et al. (2016)

Online opinion dynamics

Transmission chains for cultural evolution theories

Short-term cultural evolution

Reviews by

Mesoudi and Whiten (2008)

Whiten et al. (2016)

Leskovec et al. (2009)

Boyd & Richerson (1985, 2005)

Sperber (1996)

Hierarchicalisation

(Mesoudi and Whiten, 2004)

Social & survival information

(Stubbersfield et al., 2015)

Negativity

(Bebbington et al., 2017)

Manual

Automated (computational)

In vitro

In vivo

Moussaïd et al. (2015)

Lauf et al. (2013)

Lerique & Roth (2018)

Danescu-Niculescu-Mizil et al. (2012)

Quotes in 2011 news on Strauss-Kahn

Claidière et al. (2014)

Cornish et al. (2013)

Word list recall (Zaromb et al. 2006)

Simple sentence recall (Potter & Lombardi 1990)

Evolution of linguistic content

No ecological content

Experiment setup

Control over experimental setting

Fast iterations

Scale similar to in vivo

Browser-based

Story transformations

At Dover, the finale of the bailiffs' convention. Their duties, said a speaker, are "delicate, dangerous, and insufficiently compensated."

depth in branch

At Dover, the finale of the bailiffs convention,their duty said a speaker are delicate, dangerous and detailed

At Dover, at a Bailiffs convention. a speaker said that their duty was to patience, and determination

In Dover, at a Bailiffs convention, the speaker said that their duty was to patience.

In Dover, at a Bailiffs Convention, the speak said their duty was to patience

At Dover, the finale of the bailiffs' convention. Their duties, said a speaker, are "delicate, dangerous, and insufficiently compensated."

At Dover, the finale of the bailiffs convention,their duty said a speaker are delicate, dangerous and detailed

Modelling transformations

Needleman and Wunsch (1970)

AGAACT-
 | ||
-G-AC-G

AGAACT

GACG

Finding her son, Alvin, 69, hanged, Mrs Hunt, of Brighton, was so depressed she could not cut him down
Finding her son Arthur 69 hanged Mrs Brown from Brighton was so upset she could not cut him down
Finding her son Alvin  69 hanged Mrs Hunt of -     -    Brighton, was so depressed she could not cut him down
Finding her son Arthur 69 hanged Mrs -    -  Brown from Brighton was so upset she could not cut him down

Apply to utterances using NLP

At Dover, the finale of the bailiffs convention, their duty said a speaker are delicate, dangerous and detailed
At Dover, at a Bailiffs convention. a speaker said that their duty was to patience, and determination
At Dover the finale of the -  - bailiffs convention - -       -    -    their duty
At Dover -   -      -  -   at a Bailiffs convention a speaker said that their duty 


said a speaker are delicate dangerous -   -  -        and detailed -
-    - -       -   -        -         was to patience and -        determination
At Dover the finale of the -  - bailiffs convention |-Exchange-1------| their duty
At Dover -   -      -  -   at a Bailiffs convention a speaker said that their duty 


said a speaker are delicate dangerous -   -  -        and detailed -
|-Exchange-1------------------------| was to patience and -        determination
said a speaker are delicate dangerous |-E2----|
|E2| a speaker -   -        -         said that
said -
said that

\(\hookrightarrow E_1\)

\(\hookrightarrow E_2\)

Extend to build recursive deep alignments

Deep sequence alignments

Transformation diagrams

Results

Frequency

Frequency

Frequency

|chunk|

|chunk|

Deletion

Insertion

Replacement

Position in \(u\)

\(|u|_w\)

Number of operations vs. utterance length

Susceptibility vs. position in utterance

Deletions tend to gate other operations

Insertions relate to preceding deletions

Stubbersfield et al. (2015)

Bebbington et al. (2017)

Links the low-level with contrasted outcomes

Conclusion

In vivo & oral applications (social networks)

Parsimonious explanations of higher level evolution

Semantic parses, long-lived chains with recurring changes

Lots to do

Quantitative analysis of changes

Inner structure of transformations

WEIRD participants

Written text

No controlled context

Caveats

Very Soon™ on arXiv.org

Challenges with meaning

Can you think of anything else, Barbara, they might have told me about that party?

I've spoken to the other children who were there that day.

S

B

Abuser

The Devil's Advocate (1997)

?

Strong pragmatics (Scott-Phillips, 2017)

Access to context

Theory of the constitution of meaning

Challenges

Seeds

#participants
#root utterances
tree size
Duration
Spam rate
Usable reformulations
53 49 2 x 70
54 50 25/batch
48 49 70
64min 43min 37min/batch
22.4% + 3.5% 0.8% + 0.6% 1% + 0.1%
1980 2411 3506
Pilot 1 Exp. A Exp. A'

Pilots

MemeTracker, WikiSource,

12 Angry Men, Tales,

News stories

Exp. A

Memorable/non-memorable quote pairs

(Danescu-Niculescu-Mizil et al., 2012)

Exp. A'

Nouvelles en trois lignes

(Fénéon, 1906)

“Hanging on to the door, a traveller a tad overweight caused his carriage to topple, in Bromley, and fractured his skull.”

“Three bears driven down from the heights of the Pyrenees by snow have been decimating the sheep of the valley.”

“A dozen hawkers who had been announcing news of a nonexistent anarchist bombing at King's Cross have been arrested.”

Live experiment

Alignment optimisation

\(\theta_{open}\)

\(\theta_{extend}\)

\(\theta_{mismatch}\)

\(\theta_{exchange}\) by hand

All transformations

Hand-coded training set size?

Train the \(\theta_*\) on hand-coded alignments

Simulate the training process: imagine we know the optimal \(\theta\)

1. Sample \(\theta^0 \in [-1, 0]^3\) to generate artificial alignments for all transformations

2. From those, sample \(n\) training alignments

3. Brute-force \(\hat{\theta}_1, ..., \hat{\theta}_m\) estimators of \(\theta_0\)

4. Evaluate the number of errors per transformation on the test set

Test set

10x

10x

\(\Longrightarrow\) 100-200 hand-coded alignments yield \(\leq\) 1 error/transformation

Gap open cost \(\rightarrow \theta_{open}\)

Gap extend cost \(\rightarrow \theta_{extend}\)

Item match-mismatch

similarity(w, w') = \begin{cases} S_C \left( w, w' \right) & if\ we\ have\ word\ vectors\ for\ both\ w\ and\ w' \\ \delta_{lemma(w), lemma(w')} & otherwise \end{cases}
similarity(w,w)={SC(w,w)if we have word vectors for both w and wδlemma(w),lemma(w)otherwisesimilarity(w, w') = \begin{cases} S_C \left( w, w' \right) & if\ we\ have\ word\ vectors\ for\ both\ w\ and\ w' \\ \delta_{lemma(w), lemma(w')} & otherwise \end{cases}
score(w, w') = similarity(w, w') + \theta_{mismatch}
score(w,w)=similarity(w,w)+θmismatchscore(w, w') = similarity(w, w') + \theta_{mismatch}

Example data

Immediately after I become president I will confront this economic challenge head-on by taking all necessary steps

immediately after I become a president I will confront this economic challenge

Immediately after I become president, I will tackle this economic challenge head-on by taking all the necessary steps

This crisis did not develop overnight and it will not be solved overnight

the crisis did not developed overnight, and it will be not solved overnight

original

This, crisis, did, not, develop, overnight, and, it, will, not, be, solved, overnight

this, crisis, did, not, develop, overnight, and, it, will, not, be, solved, overnight

this, crisis, did, not, develop, overnight, and, it, will, not, be, solved, overnight

crisi, develop, overnight, solv, overnight

tokenize

lowercase & length > 2

stopwords

stem

The crisis didn't happen today won't be solved by midnight.

crisi, happen, today, solv, midnight

d = 0,6

Utterance-to-utterance distance

Aggregate trends

Size reduction

Transmissibility

Variability

Lexical evolution - POS

\sigma_g^- = \frac{s_g^-}{s_g^0}
σg=sgsg0\sigma_g^- = \frac{s_g^-}{s_g^0}
\sigma_g^+ = \frac{s_g^+}{s_g^0}
σg+=sg+sg0\sigma_g^+ = \frac{s_g^+}{s_g^0}

Step-wise

Susceptibility

Lexical evolution (1)

\sigma_g^- = \frac{s_g^-}{s_g^0}
σg=sgsg0\sigma_g^- = \frac{s_g^-}{s_g^0}
\sigma_g^+ = \frac{s_g^+}{s_g^0}
σg+=sg+sg0\sigma_g^+ = \frac{s_g^+}{s_g^0}

Step-wise

Susceptibility

Feature variation

Lexical evolution (2)

Along the branches

Descriptive modelling of utterance transformations in chains: short-term linguistic evolution in a large-scale online experiment

By Sébastien Lerique

Descriptive modelling of utterance transformations in chains: short-term linguistic evolution in a large-scale online experiment

Talk at HBES 2018, Amsterdam

  • 59
Loading comments...

More from Sébastien Lerique