Pali

(Dead) Indo-aryan language
Fusional language
Rich morphology
Sandhi

Source: https://commons.wikimedia.org/wiki/File:BoreanLanguageTree.png

Fusional language

Morphological information added by affigation

No 1:1 correspondence

DEVO

Base: DEV-
- god/deity
Ending: -O
- noun
- singular
- masculine
- nominative

Compounding

naccagītavāditavisūkadassanamālāgandhavilepanadhāraṇamaṇḍanavibhūsanaṭṭhānā

Compounding

naccagītavāditavisūka-dassanamālāgandhavilepanadhāraṇamaṇḍanavibhūsana-ṭṭhānā

dancing singing music show-watching garland perfume cosmetics wearing decoration decoration

Compounding

naccagītavāditavisūka-dassanamālāgandhavilepanadhāraṇamaṇḍanavibhūsana-ṭṭhānā

dancing, singing, music, going to see entertainments, wearing garlands, using perfumes, and beautifying the body with cosmetics

7th precept

naccagītavāditavisūkadassanamālāgandhavilepanadhāraṇamaṇḍanavibhūsanaṭṭhānā veramaṇi sikkhāpadaṃ samādiyāmi

I adopt the precept of refraining from ...

Sandhi

External sandhi

evaṃ ca (and thus) → evañca

Internal sandhi

paca + ti → pacati (he cooks)

paca + mi → pacāmi (I cook)

canda (moon) + udayo (rising) → candodayo (rising of the moon)

Internal sandhi

paca + ti → pacati (he cooks)

paca + mi → pacāmi (I cook)

canda (moon) + udayo (rising) → candodayo (rising of the moon)

The Problem

Low-resource language

Why don't we adapt resources from Sanskrit?

Top Resources

Dictionaries

Morphological analyzers

Credit: http://iflizwerequeen.com

Lingua Franca

Written in different scripts

Lingua Franca

Written in different scripts

Introduces variation!

Scripts

Sinhalese
Devanagari
Burmese
Transliterations
...

Literature

Scarce and not exhaustive

No annotated corpus

Generation

and Overgeneration

Irregular

Dictionary lookup

Rule based generation:

Lemma => Stem

Stem + Ending => Form

Regular

Dictionary lookup

Word class specific lemma ending

Lemma - Ending → Stem

Stem + Ending → Surface Form

Stem + Ending → Form

Ending

Compiled Morphological Information

<paradigms>
    <paradigm type="noun">
        <number type="singular">
            <declension type="a">
                <gender type="masculine">
                    <case type="nominative">
                        <ending>o</ending>
                        <ending type="Drare">e</ending>
                    </case>
                    <case type="vocative">
                        <ending>a</ending>
                        <ending>ā</ending>
                        <ending type="Drare">e</ending>
                        <ending type="Drare">o</ending>
                    </case>
                    <case type="accusative">
                        <ending>aṃ</ending>
                    </case>

<paradigms>
    <paradigm type="noun">
        <number type="singular">
            <declension type="a">
                <gender type="masculine">
                    <case type="nominative">
                        <ending>o</ending>
                        <ending type="Drare">e</ending>
                    </case>
                    <case type="vocative">
                        <ending>a</ending>
                        <ending>ā</ending>
                        <ending type="Drare">e</ending>
                        <ending type="Drare">o</ending>
                    </case>
                    <case type="accusative">
                        <ending>aṃ</ending>
                    </case>

<paradigms>
    <paradigm type="noun">
        <number type="singular">
            <declension type="a">
                <gender type="masculine">
                    <case type="nominative">
                        <ending>o</ending>
                        <ending type="Drare">e</ending>
                    </case>
                    <case type="vocative">
                        <ending>a</ending>
                        <ending>ā</ending>
                        <ending type="Drare">e</ending>
                        <ending type="Drare">o</ending>
                    </case>
                    <case type="accusative">
                        <ending>aṃ</ending>
                    </case>

<paradigms>
    <paradigm type="noun">
        <number type="singular">
            <declension type="a">
                <gender type="masculine">
                    <case type="nominative">
                        <ending>o</ending>
                        <ending type="Drare">e</ending>
                    </case>
                    <case type="vocative">
                        <ending>a</ending>
                        <ending>ā</ending>
                        <ending type="Drare">e</ending>
                        <ending type="Drare">o</ending>
                    </case>
                    <case type="accusative">
                        <ending>aṃ</ending>
                    </case>

Lemma => Stem

Stem + Ending => Form

deva => dev-

dev- + -o => devo

Lemma => Stem

Stem + Ending => Form

deva => dev-

dev- + -o => devo

            <declension type="ant">
                <gender type="masculine">
                    <case type="nominative">
                        <ending>aṃ</ending>
                        <ending>ā</ending>
                        <ending type="Cm2">anto</ending>
                        <ending type="Drare">o</ending>
                        <ending>ato</ending>
                    </case>

I make

I cook

karo + mi = karomi

paca + mi = pacāmi

bhavaṃ (sir)

stem: bhav-

ending: -anto

form: ~~bhavanto~~

bhanto

Lemma

Derive stem
Select paradigm(s) based on word class
Combine stem and endings
Return generated forms and associated information

Verbs

Of Roots and Bases

Abstract Root

\sqrt{kar}

\sqrt{kar}

(to make)

Base

\sqrt{kar} \to karo

\sqrt{kar} \to karo

\sqrt{pac} \to paca

\sqrt{pac} \to paca

(to make)

(to cook)

(to fight)

\sqrt{yudh} \to yujjha

\sqrt{yudh} \to yujjha

Seven declension classes

1+ bases

\sqrt{cur}

\sqrt{cur}

core-, coraya-

(to steal)

1+ bases

\sqrt{rudh}

\sqrt{rudh}

rundha-, rundhi-, rundhī-, rundhe-, rundho-

(to obstruct)

Verb forms based on
Root or Base?

Irregular forms

Dictionary lookup

Full/Partial Irregularity

Output

JSON/XML

Key:Value pairs

Receiver can decide what information to use

{" lemma":"eka","forms ":{"numeral":[{
"gender ":"masculine", "number ":" singular",
"word ":" eko", "case":" nominative"},
{"gender ":"masculine", "number ":" singular","word ":"ekassa", "case":" genitive"},...

Analysis

Lookup

Dictionary/Table lookup

Heuristic approach

Identify paradigmatic ending

→ Morphological Analysis

→ Separation Stem-Ending

buddhe

<gender type="masculine">
    <case type="nominative">
        <ending>o</ending>
        <ending type="Drare">e</ending>
    </case>
    <case type="vocative">
        <ending>a</ending>
        <ending>ā</ending>
        <ending type="Drare">e</ending>
        <ending type="Drare">o</ending>
    </case>
    <case type="accusative">
        <ending>aṃ</ending>
    </case>

buddhe

<gender type="masculine">
    <case type="nominative">
        <ending>o</ending>
        <ending type="Drare">e</ending>
    </case>
    <case type="vocative">
        <ending>a</ending>
        <ending>ā</ending>
        <ending type="Drare">e</ending>
        <ending type="Drare">o</ending>
    </case>
    <case type="accusative">
        <ending>aṃ</ending>
    </case>

Word Class Guesser

Heuristic Approach

Lemma

Free Form

Identify possible endings
Weigh by length
Weigh by frequency
Prune results

Identify possible endings

if (ends(lemma, "a", "ā", "i", "ī", "u", "ū", "ant", "vā", "mā", "at")) {
    guesses.add("adjective");
}
if (ends(lemma, "a", "i", "aṃ", "ma", "ya")) {
    guesses.add("numeral");
}
if (ends(lemma, "uṃ")) {
    guesses.add("indeclinable");
}

Word Class Guesser: Lemma

Code Excerpt

Results

	Accuracy
Nouns-Adjectives	99.96%
Pronouns	88.57%
Numerals	76.62%
Verbs	63.37%

Sandhi

Compound Sandhi

Intuition

Identify possible sandhi loci
Split into n words such that

\forall n:w_n \in D

\forall n:w_n \in D

Requires extensive Dictionary
More than one analysis possible
Not a compound

Problems

External Sandhi

Sandhi-inducing words

ca (and)
hi (because)
pi (also)

Corpus-based resolution

Hand-written rules

Regular Expressions

Replacement rules
\bpañca\b	X
ñca\b	ṃ ca
X	pañca
ñhi\b	ṃ hi
ñpi\b	ṃ pi

Replacement rules
\bpañca\b	X
ñca\b	ṃ ca
X	pañca
ñhi\b	ṃ hi
ñpi\b	ṃ pi

Internal Sandhi

Conclusion

Paradigms for Generation and Analysis

Dictionary Integration for additional information

Rule-based and heuristic backup

RegEx-based External Sandhi Resolution

Lookup

Server Architecture

Well documented REST API

Easy integration

Data Processing

Extract structured data from unstructured data

[n. ag. fr. abhijjhita in med. function] one who covets M <smallcaps>i.</smallcaps> 287 (T. abhijjhātar, v. l. °itar) = A <smallcaps>v.</smallcaps> 265 (T. °itar, v. l. °ātar).

Pacati，[Ved．pacati，Idg．*peqǔō，Av．pac-； Obulg．peka to fry，roast，Lith，kepū bake，Gr．pέssw cook，pέpwn ripe] to cook，boil，roast Vin．IV，264； fig．torment in purgatory （trs．and intrs．）：Niraye pacitvā after roasting in N．S．II，225，PvA．10，14．-- ppr．pacanto tormenting，Gen．pacato （+Caus．pācayato） D．I，52 （expld at DA．I，159，where read pacato for paccato，by pare daṇḍena pīḷentassa）．-- pp．pakka （q．v．）．‹-› Caus．pacāpeti & pāceti （q．v．）．-- Pass．paccati to be roasted or tormented （q．v．）．（Page 382）

Manual annotation

Open Problems

Verbs

Use verb form table

Attested forms only

Internal Sandhi

Illustrating Calculation

Splitting Internal Sandhi

"When two vowels meet, one may be elided."

When two vowels meet:

elide first vowel
elide second vowel
no elision

8 vowels

n-vowel-word

N=(1+(2*8))^n

N=(1+(2*8))^n

n = 2 \to N = 289

n = 2 \to N = 289

n=1\to N=17

n=1\to N=17

n=3\to N=4913

n=3\to N=4913

"A final dental is assimilated to the following consonant"

(DENTAL) (CONSONANT) : duplicate($2)

kk: t k
kk: th k
kk: d k
kk: dh k
kk: n k
kk: l k
kk: s k
...

224 possibilities

151 rules

Sandhi merge rules

151 rules

Sandhi merge rules

Sandhi split rules

1103 rules

Overall architecture

Morphological

analyzer

and generator

Dictionary

Morphological

analyzer

and generator

Dictionary

Server

Morphological

analyzer

and generator

Dictionary

Server

Dictionary

GUI

Data processor

and scripting

engine

Corpus management and processing

tool

Thank you for your attention!

Questions?

Morphological Analysis and Generation for Pali

@daalft

Pali

Pali

Fusional language

DEVO

Compounding

Compounding

Compounding

7th precept

Sandhi

External sandhi

Internal sandhi

Internal sandhi

The Problem

Low-resource language

Why don't we adapt resources from Sanskrit?

Top Resources

Dictionaries

Morphological analyzers

Lingua Franca

Lingua Franca

Lingua Franca

Scripts

Literature

Literature

No annotated corpus

Generation

Generation

Irregular

Regular

Word class specific lemma ending

Lemma - Ending → Stem

Stem + Ending → Surface Form

Stem + Ending → Form

Ending

Ending

Ending

Ending

Ending

Ending

Lemma => Stem

Stem + Ending => Form

deva => dev-

dev- + -o => devo

karo + mi = karomi

paca + mi = pacāmi

bhavaṃ (sir)

Lemma

Verbs

Of Roots and Bases

Abstract Root

Base

Seven declension classes

1+ bases

1+ bases

Verb forms based on Root or Base?

Irregular forms

Dictionary lookup

Output

JSON/XML

Analysis

Lookup

Heuristic approach

Word Class Guesser

Heuristic Approach

Lemma

Free Form

Word Class Guesser: Lemma

Results

Sandhi

Compound Sandhi

Intuition

Requires extensive Dictionary

More than one analysis possible

Not a compound

Problems

External Sandhi

Sandhi-inducing words

Corpus-based resolution

Verb forms based on
Root or Base?