Which languages are in-vocabulary?
ѢCoptic ϩ, U+03E9 ("hori")| model | token | |
|---|---|---|
| Gemma-2 | dedicated token | native |
| GPT-4o | two UTF-8 bytes | byte fallback |
| mBERT | [UNK] | no support |
Which models support my language? And how?
summer > [63666]
spending > [2203, 2518]
лето > [3114, 8341, 1482]
лѣто > [3114, 141, 96, 25657] 4 tokens for 4 characters
എന്റെ പേര് മനു ആണ് > [30 tokens for 19 charachters]
Gemma-3 natively tokenizes two-thirds of Akkadian cuneiform.
Every other model tested, including Gemma-2, is at 0–2%.
Find your language(s)
Check any character or language against every model at once, or explore a model's full coverage report, tier by tier.
https://apjanco-loove.hf.space/
Loove
By Andrew Janco
