ERA V5 · Assignment 2
One shared tokenizer.
Four writing systems.
A reproducible 10,000-token BPE trained on faithful Wikipedia Markdown in English, Hindi, Telugu, and Kannada.
01 / Measurements
Fertility by language
Lower is more compact. The score rewards a small gap between the highest and lowest fertility.
| Language | Tokens | Faithful units | Fertility | Relative fertility |
|---|---|---|---|---|
| Englishen | 1,14,626 | 1,86,666 | 0.614070 | |
| Hindihi | 52,397 | 88,772 | 0.590242 | |
| Telugute | 24,426 | 36,292 | 0.673041 | |
| Kannadakn | 9,589 | 12,293 | 0.780037 |
Counts each contiguous Unicode letter/mark/number run as one unit and each visible non-space punctuation/symbol character as one unit.
02 / Calculation
The score, shown in full
No clipped page and no hidden corpus. Every input and calculation is downloadable.
tokens ÷ faithful units0.780037 − 0.590242 = 0.1897951000 ÷ 0.189795 = 5268.8403 / Method
Exactly what was trained
The downloadable artifact is a standard Hugging Face Tokenizers JSON with its model, vocabulary, merges, normalizer, pre-tokenizer, and decoder intact.
- Model
- BPE · 10,000 vocabulary entries
- Normalization
- NFKC only
- Boundary handling
- Metaspace with
▁ - Training weights
- en:3 · hi:4 · te:4 · kn:5
- Wikipedia pages
- India · भारत · భారతదేశం · ಭಾರತ
- Corpus form
- Wiki-faithful Markdown, including links, tables, and references
python -m pip install -r requirements.txt
python build_wiki_faithful_markdown.py
python train_tokenizer.py
python evaluate_tokenizer.py04 / Vocabulary
Inspect every token
The list is read directly from tokenizer.json → model.vocab, ordered by token ID.
Loading vocabulary…
05 / Files
Take the evidence with you
Use the exact artifact and frozen corpus snapshots below to independently reproduce the reported result.