NATIVE TURKISH BPE TOKENIZER

Toward a 'Proper' Turkish Language-Native LLM: Phase 1 — Tokenization

February 2026 • Independent Research • COMPLETE

1. THE PROBLEM: THE LANGUAGE TAX

When Turkish text is processed by any major LLM, it passes through a tokenizer designed for English. Turkish's agglutinative structure — where meaning is packed into suffixes — is alien to these tokenizers.

Same sentence, different cost

Test sentence: “Türkiye Cumhuriyeti'nin başkenti Ankara'dır.”

This means every Turkish API call can cost roughly 2× more tokens. Context windows hold proportionally less Turkish text; training runs process fewer sentences per batch. The tax compounds with length.

2. RELATED WORK

Several Turkish language models and tokenizers exist. Hamza (Acikgoz, “Bridging the Bosphorus”) provides Turkish LLMs from 124M to 1.3B parameters, including models adapted from GPT-2 and Mistral; the Hamza tokenizer uses vocabulary size 50,257 — identical to GPT-2 — and is not optimized for Turkish morphology. TabiBERT (Boğaziçi Üniversitesi TabiLab) is a ModernBERT-based encoder trained on 1T tokens for Turkish NLP; vocabulary size 50,176. Kumru-2B uses a 50,176-vocabulary BPE tokenizer. LlamaTurk (METU NLP) adapts LLaMA with a 28K BPE tokenizer trained on Turkish OSCAR.

A consistent pattern is that existing Turkish decoder LLMs converge on ~50K vocabulary: 50,257 (GPT-2 size) when the base model is GPT-2, and 50,176 for others. This appears to follow from adaptation of English-base tokenizers rather than from systematic vocabulary-size experiments for Turkish. No prior work is known to the author that reports (1) the GPT-2 pre-tokenization regex breaking Turkish apostrophe suffixes, (2) vocabulary saturation versus data saturation in tokenizer training, or (3) systematic comparison of 16K–64K vocabulary on the same corpus.

3. NOVEL FINDING: GPT-2 REGEX BREAKS TURKISH

During development, it was discovered that the GPT-2 pre-tokenization regex — the same pattern used by GPT-4, GPT-4o, Llama 3, and Mistral — contains English contraction patterns ('s|'t|'re|'ve|'m|'ll|'d) that actively damage Turkish tokenization.

How GPT-4 tokenizes Turkish apostrophe suffixes

The fix: cleaned Turkish regex

4. ARCHITECTURE DECISIONS

Special tokens

5. TRAINING CORPUS: 3 ITERATIONS

The tokenizer was trained through three iterative rounds, each adding new data domains. This process revealed critical insights about the relationship between corpus diversity and tokenizer quality.

v1: Foundation (1.7 GB, 14 files)

v2: Quality Boost (10 GB, 16 files) — curated literary & academic data added

v3: Domain Coverage (22 GB, 27 files) — 7 new specialized domains

6. DATA & VOCABULARY SCALING EXPERIMENTS

Two systematic experiments were conducted: (1) scaling training data from 1.7GB to 22GB at fixed 48K vocabulary, and (2) scaling vocabulary from 48K to 64K on the full 22GB corpus. The combination revealed a critical insight about the interaction between data volume and vocabulary capacity.

Experiment A: Data scaling at 48K vocabulary

Totals above use a truncated sentence set. Section 7 reports the same tokenizers on full sentences (192 / 199 / 224).

v1→v2 (1.7GB → 10GB): +10.0% improvement. v2→v3 (10GB → 22GB): +0.9% improvement — apparent diminishing returns.

Experiment B: Vocabulary scaling — the breakthrough

The near-zero improvement from v2→v3 at 48K initially suggested data saturation. However, training a 64K tokenizer on the same v3 corpus revealed a fundamentally different result:

7. HEAD-TO-HEAD: 64K v3 vs TURKISH & ENGLISH TOKENIZERS

Comparison across tokenizers on 21 test sentences covering daily speech, formal language, agglutination, code, and six specialized domains. Turkish tokenizers (this work, Kumru, TabiBERT, Hamza) were evaluated on the same full-sentence set; GPT-4/GPT-4o use different tokenizers and are included for reference.

Totals from benchmark_tokenizers.py on 21 full sentences. Hamza uses GPT-2 tokenizer (50,257 vocab); Kumru and TabiBERT use ~50K BPE.

Observation: Kumru and TabiBERT produce identical token counts on every sentence in this benchmark (same vocabulary size 50,176; same total 224). Exact agreement across all 21 sentences is uncommon for independently trained BPE tokenizers. The finding is reported here without further interpretation.

Extended benchmark: 104 sentences (21 core + 83 hard/edgy)

The same tokenizers were run on an extended set: the 21 core sentences above plus 83 “hard” sentences (long agglutination, legal/medical/financial phrasing, colloquial/slang, numbers and dates, code snippets, punctuation and abbreviations, loanwords, case/diacritic edge cases). All counts from benchmark_tokenizers.py.

64K remains best on the extended set; Kumru and TabiBERT again match each other (1,198). The hard set includes e.g. Muvaffakiyetsizleştiricileştiriveremeyebileceklerimizdenmişsinizcesine, legal (HMK 353, tahkim), medical (pankreatikoduodenektomi, kardiyovasküler), financial (BIST 100, SPK), slang (N'olcak, bişey), code (return {'key': value}), and loanwords (Startup'lar, API endpoint'i).

8. DOMAIN-SPECIFIC ANALYSIS

Domain-targeted training data produces measurable improvements in specialized vocabulary handling. Below are token-level comparisons for each new domain.

Legal Turkish

başvuruyu (the application) and oybirliğiyle (unanimously) are each single tokens in 64K. Kumru and TabiBERT fragment the first into 2 pieces and the second into 3 pieces. reddetti (rejected) is also a single token — Kumru/TabiBERT need 2 (reddet | ti). Result: 6 vs 10 (Kumru/TabiBERT), 6 vs 23 (Hamza).

Medical Turkish

Hastanın (of the patient) is a single token. değerlendirilmelidir (must be evaluated) — a 6-morpheme suffix chain — is also a single token. Kumru/TabiBERT tie at 7; Hamza 30; GPT-4 26.

Financial Turkish

News/Journalism Turkish

Cumhurbaşkanlığı (Presidency) and Sözcüsü (Spokesperson) are each single tokens in 64K. Kumru and TabiBERT split Sözcüsü into 2 pieces; Hamza 39 tokens. The extra vocabulary capacity allows 64K to capture these high-frequency institutional terms as atomic units.

9. MORPHOLOGICAL ANALYSIS

The tokenizer learned Turkish morphology from pure statistics — no linguistic rules were programmed. BPE naturally discovered morpheme-like boundaries through frequency analysis of 22GB of text.

Verb morphology (learned, not coded)

Noun case suffixes

Six different grammatical forms of "ev" — all encoded as single tokens.

Suffix chain handling

10. DIACRITIC ROBUSTNESS

Turkish users sometimes type without diacritics (c instead of ç, s instead of ş, i instead of ı). The tokenizer handles both forms, but correct Turkish is significantly more token-efficient — by design.

11. CONTEXT WINDOW: THE COMPOUNDING ADVANTAGE

Tokenizer efficiency is not a fixed saving — it is a multiplier on context length. The longer the context window, the more the advantage compounds. This has direct architectural implications for the target model.

Effective context capacity

At each context length, the 64K tokenizer holds significantly more Turkish text than competitors could fit in the same number of token slots:

“Kumru Equivalent” = how many Kumru tokens would be needed to hold the same amount of Turkish text. Calculated from the efficiency gaps measured in Section 7.

Architectural implication: small model, massive context

The efficiency advantage fundamentally changes the optimal model architecture for Turkish. Two strategies were considered:

12. PRACTICAL IMPLICATIONS

Turkish text processed by English-centric tokenizers incurs a roughly 2× token penalty across context length, speed, cost, and training efficiency. A native Turkish tokenizer eliminates this tax entirely.

The tokenizer covers 11 specialized domains (general, academic, legal, medical, financial, education, news, code, literary, reasoning, instructions), ensuring efficient tokenization regardless of subject matter.

13. PROJECT STATUS

14. REPRODUCIBILITY

Code, data sources, and trained tokenizers are available.

• Training script: train_tokenizer.py
• Benchmark script: benchmark_tokenizers.py (104-sentence comparison: 21 core + 83 hard/edgy)
• Training data: 22 GB across 27 files, 11 domains
• Selected tokenizer: tokenizers/turkish_bpe_64k/tokenizer.json
• Versions preserved: 16K, 32K, 48K, 64K × v1/v2/v3
• Baselines: Kumru-2B (50,176), TabiBERT (50,176), Hamza (50,257, GPT-2 tokenizer), GPT-4 (cl100k_base), GPT-4o (o200k_base)

© 2026 • Independent Research