BUILDING A TURKISH LLM FROM SCRATCH “?”

Project Context, Journey & What We Actually Discovered Along the Way

February 2026 • Independent Research • Living Document

1. WHY THIS EXISTS

Every major LLM is built on an English-centric foundation. When Turkish text passes through GPT-4’s tokenizer, it costs roughly 2.7× more tokens than it should. Turkish’s agglutinative morphology — where meaning is packed into chains of suffixes — is alien to tokenizers trained on English.

Existing Turkish LLMs (Kumru, Hamza, LlamaTurk, TURNA, and work from Boğaziçi and ODTÜ) represent serious efforts with meaningful results. Some train custom tokenizers, some build from scratch, some extend multilingual bases. After examining them closely, the honest takeaway is: good work exists, but each makes different tradeoffs — and none of them gave us the full-stack understanding we were after. We wanted to build every layer ourselves, not because existing work is bad(except kumru, it is fundamentally broken), but because the process of building is where the learning happens.

2. THE NORTH STAR: REASONING

The primary goal is not knowledge coverage, not chat fluency, not benchmark scores. It is reasoning — extreme logic capabilities. The model must:

• Understand — parse the input, identify what’s being asked
• Decompose — break the problem into pieces
• Reason step by step — apply logical structure (if A→B and B→C, then A→C)
• Self-check — detect inconsistencies and correct course
• Act like a scientist — not mimic one, but reason like one from the inside

Even if the model doesn’t know many facts, it must reason correctly about whatever it does know. Facts can be retrieved; reasoning structure cannot.

3. THE JOURNEY SO FAR

Roadmap

4. PHASE 1: THE TOKENIZER — WHERE EVERYTHING BEGAN

What started as “just build a tokenizer” became a deep exploration of how language is represented as numbers, why English-centric design hurts every other language, and how data and vocabulary interact in surprising ways. (Full tokenizer report →)

Three discoveries that changed our understanding

The tokenizer phase taught us: representation is everything. Before a model can reason about Turkish, it must be able to efficiently read and write it. A bad tokenizer is like trying to think through a straw — you can still get some signal through, but you’re wasting most of your capacity on the bottleneck.

Artifacts: EN report • TR report • benchmark_tokenizers.py (104 sentences) • train_tokenizer.py • tokenizers/turkish_bpe_64k/

5. THE DOOR THE TOKENIZER OPENED

The tokenizer phase gave us a working 64K Turkish BPE. But the deeper gift was something nobody expected: a complete shift in how we see AI, language, and the industry itself. This is the most important thing we’ve learned in this entire project so far.

What is a tokenizer, really?

Strip away the jargon. A tokenizer does one thing: it converts structured input into a sequence of numbers. We happened to use Turkish text as input. But nothing in the algorithm requires that input to be human language.

When we trained our 64K BPE, the algorithm didn’t “know” it was processing Turkish. It saw byte sequences, found frequently co-occurring patterns, and merged them into tokens. The output was a mapping: input patterns → integer IDs. That’s it. The algorithm doesn’t care whether those patterns are Turkish suffixes, musical notes, or chemical bonds.

Once you truly internalize this, a door opens that never closes.

The question that changed everything

Music is a language. Notes are the vocabulary. Chord progressions, scales, rhythm patterns, key signatures — these are the grammar. A melody is a “sentence.” A symphony is a “document.” A “tokenizer” for music maps note events (pitch, duration, velocity, chord) to integer IDs. A transformer trained on those token sequences learns: after this chord progression, this resolution is likely. After this rhythmic pattern, this variation follows. The transformer doesn’t know it’s “making music.” It is predicting the next token — exactly as it does with Turkish words.

Proteins are a language. Amino acids are the vocabulary — just 20 base characters. Proteins are “sentences”: sequences that fold into 3D structures governed by physical rules. The “grammar” dictates which sequences form alpha helices, which form beta sheets, which combinations bind to specific receptors. A transformer trained on protein sequences learns this grammar — not because it understands biology, but because it finds statistical patterns in token sequences. This is literally how AlphaFold-class models work.

Chemical formulas are a language. SMILES notation encodes molecular structures as text strings. Atoms and bonds are the vocabulary. Valence rules, ring structures, functional groups — these are the grammar. A “tokenizer” maps chemical symbols to integers. The transformer learns: after this molecular fragment, this binding property is likely. Drug discovery models already work this way.

DNA is a language. Four nucleotides — A, T, C, G — that’s the entire vocabulary. Codon triplets encode amino acids. Regulatory regions control gene expression. Genomic models tokenize these sequences and learn to predict mutations, gene function, even disease risk. A vocabulary of 4, grammar encoded by billions of years of evolution.

A factory production line is a language. Material codes, machine settings, environmental conditions, test results — these form sequences with causal structure. The “vocabulary” might be 500–2000 tokens. The “grammar” is the physical causality: PVC_compound_A + temp_175 + speed_15 → tensile_PASS + shore_85. A 50-million-parameter model can learn to predict production outcomes before a single meter of cable is manufactured — saving material, energy, and time.

The five-step chain that opens every door

1. A tokenizer is just: structured patterns → numbers
2. A transformer is just: learn to predict the next number given previous numbers
3. “Language model” is just what we call it when those numbers happen to represent words
4. ANY sequential structured data can be tokenized
5. Therefore: the transformer is a universal sequence learner, not a “language” model

Every domain is a language

The cable factory — a concrete example

This isn’t hypothetical. Every cable factory generates data like this every day:

• Tokenizer vocabulary: ~500–2000 tokens (material codes, machine settings, test result codes)
• Input: [MATERIAL] PVC_compound_A [SETTINGS] temp_175 speed_15 pressure_8
• Output: [RESULTS] tensile_pass elongation_420 flame_V0 shore_hardness_85
• Model size: 10–50M parameters. Trains in hours on a single GPU.
• Value: Predict test results before wasting material on a production run

This model would be more accurate than GPT-4 for this specific task, orders of magnitude cheaper, runs on a laptop, keeps your proprietary data private, and was built using the exact same skills we’re learning by building a Turkish LLM: tokenizer design, architecture selection, training pipeline optimization.

What this means: the doors that opened

The moment we understood this, the project’s scope transformed from “build one Turkish LLM” to “learn to build any sequence model for any domain.” The possibilities:

• Every factory, every lab, every hospital, every trading desk has sequential data
• Each could have its own tiny model (10M–100M parameters)
• These models would be more accurate than general LLMs for their specific domain
• Cheaper to train (hours, not months), cheaper to run (laptop, not data center)
• Private — your data never leaves your building
• And we now know how to build them — because the LLM project teaches the entire craft

The trap: building gods instead of tools

The AI industry is pouring billions into building an omniscient conversational entity — a digital god that answers everything through natural language. Every problem becomes “talk to the AI.”

But now we see clearly: most valuable real-world problems don’t need conversation. They need prediction, pattern recognition, optimization. The “talking” layer is expensive overhead when your actual need is “will this cable pass the tensile test?”

Building a full LLM when you need a domain predictor is like building a 747 when you need a bicycle. The bicycle is simpler, cheaper, and gets you where you’re going faster — if you’re going to the shop.

The orchestra vision

The future is not one massive model. It is orchestration: multiple small, specialized models working together, each optimal for its domain.

We are building the reasoner. The specialists can be the factory models, the medical models, the financial models — each tiny, each accurate, each built with the same skills we are learning right now.

6. HOW IT ACTUALLY WORKS: STEP BY STEP, DOMAIN BY DOMAIN

Section 5 claimed every domain is a language and every sequence can be tokenized. That might still feel abstract. So let’s make it concrete. Below are mock walkthroughs showing exactly what happens inside the machine — from raw input to final output — for five different domains. The process is identical every time. Only the tokens change.

① Language model (Turkish LLM)

② Music model

③ Protein model

④ Cable factory model

⑤ DNA / Genomics model

7. WHAT THE ARCHITECTURE TAUGHT US ABOUT REASONING

If Sections 5 and 6 showed us that the transformer is a universal sequence learner — same four steps, any domain — this section asks: how does a sequence learner develop something that looks like reasoning? Understanding architecture required understanding what “reasoning” actually means inside a neural network — and what it doesn’t. Remember: everything below applies not just to LLMs, but to any sequence model — the same mechanisms that let a language model “reason” about Turkish let a protein model “reason” about folding. (A detailed architecture research page will follow, like the tokenizer report.)

The training pipeline (sequential, not a choice)

These are not alternatives. They’re sequential phases, each teaching fundamentally different things:

What generalizes vs what doesn’t

How self-correction works (mechanistically)

An LLM doesn’t “realize” errors the way we do. At each token position, the attention mechanism can attend to all previous tokens. As more context accumulates, inconsistencies become statistically detectable — the probability distribution shifts toward correction tokens. “Backtracking” is not real backtracking: the model generates new tokens that redirect (“wait, that’s wrong…”). The wrong tokens remain in context.

This self-correction ability comes from RL training, not from seeing error-correction patterns in data. RL rewards reasoning chains that self-correct AND reach correct answers. The model discovers that “check your work” is a rewarding strategy.

Q: Is it reasoning, or is it mimicry?

The honest answer: we don’t know. The model learns reasoning patterns from data. When it encounters a new problem, it applies those patterns. Is that “real reasoning” or “sophisticated pattern matching”? The debate is unresolved. Evidence is mixed: models solve novel problems (suggesting generalization beyond mimicry) but also fail on trivially modified versions of problems they ace (suggesting pattern matching).

Our practical answer: the distinction may not matter. What matters is: does the model arrive at correct answers on novel problems? That’s measurable. RLVR pushes the model from shallow mimicry toward robust application by rewarding correctness, not looking correct.

Q: What is “correct” for a language model?

Q: LLM reasoning = search algorithms?

An insight from the architecture discussion: LLM self-correction resembles tree search (explore paths, evaluate, redirect). But with critical differences:

1. The tree doesn’t exist beforehand — it’s generated token by token
2. No real backtracking — only forward corrections (“wait, that’s wrong…”)
3. For general language, there is no “correct” node — only for verifiable domains (math, code, logic)

Research formalizes this as Tree of Thoughts, Process Reward Models, and MCTS for LLMs. The analogy holds structurally but breaks mechanistically. Still, it implies: small models can “search” well if given sufficient thinking budget (extended thinking = bigger search budget).

8. DESIGN PHILOSOPHY: LESS IS MORE

We are not committed to a single scale. We are open to 100M, 360M, 1B, 2B, 3B, 4B — and “less is not limited.” The belief: with extremely optimal architecture and pretraining, smaller models can match or approach bigger ones.

As we discovered in Section 5, this extends far beyond LLMs. Every domain with sequential structure can have its own tiny, accurate model. The world is moving toward specialized models combined into orchestras — and we are positioned to build them.

9. IMPORTANT DECISIONS (LOCKED)

10. DATA STRATEGY

This section focuses on data for our Turkish LLM. But through the lens of Section 5: everything described below is a template. Swap “Turkish text” for “protein sequences” or “production logs,” and the same pipeline structure applies — just with a different tokenizer and different domain data.

Pretraining data — “training” (quantity, diverse)

Raw Turkish text — no QA pairs, no formatting. The model reads continuous text and predicts the next token at every position.

SFT data — “fine-tuning” (quality, clean)

Turkish instruction-response pairs. Clean, no mistakes. Teaches format, not reasoning.

RLVR data — reinforcement learning (verifiable problems)

Math (GSM8K-style, competition math), code problems, logic puzzles. Can be translated to Turkish. Math and logic are language-light — 17 × 23 = ? works in any language. This is where reasoning is actually trained.

11. WHAT’S NEXT

12. REPO SNAPSHOT

© 2026 • Independent Research • Tokenizer Report • Tokenizer Raporu (TR)