ARCHITECTURE & PRETRAINING

Phase 2 & 3 — Two Generations of Turkish Transformers: 24.7M (v1) & 67.6M (v2) from Scratch

February 2026 • Independent Research • IN PROGRESS

1. MOTIVATION: FROM TOKENIZER TO MODEL

Phase 1 produced a 64K Turkish BPE tokenizer that is ~14% more efficient than existing Turkish tokenizers and ~2.7× more efficient than GPT-4 on Turkish text. The natural question follows: can this tokenizer serve as the foundation for a purpose-built Turkish language model?

The objective is not to compete with production LLMs. The objective is threefold: (1) validate the tokenizer in an end-to-end training pipeline, (2) establish every component of the training infrastructure from scratch, and (3) produce models that demonstrably learn Turkish language patterns and can serve as domain-specific RAG assistants. This led to two generations: v1 (24.7M params) to prove the pipeline works, and v2 (67.6M params) purpose-built for RAG context comprehension.

2. V1 ARCHITECTURE OVERVIEW

The v1 model is a decoder-only Transformer with 24,697,088 parameters. Every component was selected against specific alternatives; no default was accepted without justification. All architectural decisions below (ALiBi, GQA, SwiGLU, RMSNorm, weight tying) carry forward to v2 — only the dimensions change.

Configuration

3. POSITIONAL ENCODING: ALiBi (NOT ROPE)

Positional encoding informs the model where tokens are in the sequence. The dominant approach in 2024–2026 is Rotary Position Embeddings (RoPE), used by Llama, Mistral, and Qwen. This work rejects RoPE in favor of ALiBi (Attention with Linear Biases, Press et al. 2022).

Why not RoPE

RoPE applies rotation matrices to query and key vectors based on position. While mathematically elegant, it exhibits two practical problems: (1) poor extrapolation beyond training length without ad-hoc scaling hacks (NTK-aware, YaRN, etc.), and (2) additional learned parameters that interact with the attention computation in ways that are difficult to debug at small scale. Prior empirical observation on this project confirmed unstable long-context behavior with RoPE.

ALiBi mechanism

ALiBi adds a linear distance penalty directly to attention scores. No parameters are learned. Each attention head receives a different slope (geometric sequence), creating a spectrum from sharp local attention to broad distant attention.

Slope computation

For n attention heads, slopes form a geometric sequence: slope_i = 2^-8i/n for i ∈ {1, ..., n}. With 8 heads, slopes range from 0.5 (sharp, local focus) to 2^-8 ≈ 0.0039 (broad, distant attention).

4. GROUPED QUERY ATTENTION

Standard Multi-Head Attention (MHA) assigns independent Key and Value projections to each attention head. Grouped Query Attention (GQA, Ainslie et al. 2023) shares KV projections across groups of query heads, reducing memory and parameter cost without proportional quality loss.

The 4:1 ratio saves ~98K parameters per layer × 12 layers = 1.18M parameters total versus MHA. At 24.7M total, this represents a 4.8% budget reallocation. Four query heads share each KV head via tensor repetition (_repeat_kv): the KV tensor is expanded along the head dimension without copying data.

Projection dimensions

5. SwiGLU FEED-FORWARD NETWORK

The standard Transformer FFN uses two projections with a nonlinear activation: FFN(x) = W₂ · σ(W₁ · x). SwiGLU (Shazeer 2020) replaces this with a gated variant using three projections: FFN(x) = W_down · (SiLU(W_gate · x) ⊙ W_up · x).

Parameter equivalence

Standard FFN with expansion factor 4: 2 × d × 4d = 8d². SwiGLU with 3 matrices: 3 × d × d_ff. Setting 3 × d_ff = 8d gives d_ff = (8/3)d ≈ 688 for d = 256. Total FFN parameters per layer:

Nearly identical parameter budget (+0.8%), empirically superior activation function. The gating mechanism allows the network to selectively amplify or suppress information — a capability that standard FFNs lack.

6. NORMALIZATION & RESIDUAL DESIGN

RMSNorm over LayerNorm

LayerNorm (Ba et al. 2016) performs two operations: mean subtraction and variance normalization. RMSNorm (Zhang & Sennrich 2019) drops the mean subtraction entirely, computing only the root mean square. With 12 layers × 2 norms per layer = 24 norm operations per forward pass, the cumulative speedup is measurable. Each RMSNorm has exactly d_model = 256 learnable parameters (the scale vector).

Pre-norm residual structure

Each Transformer block follows the pattern:

x = x + Attention(RMSNorm(x))     # residual path is unobstructed
x = x + FFN(RMSNorm(x))           # gradient flows directly through addition

The alternative — post-norm — places the normalization after the residual addition: x = RMSNorm(x + Attention(x)). This creates a gradient bottleneck through the norm layer. Pre-norm eliminates this bottleneck, producing more stable training in deep networks without requiring careful learning rate tuning.

7. V1 PARAMETER BUDGET ANALYSIS

At 24.7M parameters, every allocation decision is visible. The embedding layer dominates — a structural consequence of pairing a large vocabulary (64K) with a small hidden dimension (256).

Weight initialization

All 2D weight matrices are initialized with Normal(0, 0.02). Output projections (o_proj, down_proj) are additionally scaled by 1/√(2 × n_layers) = 1/√24 ≈ 0.204 to prevent residual stream explosion through 12 layers of additive contributions. This follows the GPT-2 initialization scheme.

8. TRAINING CORPUS CURATION

The full Phase 1 corpus spans 22 GB across 11 domains. For pretraining a 24.7M model, a curated subset of ~440 MB (raw text) was selected. The selection criteria: (1) prioritize raw text over structured data, (2) include reasoning-oriented material, (3) exclude instruction-tuning data (reserved for SFT in Phase 4), (4) cap Wikipedia to prevent encyclopedic dominance.

Excluded data in Round 1 (later used in Round 2)

9. DATA PIPELINE

The pipeline converts raw text to training batches in two offline steps, followed by real-time random sampling during training.

Step 1: Tokenization (offline, one-time)

Raw .txt files are split into documents by double newline, filtered (minimum 20 characters, 5 tokens), wrapped with BOS/EOS markers, and tokenized with the 64K BPE tokenizer. The resulting integer sequence is stored as a contiguous uint16 binary file (190 MB). The choice of uint16 is exact: vocab_size = 64,000 < 65,535 (max uint16), using exactly 2 bytes per token with no waste.

Step 2: Memory-mapped random sampling

The binary file is memory-mapped via numpy.memmap. Each training sample is drawn by selecting a random starting position and extracting a contiguous window of seq_len + 1 = 513 tokens. The first 512 tokens serve as input; the last 512 (shifted by one) serve as targets. This approach has no concept of epochs — with 99.6M tokens and 512-token windows, the number of possible starting positions (~194K) vastly exceeds any practical training run.

10. V1 TRAINING CONFIGURATION

Effective batch size

11. VALIDATION SUITE: 60 PARANOID TESTS

Before committing compute to training, every component of the pipeline was validated by a 60-test suite covering 13 categories. The suite was designed to catch the class of bugs that produce models which appear to train normally but fail silently.

12. INFRASTRUCTURE: MPS TO H100

Training was initiated on Apple M4 (MacBook Air, passive cooling) to validate the pipeline, then migrated to NVIDIA H100 NVL on RunPod for production training.

Migration changes

Three modifications were required to move from MPS to CUDA:

• Precision: float32 → bfloat16 via torch.amp.autocast. bfloat16 has the dynamic range of float32 (8 exponent bits) with the memory footprint of float16 (16 bits total), eliminating the need for loss scaling.
• Compilation: torch.compile(model) JIT-compiles the model graph into fused CUDA kernels, eliminating Python overhead and enabling kernel fusion across operations.
• Batch scaling: Physical batch increased from 4 to 128; gradient accumulation removed. The H100’s 95 GB VRAM accommodates the full effective batch in a single forward pass.

13. V1 ROUND 1 RESULTS (50K STEPS, 440 MB)

COMPLETE — 50,000 steps, 2.0 hours, final loss 2.62. This section documents Round 1 on the ~440 MB curated subset. Round 2 on the full 22 GB corpus is documented in Section 14.

Loss curve

Generated samples across training

Three hardcoded prompts (“Merhaba”, “Türkiye”, “Stok”) were sampled every 1,000 steps at temperature 0.8 with top-k 40. The prompt is selected randomly each time. No rules were programmed; all linguistic structure was learned from next-token prediction alone.

14. V1 ROUND 2: FULL CORPUS (228K STEPS, 22 GB)

Round 1 demonstrated the pipeline worked and the model could learn Turkish from a ~440 MB subset. Round 2 scaled to the full 22 GB corpus — the same corpus used to train the 64K tokenizer in Phase 1 — across all 11 domains: general knowledge, academic, legal, medical, financial, education, news, code, literary, reasoning, and instructions.

Why Round 2?

Round 1 trained on 99.6M tokens (441 MB subset, 67.7% Wikipedia). The model learned grammar and basic vocabulary but lacked domain diversity. Round 2 exposed the model to legal Turkish (court decisions), medical terminology, financial reporting, news journalism, academic writing, and instruction-following patterns — preparing a more robust base for SFT specialization.

Training data: full corpus

Round 2 configuration changes

Loss curve: Round 2

Sample evolution: Round 2

15. ROUND 2.5: 2048-CONTEXT FOR RAG (228K STEPS)

Round 2 trained with max_seq_len=512, inherited from the initial architecture. However, the RAG use case requires processing system prompt + retrieved context chunk + user question + generated answer in a single sequence. Typical RAG prompts consume 600–1,300 tokens. A 512-token model cannot serve RAG — so Round 2.5 retrained the same 24.7M model with 2048-token context.

Why Round 2.5?

Round 2’s 512-token context is a hard ceiling for downstream tasks. The SFT phase needs to fit:

• System prompt (~38 tokens): ERP sistemi asistanısın. Verilen bağlam bilgilerini kullanarak soruyu yanıtla...
• Retrieved context chunk (200–800 tokens): ERP documentation from the RAG retriever
• User question (20–60 tokens): Natural Turkish query
• Generated answer (50–300 tokens): Model’s response

Total: 308–1,198 tokens per turn. A 512-token model would truncate most inputs. ALiBi’s extrapolation property helps, but training at the target context length yields far better attention patterns. 2048 tokens provides comfortable headroom for even the longest multi-chunk RAG prompts.

Configuration changes from Round 2

Loss curve: Round 2.5

16. V2 ARCHITECTURE: 67.6M RAG-OPTIMIZED MODEL

The v1 model (24.7M params) proved that the training pipeline, tokenizer, and infrastructure work. But 24.7M parameters is severely capacity-limited for a RAG assistant that must read context, understand questions, and generate coherent answers. The v2 architecture was designed from scratch with a single principle: this model is a context converter, not a knowledge base.

Architecture comparison: v1 vs v2

Parameter budget: v2

Per-layer breakdown (2,900,992 params)

V2 training configuration

17. REPRODUCIBILITY & EXPERIMENT LOG

Every file, command, and decision is documented below to enable exact reproduction and — equally important — to prevent re-running experiments that were already tried.

17.1 File inventory

17.2 Checkpoint inventory

17.3 Reproduction commands

Step 1: Prepare Round 1 data (local)

python -m tiny_llm.data                    # tokenizes 8 files → tiny_llm/data/train.bin (190 MB, ~2.5 min)

Step 2: Run validation suite

python -m tiny_llm.test_everything          # 60 tests, all must pass before training

Step 3: Round 1 training (RunPod H100)

# Upload project to RunPod, then:
python -m tiny_llm.train --resume None      # 50K steps, ~2.0 hours, loss → 2.62
                                             # Config: batch=128, lr=3e-4→3e-5, bfloat16, torch.compile

Step 4: Prepare Round 2 data

python -m tiny_llm.data_full                # tokenizes all 27 files (22 GB) → train_full.bin (streaming, ~15 min)

Step 5: Round 2 training (RunPod H100)

nohup python -m tiny_llm.train \
    --resume tiny_llm/checkpoints/step_050000.pt \
    --max-steps 228000 \
    --data tiny_llm/data/train_full.bin \
    > training_round2.log 2>&1 &           # 228K steps, ~10.5 hours, loss → 3.46

Step 6: Round 2.5 training — 2048 context (RunPod H100)

# Modify config: max_seq_len=2048, dropout=0.02, batch_size=8, grad_accum=4
nohup python -m tiny_llm.train \
    --resume tiny_llm/checkpoints/step_228000.pt \
    --max-steps 228000 \
    --data tiny_llm/data/train_full.bin \
    --checkpoint-dir tiny_llm/checkpoints_2048 \
    > training_r25.log 2>&1 &              # 228K steps, ~26.5 hours, loss → 3.33

Step 7: V2 pretraining (RunPod H100)

nohup python -u -m tiny_llm.train_v2 \
    > training_v2.log 2>&1 &               # 228K steps, uses config_v2.py + model_v2.py

Step 8: Generate SFT data (local, API-based)

python -m erp_rag.generate.sft_generate \
    --provider anthropic \
    --model claude-sonnet-4-6                # 707 groups → ~8K-10K Q&A pairs, saves to sft_raw_pairs.json

17.4 Experiments tried & decisions locked

17.5 Training logs

18. PROJECT STATUS

© 2026 • Independent Research