SUPERVISED FINE-TUNING (SFT)

Phase 4 — From 3,790 to 10,000 QA Pairs: Two Generations of ERP Assistant Training

February 2026 • Independent Research • IN PROGRESS

1. THE FULL PIPELINE

The model follows a standard modern LLM training pipeline. Each phase builds on the previous one, progressively narrowing the model’s capabilities from general language understanding to domain-specific instruction following.

Two generations of SFT are documented here: v1 (24.7M, proof of concept) and v2 (67.6M, production pipeline).

2. PRETRAINING HISTORY

The v1 model went through three pretraining rounds before SFT. Round 2 (shown below) established deep language understanding. Round 2.5 later extended context to 2048 tokens for RAG. The v2 model (67.6M) is a separate architecture pretrained from scratch. Full pretraining details are in the Architecture & Pretraining report.

Loss curve progression

Sample evolution during Round 2

3. ERP DOCUMENTATION SOURCE

The SFT training data was generated from the Solen Kablo ERP system documentation — the same system the model is being built to assist with. The documentation was already pre-processed into structured chunks with rich metadata as part of a RAG (Retrieval-Augmented Generation) pipeline.

Modules covered

Chunk metadata

Each chunk contains structured metadata used to guide the QA generation:

4. SYNTHETIC DATA GENERATION (V1)

The central challenge of SFT for a domain-specific model is data. Hand-writing thousands of QA pairs is impractical; using generic instruction datasets would not teach the model about Solen Kablo’s ERP system. The solution: use a large cloud LLM as a data conversion tool — not a knowledge source — to transform existing ERP documentation into training-ready QA pairs.

V1 design decisions

5. PROMPT ENGINEERING (V1)

The v1 generation pipeline uses a two-part prompt: a comprehensive system prompt (the “Grand Prompt”) that defines the task, formats, rules, and examples; and a per-chunk user message that provides metadata and the documentation text. The v2 prompt is significantly redesigned — see Section 13.

System prompt structure

Answer length calibration for v1 (26M parameters)

Bad vs good question examples (from the prompt)

6. V1 QA PAIR STATISTICS

The v1 generation script processed 320 out of 529 Turkish chunks before API credits were exhausted (~$20 on Claude Opus 4.5). The resulting dataset was sufficient for the 26M parameter model. For v2 statistics (~8–10K pairs from 707 groups), see Section 13.

Difficulty distribution

Generation cost

7. CHAT TEMPLATE & TOKENIZATION

The SFT data uses a Llama 3–style chat template, leveraging the special tokens already built into the tokenizer during Phase 1. This was a deliberate design decision: the tokenizer was built with instruction-tuning tokens before the model existed, anticipating this exact use case.

Special tokens used

Template structure

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Sen Solen Kablo ERP sisteminin yapay zeka asistanısın...
<|eot_id|><|start_header_id|>user<|end_header_id|>

QR kod ne işe yarar?
<|eot_id|><|start_header_id|>assistant<|end_header_id|>

QR kod her hammaddeye atanan benzersiz bir takip kodudur...
<|eot_id|>

Multi-turn extension

For multi-turn conversations (2–3 turns), the template simply repeats the user/assistant blocks. Each turn ends with <|eot_id|>, and the model learns to generate until it produces this token.

8. LOSS MASKING STRATEGY

A critical detail of SFT: loss is computed only on assistant tokens. The model must learn to predict assistant responses, but it should not be penalized for failing to predict the system prompt or user questions (which are given as input, not generated).

Token-level loss mask

Token sequence:
[BOS] [HEADER:system] system content... [EOT]
      [HEADER:user]   user question... [EOT]
      [HEADER:asst]   assistant answer... [EOT]

Loss mask:
  -1    -1  -1  -1  -1  ...  -1     ← system (ignored)
  -1    -1  -1  -1  -1  ...  -1     ← user (ignored)
  -1    -1  YES YES YES ... YES    ← assistant content + EOT (trained)

The ignore_index=-1 parameter in PyTorch’s cross_entropy function handles this natively. Positions marked -1 contribute zero to the loss. Only the assistant’s content tokens and the trailing <|eot_id|> are trained on.

9. V1 SFT TRAINING CONFIGURATION

10. V1 RESULTS

Per-epoch breakdown

Train loss is higher than val loss because the training loss is computed per-batch during gradient updates (with dropout active), while validation loss is computed over the full validation set with dropout disabled.

11. V1 MODEL OUTPUTS

Before SFT (pretrained only)

The pretrained model has no concept of the chat template, question-answering, or ERP knowledge. Given the structured prompt, it falls into degenerate repetition.

After SFT (epoch 3)

12. ANALYSIS & LESSONS LEARNED

What v1 SFT proved

• Turkish grammar and morphology are solid — learned during pretraining, preserved through SFT
• Domain vocabulary (hammadde, sipariş, tedarikçi, malzeme) is correctly used
• Chat template pattern (system → user → assistant) is reliably followed
• The model stops generating at the right point (produces <|eot_id|>)
• SFT on a tiny model is fast (<2 minutes) and cheap (<$0.10), enabling rapid iteration

V1 limitations that drove the v2 redesign

Key insight: SFT is the critical phase at this scale

13. V2 SFT PIPELINE REDESIGN

The v1 SFT (3,790 pairs from Claude Opus 4.5) proved the concept but exposed limitations: only 320 of 529 chunks were processed before API credits ran out, the grouping strategy was basic (individual chunks only), and the 512-token context couldn’t fit real RAG prompts. The v2 pipeline was redesigned from scratch for the 67.6M RAG-optimized model.

Why redesign?

• Coverage gap: v1 only processed 60% of chunks. v2 processes all 532 chunks in multiple configurations.
• Single-chunk limitation: v1 generated Q&A from individual chunks only. v2 uses 11 grouping rules that combine chunks by submodule, module, cross-module, data flow, and more — producing questions that require synthesizing information across chunks.
• Model upgrade: v2 targets a 67.6M model with 2048-token context, enabling much richer RAG prompts with longer context chunks.
• API model quality: v1 used Claude Opus 4.5. A 4-way comparison showed Claude Sonnet 4.6 produces superior question diversity and inference depth.

Master grouping strategy: 11 rules

Each rule produces a different perspective on the same documentation, forcing diverse question types:

API model comparison: 4-way pilot

Before committing to a single API model for all 707 groups, a controlled pilot was run on the same 10 chunk groups with 4 different models:

V2 prompt engineering

The data generation uses a two-level prompt architecture:

1. SFT system prompt (embedded in every training example, 38 tokens):

ERP sistemi asistanısın. Verilen bağlam bilgilerini kullanarak
soruyu yanıtla. Bağlamda cevap yoksa "Bu konuda bilgim yok" de.

2. API system prompt (sent to Claude Sonnet 4.6, not seen by the model):

A detailed Turkish-language instruction set covering:

• 6 core rules: context-only answers, natural questions, complete/correct, use technical terms as-is, independent pairs, no repetition
• 5 question types: olgusal (factual), prosüdürel (procedural), karşılaştırmalı (comparative), çıkarımsal (inferential), liste (list)
• 4 answer formats: numbered steps for procedures, short paragraphs for definitions, bullet lists for enumerations, concise explanations for comparisons
• 3 difficulty levels: kolay ~40% (single-fact lookup), orta ~40% (multi-fact synthesis), zor ~20% (inference/comparison)
• Dynamic pair count: 5–28 pairs per group based on input token count

V2 generation: final results

Question type distribution

Difficulty distribution

Pairs by grouping rule

14. REPRODUCIBILITY

Complete file inventory

V1 SFT files:

V2 SFT files:

Shared files:

To reproduce

# V2 pipeline (recommended)

# 1. Generate QA pairs (requires Anthropic API key, ~$20-25)
pip install anthropic
export ANTHROPIC_API_KEY="sk-ant-..."
python -m erp_rag.generate.sft_generate \
    --provider anthropic --model claude-sonnet-4-6   # 707 groups → 7,595 pairs, auto-resume on interruption

# 2. Upload data to RunPod and run SFT training
python -m tiny_llm.train_sft_rag \
    --model v2 \
    --data erp_rag/data/sft_raw_pairs.json           # reads raw pairs, builds ChatML, trains with loss masking

# 3. Chat with the model
python -m tiny_llm.chat

Experiments tried & decisions locked

© 2026 • Independent Research