--- name: train-model description: Reproducible model training — eval-harness-first, fixed seeds, pinned environment, logged configs, checkpointing, experiment tracking. keywords: training, reproducibility, seed, checkpoint, mlflow, wandb, fine-tune --- # Train Model > **Eval-harness-first is the ML analog of `superpowers:test-driven-development`** — build the metric/eval harness and a reproducible baseline BEFORE writing any model improvement code. If the harness does not run clean, stop and fix it first. --- ## When to use this skill - Gate 3 of the ML workflow, after the experiment plan has been APPROVED - Any time a new model or training run is initiated - Fine-tuning or transfer-learning tasks where reproducibility is critical --- ## Steps ### 1. Eval-harness-first Implement the metric function and the validation loop as defined in the approved experiment plan. Run it against the baseline (majority class, heuristic, or current production model). Confirm you get a reproducible number before touching any model code. This number is the target to beat. ### 2. Enforce reproducibility Set all random seeds at the top of every training script: - `random.seed(seed)` and `numpy.random.seed(seed)` - Framework seed: `torch.manual_seed(seed)` + `torch.cuda.manual_seed_all(seed)` for PyTorch; `tf.random.set_seed(seed)` for TF/Keras - Enable deterministic ops where feasible (`torch.use_deterministic_algorithms(True)`) Pin the environment: `requirements.txt` or a lockfile (`pip freeze > requirements.txt`). No floating version ranges in the pinned file. ### 3. Config over hardcoding Place all hyperparameters (learning rate, batch size, number of layers, regularization strength, etc.) in a YAML, Hydra config, or argparse definition — never as literals in the training script. This ensures any run can be exactly reproduced from its config file alone. ### 4. Experiment tracking Log the following to MLflow or wandb for every run: - All hyperparameters (from config) - Metric per epoch / per fold - Seed, data version (DVC hash or path), and git commit SHA - Run name following the naming convention in `custom/rules/ml-conventions.md`: `[ticket-id]_[approach]_[yyyymmdd-n]` Do not leave runs unnamed or untracked. Output language: auto-detect from the ticket/task input — see `custom/rules/output-language.md` (Vietnamese input → Vietnamese output; otherwise English). ### 5. Checkpointing Save the best checkpoint by the primary validation metric (not the final epoch). Store the config file alongside the artifact so the checkpoint is self-contained. Use the model registry (MLflow Model Registry or equivalent) when available. ### 6. Framework-specific considerations - **scikit-learn:** wrap preprocessing + model in a single `Pipeline` so transforms are fitted on training data only — never on the full dataset. - **PyTorch:** seed `DataLoader` workers (`worker_init_fn`), toggle `model.train()` before training and `model.eval()` before validation/inference. Use `torch.no_grad()` for inference. - **TF/Keras:** use callbacks for `ModelCheckpoint` (save best) and `EarlyStopping`; set `tf.config.experimental.enable_op_determinism()` if needed. - **Fine-tuning:** apply a freeze/unfreeze schedule; use LR warmup before unfreezing. Log the freeze schedule as a config parameter. ### 7. Iterate per the ablation plan Follow the ablation plan from the experiment design — change exactly one factor per run, track it, compare to the established baseline on the same harness. Do not run ad-hoc experiments that are not tracked or not comparable. --- ## Completion Checklist - [ ] Eval harness implemented and confirmed running on baseline before any model change - [ ] Reproducible baseline number recorded - [ ] All random seeds set (language, numpy, framework, CUDA) - [ ] Environment pinned (requirements.txt or lockfile committed) - [ ] All hyperparameters in a config file — no literals in training code - [ ] Every run tracked with params, metrics, seed, data version, and git commit - [ ] Best checkpoint saved with its config alongside the artifact - [ ] Experiments follow the ablation plan (one change at a time)