--- name: evaluate-model description: Rigorous model evaluation — held-out metrics vs baseline, error analysis, and overfitting/leakage/drift detection with reproducibility verification. keywords: evaluation, error analysis, overfitting, leakage, robustness, model card --- # Evaluate Model > Run `superpowers:verification-before-completion` first — evaluation runs must reproduce cleanly before any success claim is made. If results cannot be reproduced from the logged seed and config, the evaluation is not valid. --- ## When to use this skill - Gate 4 of the ML workflow, after training and iteration are complete - Any time a claim about model quality needs to be verified before sharing with stakeholders - When preparing the eval report and model card for review --- ## Steps ### 1. Evaluate on the held-out test set Run inference on the test set — the split that was never touched during training or hyperparameter tuning. Report the primary metric agreed in Gate 1. Where feasible, include a confidence interval (bootstrap CI, or Wilson interval for classification). Report secondary metrics if relevant to the use case. ### 2. Compare to baseline and success threshold State pass or fail explicitly: - Did the model beat the baseline (majority class / heuristic / current production model)? - Did it meet the success threshold defined in Gate 1? A result that beats baseline but falls short of the threshold is a fail — report it honestly and return to Gate 3 or Gate 1 as appropriate. ### 3. Error analysis Identify where and how the model fails: - Break down the metric by slice (class, demographic, time period, data source, input length, etc.) - For classification: examine the confusion matrix; find the most common error patterns - Select representative failure cases and inspect them manually — understand the failure mode before claiming the model is production-ready ### 4. Overfitting check Compare train, validation, and test metrics side by side. Plot learning curves (metric vs training set size or epoch). Characterize the gap: - Large train/val gap → overfitting; the model did not generalize - Large train/test gap after acceptable val → distribution shift or leakage in val A large gap must be resolved or explained before the evaluation is APPROVED. ### 5. Leakage re-check Re-run all three leakage checks from `explore-data` on the final pipeline: 1. **Target leakage** — does any feature embed information derived from or computed after the target? 2. **Train/test contamination** — were any transforms fitted on data that includes test rows? Are there duplicate IDs across splits? 3. **Temporal leakage** — for time-ordered data, does any feature use information that would not have been available at prediction time? Document the result of each check. A positive finding is a Gate 1 regression, not a Gate 4 footnote. ### 6. Robustness assessment Test behavior under distribution shifts and edge cases relevant to the production use case: - Inject noise or missing values; check metric degradation - Test on known difficult slices (rare classes, short inputs, boundary values) - If the model affects decisions about people, check for disparate performance across sensitive groups and flag for review ### 7. Verify reproducibility Re-run the final eval using only the logged seed, the pinned config, and the registered data version. Confirm the reported number matches to within numerical noise. If it does not reproduce, stop — do not proceed to Gate 5 until the discrepancy is resolved. ### 8. Write the eval report and model card Output `AK-Docs/04.Coding/04.Reviews/[functionId]/[ticketId].md` (see `custom/rules/ml-conventions.md` — do NOT use the legacy `plan/[ticket-id]/` path) containing: the metric table (baseline, threshold, achieved), error analysis findings, overfitting gap, leakage check results, and robustness notes. Draft a model card covering: intended use, training data description, evaluation metrics and their context, known limitations, and owner/contact. Output language: auto-detect from the ticket/task input — see `custom/rules/output-language.md` (Vietnamese input → Vietnamese output; otherwise English). --- ## Completion Checklist - [ ] Held-out test metric reported with confidence interval where feasible - [ ] Explicit pass/fail against baseline and success threshold stated - [ ] Error analysis performed — worst slices, confusion patterns, representative failure cases - [ ] Train / validation / test gap inspected; learning curves reviewed - [ ] All three leakage types re-checked and documented (target, contamination, temporal) - [ ] Robustness assessed under noise, edge cases, and sensitive slices - [ ] Result reproduced from logged seed and config — numbers match - [ ] `AK-Docs/04.Coding/04.Reviews/[functionId]/[ticketId].md` (see `custom/rules/ml-conventions.md` — do NOT use the legacy `plan/[ticket-id]/` path) written - [ ] Model card drafted (intended use, data, metrics, limitations, owner)