# Changelog - M365 Copilot Eval Document Schema All notable changes to the eval document schema will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [1.10.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.9.0...schema-v1.10.0) (2026-08-18) ### Features * add evaluate-only mode for captured responses ([#541](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/541)) ([6c7b1f7](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/6c7b1f77a197f8317a76b50e93dbcb3a063787ee)) ## [1.9.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.8.0...schema-v1.9.0) (2026-07-29) ### Features * add categorized error codes for evaluator and agent failures ([#515](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/515)) ([8e01903](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/8e01903f9a5d677f932bea7b6f86e167f9507184)) * echo tags and extensions metadata verbatim into evaluation outputs ([#506](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/506)) ([1f806cc](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/1f806ccb086524eef22207a9a31361484064109a)) ## [1.8.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.7.0...schema-v1.8.0) (2026-07-20) ### Features * **citations:** detect markdown-link citation format ([N](url#suffix)) ([#480](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/480)) ([32601dc](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/32601dcd2e5b047d4214a84f43486270fb3aebff)) ## [1.7.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.6.0...schema-v1.7.0) (2026-07-10) ### Features * support non-LLM custom evaluators ([#442](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/442)) ([be498e1](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/be498e19257686f55a693b0ede9d16814f1996f4)) ## [1.6.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.5.0...schema-v1.6.0) (2026-06-15) ### Features * **evaluators:** add retrieval diagnostics to evaluation output ([#401](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/401)) ([e3f4f78](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/e3f4f7898f2dd9fb39ef0d45d9a1da67756c1427)) * **evaluators:** use LLM entailment judge for retrieval extract assertions ([#404](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/404)) ([9602168](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/9602168ecfc7911e42a75c78d51219b2404ba6ba)) * support custom evaluators ([#378](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/378)) ([d85afdc](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/d85afdcc81e1c4b0ea4dbbcd4391dd19a0abb3ea)) ## [1.5.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.4.0...schema-v1.5.0) (2026-05-27) ### Features * **evaluators:** Add retrieval query and retrieval result evaluators ([#360](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/360)) ([af42985](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/af42985598388e039e343866e9bca96e8f32b693)) ## [1.4.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.3.0...schema-v1.4.0) (2026-05-14) ### Features * unified error reporting in evaluation output ([#288](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/288)) ([a1acfce](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/a1acfce24a558bf855788d552266bc87cecf3251)) ## [1.3.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.2.0...schema-v1.3.0) (2026-04-30) ### Features * Added similarity evaluator for compatibility with MCS Evals. ([#228](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/228)) ([0fe8315](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/0fe8315abc8e0422d1ac9117fe9f29195f29044f)) ## [1.2.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.1.0...schema-v1.2.0) (2026-04-22) ### Features * **schema:** add multi-turn evaluation support (v1.2.0) ([#208](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/208)) ([a5ad22b](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/a5ad22bb4f6ac8ba548dc7f431ace073fa5970ce)) ## [1.1.0](https://github.com/microsoft/M365-Copilot-Agent-Evals/compare/schema-v1.0.0...schema-v1.1.0) (2026-03-30) ### Features * **WI-6855059:** add agentName/cliVersion to schema, fix duplicate prompt loss, include default_evaluators in output ([#181](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/181)) ([9321474](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/93214746144e9d11f507433eff185aefac4a858a)) * **WI-6855059:** implement per-prompt evaluator configuration ([#168](https://github.com/microsoft/M365-Copilot-Agent-Evals/issues/168)) ([eface7e](https://github.com/microsoft/M365-Copilot-Agent-Evals/commit/eface7e7041b118681cd4c68582fe903640bf6c0)) ## [1.0.0] - 2026-02-19 ### Added - Initial schema version for eval document contract - Root document structure: `schemaVersion` (required), `metadata` (optional), `items` (required) - `DocumentMetadata` with `name`, `description`, `createdAt`, `createdBy`, `evaluatedAt`, `tags`, `agentId`, and `extensions` - `EvalItem` with `prompt` (required), `expected_response`, `response`, `context`, `citations`, `scores`, and `extensions` - `ScoreCollection` with `relevance`, `coherence`, `groundedness`, `toolCallAccuracy`, and `citations` scores - `EvalScore` standard score structure (1-5 scale) with `score`, `result`, `threshold`, `reason`, `evaluator` - `CitationScore` for citation-specific evaluation with `count`, `result`, `threshold`, `format`, `citations` - `Citation` reference object with `index`, `text`, `source` - Extension points at `metadata.extensions` and `items[].extensions` for forward compatibility - `additionalProperties: true` on all objects for forward compatibility within major version