{ "schema": "comet.native.benchmark.v1", "title": "Comet Native beta16 vs beta17 benchmark", "generated_at": "2026-08-10", "model_provider": "cc-switch active Claude-compatible provider (model name intentionally omitted)", "task_scope": [ "B", "C", "D", "E", "F" ], "note": "Beta17 uses portable v4 validators; beta16 retains its published legacy validators. Scores are therefore directional by task, not a single protocol-equivalent percentage.", "reruns": { "beta17_E": "single-worker retry; rate-limited/timeout and excluded", "beta16_D": "single-worker retry; included", "beta16_E": "single-worker retry; rate-limited/timeout and excluded" }, "summary": { "beta16": { "protocol": "beta16 legacy validators", "usable_waves": [ "B", "C", "D", "F" ], "excluded_waves": [ "E" ], "checks_passed": 54, "checks_total": 73, "pass_rate": 0.7397, "duration_seconds": 3124.601, "cost_usd": 25.1271 }, "beta17": { "protocol": "beta17 portable v4 validators", "usable_waves": [ "B", "C", "D", "F" ], "excluded_waves": [ "E" ], "checks_passed": 58, "checks_total": 65, "pass_rate": 0.8923, "duration_seconds": 2544.023, "cost_usd": 21.079 } }, "waves": [ { "version": "beta17", "wave": "B", "label": "Decision / resume", "passed_checks": 13, "failed_checks": 1, "total_checks": 14, "passed": false, "sample_status": "included", "included": true, "reason": "Run completed with validator evidence", "turns": 124, "duration_seconds": 735.243, "total_tokens": 5121440, "cost_usd": 5.221953, "failures": [ "Artifact not found: docs/comet/specs/unique-word-counting/spec.md" ], "experiment": "beta17_native_bcdf_ds_20260810", "report_file": "comet_native_wave_b_decision_resume_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta17", "wave": "C", "label": "Verification integrity", "passed_checks": 18, "failed_checks": 0, "total_checks": 18, "passed": true, "sample_status": "included", "included": true, "reason": "Run completed with validator evidence", "turns": 109, "duration_seconds": 563.831, "total_tokens": 6978384, "cost_usd": 5.736421000000001, "failures": [], "experiment": "beta17_native_bcdf_ds_20260810", "report_file": "comet_native_wave_c_verification_integrity_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta17", "wave": "D", "label": "Stagnation stop", "passed_checks": 13, "failed_checks": 3, "total_checks": 16, "passed": false, "sample_status": "included", "included": true, "reason": "Run completed with validator evidence", "turns": 109, "duration_seconds": 858.118, "total_tokens": 8257707, "cost_usd": 7.366579, "failures": [ "stagnation_stop: The twelfth failure did not reach the v4 repair ceiling", "native_state: expected blocked status and loop stage", "native_loop: blocked state has no blocked loop stage" ], "experiment": "beta17_native_bcdf_ds_20260810", "report_file": "comet_native_wave_d_stagnation_stop_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta17", "wave": "E", "label": "Parallel safety", "passed_checks": 12, "failed_checks": 4, "total_checks": 16, "passed": false, "sample_status": "excluded", "included": false, "reason": "API rate limit or quota failure", "turns": null, "duration_seconds": null, "total_tokens": null, "cost_usd": null, "failures": [ "Artifact not found: .cache/comet-native-eval/conflict-status.json", "Artifact not found: .cache/comet-native-eval/mutation-attempt-a.json", "Artifact not found: .cache/comet-native-eval/mutation-attempt-b.json", "native_state: expected exactly one active Native change" ], "experiment": "beta17_native_wave_e_retry_ds_20260810", "report_file": "comet_native_wave_e_parallel_safety_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta17", "wave": "F", "label": "Dashboard read-only", "passed_checks": 14, "failed_checks": 3, "total_checks": 17, "passed": false, "sample_status": "included", "included": true, "reason": "Run completed with validator evidence", "turns": 74, "duration_seconds": 386.831, "total_tokens": 2919932, "cost_usd": 2.7540919999999995, "failures": [ "dashboard_readonly: Native tree manifest does not match live files", "native_state: no terminal Native archive exists", "native_loop: portable state is unavailable" ], "experiment": "beta17_native_bcdf_ds_20260810", "report_file": "comet_native_wave_f_dashboard_readonly_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta16", "wave": "B", "label": "Decision / resume", "passed_checks": 10, "failed_checks": 7, "total_checks": 17, "passed": false, "sample_status": "included", "included": true, "reason": "Run completed with validator evidence", "turns": 21, "duration_seconds": 155.921, "total_tokens": 646165, "cost_usd": 0.968647, "failures": [ "Artifact not found: .cache/comet-native-eval/archive-preview.json", "Artifact not found: .cache/comet-native-eval/archive-commit.json", "Artifact not found: docs/comet/archive", "Artifact not found: docs/comet/specs/unique-word-counting/spec.md", "native_artifacts: valid .comet/config.yaml is missing" ], "experiment": "beta16_native_bf_ds_20260810", "report_file": "comet_native_wave_b_decision_resume_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta16", "wave": "C", "label": "Verification integrity", "passed_checks": 16, "failed_checks": 1, "total_checks": 17, "passed": false, "sample_status": "included", "included": true, "reason": "Run completed with validator evidence", "turns": 123, "duration_seconds": 742.303, "total_tokens": 7626436, "cost_usd": 5.703676000000001, "failures": [ "verification_integrity: Invalid archived Native evidence: Verification envelope fields are invalid: missing=['receiptRef'], unknown=['receiptRefs', 'requiredReceiptRefs']" ], "experiment": "beta16_native_bf_ds_20260810", "report_file": "comet_native_wave_c_verification_integrity_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta16", "wave": "D", "label": "Stagnation stop", "passed_checks": 15, "failed_checks": 3, "total_checks": 18, "passed": false, "sample_status": "included", "included": true, "reason": "Run completed with validator evidence", "turns": 158, "duration_seconds": 1563.607, "total_tokens": 21762731, "cost_usd": 14.919573000000002, "failures": [ "stagnation_stop: The twelfth total failure is not the hard repair ceiling", "native_isolation: Forbidden workflow artifacts exist: ['.comet/current-change.json']", "native_isolation: Classic or hidden workflow artifacts exist" ], "experiment": "beta16_native_wave_d_retry_ds_20260810", "report_file": "comet_native_wave_d_stagnation_stop_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta16", "wave": "E", "label": "Parallel safety", "passed_checks": 12, "failed_checks": 4, "total_checks": 16, "passed": false, "sample_status": "excluded", "included": false, "reason": "API rate limit or quota failure", "turns": null, "duration_seconds": null, "total_tokens": null, "cost_usd": null, "failures": [ "Artifact not found: .cache/comet-native-eval/conflict-status.json", "Artifact not found: .cache/comet-native-eval/checkpoint-attempt-a.json", "Artifact not found: .cache/comet-native-eval/checkpoint-attempt-b.json", "native_isolation: Classic or hidden workflow artifacts exist" ], "experiment": "beta16_native_wave_e_retry_ds_20260810", "report_file": "comet_native_wave_e_parallel_safety_COMET_NATIVE_PHASE1_r1_rep1_report.json" }, { "version": "beta16", "wave": "F", "label": "Dashboard read-only", "passed_checks": 13, "failed_checks": 8, "total_checks": 21, "passed": false, "sample_status": "included", "included": true, "reason": "Run completed with validator evidence", "turns": 85, "duration_seconds": 662.77, "total_tokens": 3923122, "cost_usd": 3.5352530000000004, "failures": [ "current_cli_source_build: Controller source-build snapshot identity does not match its files", "dashboard_readonly: dashboard-visible-change: CLI={'name': 'dashboard-visible-change', 'phase': 'shape', 'revision': 1, 'selected': True, 'nextCommand': 'comet native next dashboard-visible-change --summary \"\" --confirmed', 'verificationResult': 'pending', 'archiveReady': False, 'findings':", "public_native_projection: Dashboard native root fields are not bounded: ['activeChangeCount', 'archivedChangeCount', 'changes', 'changesTruncated', 'conflicts', 'generatedAt', 'omittedChangeCount', 'schema', 'totalChangeCount', 'visibleChangeCount']", "live_dashboard_and_tree: dashboard-visible-change: CLI={'name': 'dashboard-visible-change', 'phase': 'shape', 'revision': 1, 'selected': True, 'nextCommand': 'comet native next dashboard-visible-change --summary \"\" --confirmed', 'verificationResult': 'pending', 'archiveReady': False, 'findi", "native_isolation: Forbidden workflow artifacts exist: ['.comet/current-change.json']" ], "experiment": "beta16_native_bf_ds_20260810", "report_file": "comet_native_wave_f_dashboard_readonly_COMET_NATIVE_PHASE1_r1_rep1_report.json" } ], "html": "\nComet Native beta16 vs beta17\n\n

Comet Native beta16 vs beta17

DeepSeek · cc-switch 当前配置 · 2026-08-10 · B–F 五组困难评估

\n
这不是同一协议的单一分数比较：beta16 继续使用旧版 snapshot/receipt/trajectory/checkpoint 评估器，beta17 使用 portable v4 评估器。E 组两个版本都在单 worker 重跑后仍被 API 限流/超时排除。
\n

beta16

54/73

可用组通过检查数 · 74.0%

总耗时 52.1 分钟 · 成本 $25.13

协议：beta16 legacy validators

beta17

58/65

可用组通过检查数 · 89.2%

总耗时 42.4 分钟 · 成本 $21.08

协议：beta17 portable v4 validators

逐组结果

版本任务检查状态轮次耗时(s)成本($)首个失败/说明
beta17B Decision / resume13/14有效124735.25.2Artifact not found: docs/comet/specs/unique-word-counting/spec.md
beta17C Verification integrity18/18有效109563.85.7Run completed with validator evidence
beta17D Stagnation stop13/16有效109858.17.4stagnation_stop: The twelfth failure did not reach the v4 repair ceiling
beta17E Parallel safety12/16排除———Artifact not found: .cache/comet-native-eval/conflict-status.json
beta17F Dashboard read-only14/17有效74386.82.8dashboard_readonly: Native tree manifest does not match live files
beta16B Decision / resume10/17有效21155.91.0Artifact not found: .cache/comet-native-eval/archive-preview.json
beta16C Verification integrity16/17有效123742.35.7verification_integrity: Invalid archived Native evidence: Verification envelope fields are invalid: missing=['receiptRef'], unknown=['receiptRefs', 'requiredReceiptRefs']
beta16D Stagnation stop15/18有效1581,563.614.9stagnation_stop: The twelfth total failure is not the hard repair ceiling
beta16E Parallel safety12/16排除———Artifact not found: .cache/comet-native-eval/conflict-status.json
beta16F Dashboard read-only13/21有效85662.83.5current_cli_source_build: Controller source-build snapshot identity does not match its files
\n

解读

  • beta17 C 组在 v4 验收下 18/18 通过，说明 Verifier 全覆盖、Loop 与归档路径在该样本中完整闭合。
  • beta17 B/D/F 的失败来自模型行为：分别是目标 spec 路径、停滞终止、归档/树一致性；不是时间字段 validator 误判。
  • beta16 的 B/C/F 失败包含旧协议要求的归档预览、receipt/evidence、trajectory、Dashboard v1 等检查，不能直接解释为 beta17 回归。
  • 评估器自身的 build identity 元数据仍保留用于实验可重复性，但不当作 beta17 Native 产品语义。
\n

原始报告保存在各实验目录；本文件只保留脱敏统计、失败摘要与协议说明，不包含 API 凭据、原始模型日志或绝对工作区路径。

" }