# Multi-Pass Skill Audit Protocol Detailed reference for the 4-phase adversarial skill quality audit. Read this for the full review dimensions, scoring system, and output templates. ## Phase 1 — Inventory & Profiling (Agent A: Evaluator) ### Inputs Before starting, gather context: 1. **Skills directory listing** — enumerate all skill directories 2. **`skill-builder` standards** — read `skill-builder/SKILL.md` and `skill-builder/checklist.md` to calibrate quality expectations 3. **Prior audits** — search the journal: ``` search_entries({ query: "skill audit", entry_type: "skill_audit_profile", tags: ["adversarial-skill-audit"] }) ``` 4. **README & Package inventory** — check if `README.md` lists all skills, if `package.json` includes them in `files`, and whether they are current ### Per-Skill Scorecard For each skill, produce a scorecard: ```markdown ### [skill-name] **Files**: SKILL.md (X lines, Y KB) + N reference files **Description**: [first 100 chars of description] | Category | Score (1–5) | Notes | | ------------------------------------- | ----------- | ----- | | 1. Frontmatter & Triggering | | | | 2. Instruction Clarity | | | | 3. Structure & Progressive Disclosure | | | | 4. Output Formats | | | | 5. Edge Cases & Errors | | | | 6. Security & Safety | | | | 7. Token Efficiency | | | | 8. Maintenance | | | | **Average** | **X.X** | | **Top Issue**: [Single most impactful improvement] **Grade**: [A–F] ``` ### Scoring Guide | Score | Meaning | | ----- | -------------------------------------------------------- | | 5 | Excellent — meets or exceeds all skill-builder standards | | 4 | Good — minor gaps, functional as-is | | 3 | Acceptable — noticeable issues but usable | | 2 | Poor — significant gaps affecting agent behavior | | 1 | Failing — broken, misleading, or unusable | ### Directory-Level Assessment After scoring all skills individually, produce a directory-level overview: ```markdown ## Directory Overview **Total Skills**: N **Average Quality Score**: X.X / 5.0 **Grade Distribution**: A: N, B: N, C: N, D: N, F: N ### Score Distribution | Grade | Skills | | ----------- | ---------------- | | A (4.5–5.0) | skill-a, skill-b | | B (3.5–4.4) | skill-c, skill-d | | C (2.5–3.4) | skill-e | | D (1.5–2.4) | skill-f | | F (1.0–1.4) | — | ### Cross-Skill Properties - **Potential overlaps**: [skills that cover similar domains] - **Coverage gaps**: [domains with no skill coverage] - **Consistency issues**: [naming conventions, structure differences] - **Inventory accuracy**: [is the README table and package.json current?] ``` ### Journal ``` create_entry({ content: "", entry_type: "skill_audit_profile", tags: ["adversarial-skill-audit", "profile"], project_number: }) ``` --- ## Phase 2 — Adversarial User Review (Agent B: Adversarial User) Switch mental models. You are now a user who types imprecise, ambiguous prompts. Your job is to find where the skills fail — wrong skill triggers, no skill triggers, confusing output, or dangerous behavior. ### Review Dimensions Score each dimension on a 1–5 scale: | Dimension | Weight | Focus Areas | | ------------------------ | ------ | ---------------------------------------------------------------------------------------------------------------- | | **Trigger Reliability** | 4 | Does the skill load when it should? Does it avoid loading when it shouldn't? Are trigger keywords comprehensive? | | **Instruction Fidelity** | 3 | Will an agent actually follow these instructions correctly? Are there ambiguous steps that invite hallucination? | | **Failure Graceful** | 2 | What happens when prerequisites are missing, inputs are weird, or the project doesn't match assumptions? | | **Token ROI** | 1 | Is the token cost justified by the value provided? Could the same effect be achieved with fewer tokens? | ### Trigger Testing For each skill, construct 3 test prompts: 1. **Direct trigger** — a prompt that obviously should load this skill 2. **Oblique trigger** — a prompt that should load this skill but uses different phrasing than the description 3. **Anti-trigger** — a prompt that sounds related but should NOT load this skill Evaluate each: ```markdown ### Trigger Test: [skill-name] | # | Prompt | Should Trigger? | Would It? | Confidence | | --- | -------------------------- | --------------- | --------- | --------------------------- | | 1 | "Build an MCP server" | Yes | Yes | High | | 2 | "Connect my API to Claude" | Yes | Maybe | Medium | | 3 | "Write a REST API" | No | Maybe | Low — description too broad | ``` ### Trigger Collision Map Identify prompts where multiple skills would compete to load: ```markdown ### Trigger Collisions | Prompt | Competing Skills | Winner | Issue | | ---------------------- | ------------------------------------------ | ------- | --------------------- | | "Set up CI/CD" | github-actions, docker, cloudflare | Unclear | All three claim CI/CD | | "Deploy to production" | cloudflare, docker, workers-best-practices | Unclear | Ambiguous platform | ``` ### Instruction Stress Tests For each skill, identify instructions that an agent might: - **Misinterpret** — ambiguous phrasing that could lead to wrong behavior - **Skip** — low-priority-sounding steps that are actually critical - **Over-execute** — instructions that could cause an agent to do too much - **Hallucinate from** — vague instructions that invite fabrication ```markdown ### Instruction Issues: [skill-name] | Line/Section | Issue | Type | Example Failure | | ------------------------ | ---------------------- | ------------- | ------------------------------- | | "Consider using caching" | Vague — agent may skip | Skip risk | Agent ignores caching entirely | | "Set up the project" | No specifics | Hallucination | Agent invents project structure | ``` ### Critique Output Format ```markdown ## Adversarial User Review — [Skills Directory] **Overall Quality Score:** [weighted average] / 5.0 **Directory Grade:** [A–F] ### Findings | # | Skill | Category | Severity | Finding | Remediation | | --- | ------------- | ---------- | -------- | --------------------------------- | ---------------------------------- | | 1 | mcp-builder | Triggering | High | "Connect AI to API" won't trigger | Add to description keywords | | 2 | docker | Clarity | Medium | "Set up multi-stage" is ambiguous | Add concrete Dockerfile template | | 3 | — (directory) | Collision | High | 3 skills fight over "deploy" | Add disambiguation in descriptions | ### Dimension Scores | Dimension | Score | Weight | Weighted | | -------------------- | ----- | ------ | -------------------- | | Trigger Reliability | [1–5] | 4 | [score × 4] | | Instruction Fidelity | [1–5] | 3 | [score × 3] | | Failure Graceful | [1–5] | 2 | [score × 2] | | Token ROI | [1–5] | 1 | [score × 1] | | **Total** | | **10** | **[sum]/50 = [avg]** | ### Blocking Issues Skills that are actively harmful, misleading, or security-risk. ``` ### Journal ``` create_entry({ content: "", entry_type: "skill_audit_stress", tags: ["adversarial-skill-audit", "stress-test"], project_number: }) ``` --- ## Phase 3 — Improvement Plan (Agent A: Evaluator) Switch back to the Evaluator. Address every finding with a disposition. ### Disposition Table | Disposition | Meaning | | ----------- | ------------------------------------------------- | | **Accept** | Implement the improvement | | **Reject** | Finding doesn't apply or trade-off isn't worth it | | **Modify** | Accept the spirit but implement differently | | **Defer** | Acknowledge but defer (document why) | ### Improvement Plan Output ```markdown ## Improvement Plan — [Skills Directory] ### Disposition Summary | # | Skill | Finding | Disposition | Rationale | | --- | ------------- | ----------------------------- | ----------- | -------------------------------- | | 1 | mcp-builder | Missing trigger keywords | Accept | Add 5 alternative phrasings | | 2 | docker | Ambiguous instruction | Modify | Add example instead of rewriting | | 3 | — (directory) | Trigger collision on "deploy" | Accept | Add "NOT for..." clauses | ### Priority 1 — Quick Wins (per skill) | Skill | Fix | Effort | | ----- | --- | ------- | | ... | ... | <15 min | ### Priority 2 — Structural Improvements | Skill | Fix | Effort | | ----- | --- | --------- | | ... | ... | 30–60 min | ### Priority 3 — New Skills Needed | Gap | Suggested Skill | Rationale | | --- | --------------- | --------- | | ... | ... | ... | ### Skills to Deprecate or Merge | Skill | Action | Rationale | | ----- | ------------ | ------------------ | | ... | Merge into X | 90% overlap with X | ### Projected Quality Improvement | Metric | Before | After | | ------------------ | ------ | ----- | | Average score | X.X | X.X | | Skills graded A/B | N/M | N/M | | Trigger collisions | N | N | | Coverage gaps | N | N | ``` ### Iteration Control Check: has `MAX_AUDIT_PASSES` been reached? - **No** → return to Phase 2 (stress-test the improvement plan itself) - **Yes** → proceed to Phase 4 ### Journal ``` create_entry({ content: "", entry_type: "skill_audit_remediation", tags: ["adversarial-skill-audit", "remediation"], project_number: }) ``` --- ## Phase 4 — Copilot Validation (External) If available, invoke Copilot CLI. See [copilot-skill-prompts.md](copilot-skill-prompts.md). If unavailable, skip gracefully. ### Journal ``` create_entry({ content: "", entry_type: "skill_audit_copilot", tags: ["adversarial-skill-audit", "copilot"], project_number: }) ``` --- ## Final Report Assembly ```markdown # Adversarial Skill Audit — [Skills Directory] **Date**: [ISO date] **Audit Depth**: [surface | standard | thorough] **Skills Audited**: N **Passes Completed**: N **Copilot Validation**: [yes | skipped] ## Executive Summary [2–3 sentences: overall quality, critical issues, top recommendation] **Directory Quality Score**: [X.X] / 5.0 — Grade [A–F] ## Skill Scoreboard | Skill | Score | Grade | Top Issue | | ------------------- | ----- | ----- | ---------------------------------- | | adversarial-planner | 4.5 | A | — | | docker | 3.2 | C | Ambiguous multi-stage instructions | | ... | | | | ## Top 5 Improvements (by impact) 1. [Highest-impact fix] 2. ... ## Trigger Collision Map [From Phase 2] ## Coverage Gap Analysis [Domains without skill coverage] ## Full Findings (by severity) [All findings from Phase 2] ## Improvement Plan [From Phase 3] ## Per-Skill Scorecards [From Phase 1 — all individual scorecards] ```