# Context Engineering Context engineering is the first pillar of harness engineering. It is the practice of curating, structuring, and delivering the right information to AI agents at the right time so they can produce correct, consistent code. Spotify's public writing on background coding agents is one of the clearest published examples of context engineering in practice: the quality of the agent depends heavily on how information is assembled, scoped, and delivered. --- ## Context Engineering vs. Prompt Engineering These terms are related but distinct: | Prompt Engineering | Context Engineering | |-------------------|-------------------| | Crafting a single input to get a good output | Designing a system that provides optimal context across many sessions | | One-shot | Persistent, evolving | | Lives in chat messages | Lives in the repository | | Per-task effort | One-time effort with compounding returns | | "How do I ask this question?" | "What should the agent already know?" | Prompt engineering is a *skill*. Context engineering is an *infrastructure discipline*. You still need good prompts, but the context system does the heavy lifting. **The analogy**: Prompt engineering is like writing a good Google search query. Context engineering is like building Google's index. One makes a single search work. The other makes all searches work. --- ## The Repository as Knowledge Store In harness engineering, the repository is the primary knowledge store for AI agents. Everything an agent needs to know should be discoverable from the repository root. ### Why the Repository? 1. **Always available**: Every agent session starts with access to the repo. 2. **Always versioned**: Context evolves with the code it describes. 3. **Always testable**: You can write tests to verify your docs are current. 4. **Always consistent**: One source of truth, not multiple wikis diverging. ### What Goes in the Repository ``` project-root/ .github/ copilot-instructions.md # Repo-wide Copilot context (~100 lines) instructions/ # Scoped rules with applyTo globs prompts/ # Reusable slash-command prompts agents/ # Custom Copilot agent profiles AGENTS.md # Universal agent config ARCHITECTURE.md # System design, module boundaries docs/ conventions.md # Coding patterns and style decisions api-guide.md # API design rules testing-guide.md # How to write tests in this project decisions/ # Architecture Decision Records (ADRs) 001-use-azure-sql.md 002-event-sourcing.md .vscode/ mcp.json # MCP server wiring (optional) ``` ### What Does Not Go in the Repository - Secrets, credentials, API keys (use `.env.example` with placeholders). - Auto-generated documentation that duplicates source code. - Information that changes faster than you can commit (use dynamic context instead). - Personal preferences that are not team conventions. --- ## AGENTS.md as Table of Contents Your top-level agent configuration file is the most important context engineering artifact. It is the first thing the agent reads and it shapes every decision that follows. ### Target Length: ~100 Lines This is not arbitrary. It is driven by two constraints: 1. **Context window cost**: Every token in the top-level file is loaded on every session. 100 lines is approximately 300-400 tokens -- less than 1% of even a small context window. 2. **Signal-to-noise ratio**: Beyond 100 lines, you are likely including details that are not relevant to every task. Those details belong in linked documents. ### Structure ```markdown # [Project Name] ## What This Project Does [2-3 sentences. Orient the agent.] ## Tech Stack [Language, framework, database, key dependencies.] ## Architecture Overview [3-5 sentences. Link to detailed doc.] See docs/architecture.md for full details and module boundaries. ## Commands - Build: `make build` - Test: `make test` - Lint: `make lint` - All checks: `make verify` ## Key Rules 1. [Most important architectural constraint] 2. [Second most important constraint] 3. [Third most important constraint] ## Code Organization - `src/domain/` -- Core business logic, no external dependencies - `src/services/` -- Application services, orchestrate domain logic - `src/api/` -- HTTP layer, thin handlers - `src/infra/` -- Database, cache, external API clients ## Documentation Index - [Architecture](docs/architecture.md) -- System design - [API Conventions](docs/api-conventions.md) -- Endpoint design - [Testing Guide](docs/testing-guide.md) -- Testing patterns - [Decisions](docs/decisions/) -- Why we chose X over Y ``` The agent reads this in seconds and knows: what the project is, how to build and test it, what the critical rules are, where to find details. --- ## Progressive Disclosure Progressive disclosure is the practice of providing information in layers -- general first, specific on demand. It is the single most important pattern in context engineering. ### How It Works ``` Layer 1: .github/copilot-instructions.md (always loaded, ~100 lines) "All API endpoints follow docs/api-conventions.md" Layer 2: docs/api-conventions.md (loaded when working on APIs, ~200 lines) "POST endpoints use request validation middleware. See src/middleware/validate.ts for the implementation." Layer 3: src/middleware/validate.ts (loaded when modifying validation, actual code) The real implementation, with doc comments explaining patterns. Layer 4: tests/middleware/validate.test.ts (loaded when agent needs examples) Concrete examples of how validation works. ``` Each layer adds detail. The agent only loads deeper layers when the task requires it. This keeps the context window focused on relevant information. ### Implementing Progressive Disclosure 1. **Top-level config**: Short rules and links. Always loaded. 2. **docs/ directory**: Detailed guides. Loaded when the agent encounters a relevant task. 3. **Code comments**: In-place context. Loaded when the agent reads a specific file. 4. **Test files**: Examples. Loaded when the agent needs to understand patterns. ### The Key Insight Agents are good at following links. When your .github/copilot-instructions.md says "See docs/testing-guide.md for testing patterns," and the agent is writing a test, it *will* read that file. You do not need to dump all testing instructions into the top-level config. --- ## Dynamic Context Not all context can be pre-written. Some context must be gathered at runtime: ### Types of Dynamic Context **Test output**: When the agent runs tests and they fail, the failure output becomes context. Good test failure messages are context engineering. ```csharp // Bad test output (low context) Assert.True() Failure Expected: True Actual: False // Good test output (high context) Assert.Equal() Failure: UserService.CreateUserAsync should throw ValidationException when email is invalid, but it returned a User instead. Expected: ValidationException with message "Invalid email format" Actual: User { Id = 1, Email = "not-an-email", … } Hint: Check the email validation in src/Project.Application/Users/CreateUserHandler.cs:42 ``` **Linter / analyzer output**: Roslyn analyzers and PSScriptAnalyzer rules should include remediation instructions that the agent can act on directly. ``` PRJ001: Application layer must not depend on Infrastructure --> src/Project.Application/Orders/PlaceOrderHandler.cs(12,9) | 12| using Project.Infrastructure.Stripe; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ = help: Application code must not reference Infrastructure types directly. Define an interface (e.g. IPaymentGateway) in Project.Application and resolve the concrete implementation through DI. Example: src/Project.Application/Abstractions/IPaymentGateway.cs See: docs/architecture.md#data-access-patterns ``` **Runtime logs**: When debugging, agent access to application logs provides context that no pre-written document can match. **Browser / UI state**: For frontend work, agents that can see screenshots or DOM snapshots have visual context that text cannot convey. **Observability data**: Error rates, latency metrics, and trace data help agents understand the impact of their changes in production. ### Providing Dynamic Context Most agent tools support dynamic context through: - **Tool use**: The agent calls tools (run tests, read logs, fetch metrics) that return context. - **MCP servers**: Model Context Protocol servers provide structured access to external systems. - **File watchers**: Files that are automatically updated (build output, coverage reports) and read by the agent. --- ## Context Window Management Context windows are finite. Managing them well is critical. ### The Context Budget Think of the context window as a budget: ``` Total context window: 200,000 tokens (Claude) - System prompt + config: ~500 tokens (0.25%) - Top-level .github/copilot-instructions.md: ~400 tokens (0.2%) - Referenced docs (loaded as needed): ~2,000-5,000 tokens (1-2.5%) - Source code files being modified: ~5,000-20,000 tokens (2.5-10%) - Test output and tool results: ~2,000-10,000 tokens (1-5%) - Conversation history: ~10,000-50,000 tokens (5-25%) - Available for reasoning: remainder ``` The goal is to minimize fixed overhead (things loaded on every session) and maximize the space available for task-specific context and reasoning. ### Strategies 1. **Keep .github/copilot-instructions.md short**: 100 lines, not 1,000. 2. **Link, don't inline**: Reference docs rather than pasting their content. 3. **Scope sessions**: One task per session. Do not let conversation history accumulate irrelevant context from previous tasks. 4. **Summarize, then detail**: Start docs with a summary paragraph, then provide details. The agent can stop reading when it has enough. 5. **Prune stale context**: Remove docs that describe features or patterns that no longer exist. #### Cross-Session Context Management Within a single session, context engineering focuses on what to put in the prompt. Across multiple sessions, the challenge shifts to **continuity** -- how does the next session's agent know what happened? **Why compaction alone fails:** Built-in context compaction (summarizing earlier conversation to free window space) optimizes for token efficiency, not task continuity. It "doesn't always pass perfectly clear instructions to the next agent." Compaction is necessary within a session but insufficient between sessions. **Artifact-based continuity (the solution):** Three artifacts bridge sessions: 1. **Progress file** (human-readable): what was done, what's next, blockers 2. **Git history** (machine-readable): commit messages as session summaries, diff for understanding changes 3. **Feature list** (structured JSON): immutable scope tracking with pass/fail status The key insight: these are the same tools effective human engineers use for shift handoffs. The format that works for humans also works for agents. See [Long-Running Agents](long-running-agents.md) for the full multi-session protocol. --- ## Spotify's Principles for AI-Readable Context Spotify's "Honk" engineering blog series articulated several principles for writing context that AI agents can use effectively: ### 1. Preconditions State what must be true before the agent starts. This prevents wasted work. ```markdown ## Preconditions for API work - The database migration for the relevant table must exist in `migrations/` - The OpenAPI spec must be updated in `api/spec.yaml` before implementing - The corresponding repository interface must exist in `src/repos/` ``` ### 2. Examples Over Explanations Show the agent a concrete example rather than describing an abstract pattern. ```markdown ## How to add a new API endpoint See `src/api/users.ts` for a complete example of: - Request validation - Service delegation - Error handling - Response formatting - Test file structure ``` ### 3. Verifiable Goals End every task description with a way to verify success. ```markdown ## Verification After making changes: 1. `make test` passes with no failures 2. `make lint` passes with no errors 3. New endpoint appears in the OpenAPI spec 4. New endpoint has at least 3 test cases (happy path, validation error, not found) ``` ### 4. Negative Examples Show what *not* to do. Agents learn from counter-examples. ```markdown ## Anti-patterns to avoid - Do NOT use `any` type -- always define specific types - Do NOT catch errors silently -- always log and re-throw or return error response - Do NOT put business logic in API handlers -- delegate to services ``` --- ## Anti-Patterns in Context Engineering ### Context Overload **Symptom**: The agent config file is 500+ lines. Every possible instruction is in one place. **Problem**: The agent's attention is diluted. Important rules are lost in a sea of details. The context window is consumed by fixed overhead. **Fix**: Ruthlessly prune the top-level config to ~100 lines. Move everything else to linked documents. ### Stale Documentation **Symptom**: Docs describe patterns, APIs, or structures that no longer exist in the code. **Problem**: The agent follows outdated instructions and produces code that does not fit the current architecture. **Fix**: Treat docs as code. Review them in PRs. Set up entropy management agents that verify docs match code. See [Entropy Management](entropy-management.md). ### Instruction Rot **Symptom**: Instructions are added over time but never removed. Contradictory instructions accumulate. **Problem**: The agent encounters conflicting guidance and picks one arbitrarily (or averages them, producing something neither instruction intended). **Fix**: Periodic review of all agent-facing docs. Remove instructions that are superseded. Resolve contradictions explicitly. One agent instruction should handle one concern. ### Implicit Knowledge **Symptom**: The team knows things that are not written down. "Everyone knows we use the repository pattern." "Everyone knows the auth middleware runs first." **Problem**: The agent is not part of "everyone." It only knows what is written in the repository. **Fix**: When you find yourself explaining something to an agent in a chat message, immediately encode that knowledge in a docs file. The chat message fixes one session. The doc fixes all future sessions. ### Copy-Paste Context **Symptom**: The same information is duplicated in `.github/copilot-instructions.md`, `AGENTS.md`, `ARCHITECTURE.md`, and a team wiki. **Problem**: Copies diverge. One gets updated, others do not. The agent sees the stale copy. **Fix**: Single source of truth in the repository. Tool-specific config files should be thin wrappers that link to the shared docs. --- ## Measuring Context Quality How do you know if your context engineering is working? ### Leading Indicators - **Agent orientation time**: How many questions does the agent ask before starting work? Fewer is better. - **File discovery accuracy**: Does the agent find the right files on the first try? Track how often it reads irrelevant files. - **Convention compliance**: Does agent output match your conventions without explicit reminders? ### Lagging Indicators - **First-pass success rate**: What percentage of agent output passes all verification on the first run? - **Code review comment density**: How many comments per PR? Trending down means context is improving. - **Rework rate**: How often does the agent need to redo work because it misunderstood the task? ### The Context Quality Test Ask a new team member (or a fresh agent session) to answer these questions from your repository alone: 1. What does this project do? 2. How do I build and test it? 3. What are the top 3 architectural rules? 4. Where do I find the code for [specific feature]? 5. What patterns should I follow for [common task type]? If the answers are correct and fast, your context engineering is good. If they require searching, guessing, or asking someone -- your context has gaps. --- ## Further Reading - [What Is Harness Engineering?](what-is-harness-copilot.md) -- The three pillars overview. - [Principles: Map, Not Manual](principles.md#1-give-agents-a-map-not-a-manual) -- The principle behind progressive disclosure. - [Architectural Constraints](architectural-constraints.md) -- The second pillar: enforcing what context describes. - [Anti-Patterns](anti-patterns.md) -- More context engineering mistakes to avoid.