# Internal Ollama Agent Evaluation And Tuning Design

Task: GH-534-INTERNAL-OLLAMA-AGENT-DESIGN

This document defines the governed local path for improving internal
Ollama-backed agents through evaluation, prompt/context tuning, and future
training readiness. It does not authorize model training, fine-tuning, hosted
provider execution, unsafe local provider egress, or automatic collection of
private workspace data.

## Goals

- Define local model boundaries for the Ollama-backed internal agent path.
- Define evaluation dataset provenance, consent, redaction, and retention
  rules.
- Define readiness gates before prompt tuning, retrieval/context changes,
  adapter changes, LoRA/fine-tuning, or model release.
- Define offline execution and provider-safety controls for local evaluation.
- Define acceptance criteria, test evidence, and CLI/runtime capabilities agents
  should consume.

## Non-Goals

- No training, fine-tuning, LoRA adapter generation, quantization, or model
  weight publishing in this task.
- No change to provider implementation, `src/model-*`, or runtime execution
  behavior.
- No default use of real Ollama models in CI.
- No automatic mining of repository contents, chat history, prompts, provider
  responses, issues, or customer data for datasets.
- No public-network Ollama endpoint support.
- No hosted provider fallback when a local model cannot answer or is offline.
- No acceptance of model-generated output as trusted commands, workflow state,
  runtime rules, or security decisions.

## Local Model Boundary

The Ollama path is a local-provider path, not a runtime-native child-agent path.
It may be used only through the provider-backed phase layer when policy records
explicit direct-provider opt-in for `ollama`. Runtime-native packets for Codex,
Claude, Cursor, OpenCode, VS Code, Windsurf, or generic runtimes remain
`directProviderApiAllowed: false` and must not embed Ollama credentials or
private prompts.

Allowed endpoints are limited to:

- `localhost` and loopback addresses;
- link-local addresses;
- RFC1918 private-network IP addresses reachable only inside a trusted
  deployment network.

Disallowed endpoints include:

- public DNS names and public IP addresses;
- URLs with credentials;
- URLs resolved through untrusted proxies;
- endpoints that require sending workspace data outside the trusted local or
  private network boundary.

Ollama outputs are untrusted provider output. They may propose text for a
handoff, review, plan, or evidence draft, but the parent runtime and Orchestra
workflow gates remain responsible for policy checks, evidence verification, and
human approval where gates require it.

## Evaluation Dataset Governance

Evaluation fixtures must be explicit, versioned, and provenance-labeled before
use. A dataset manifest should be introduced in a future child task before any
automated eval harness promotes results. The manifest should include:

- dataset id, version, owner, creation date, and linked backlog item;
- source type such as synthetic, curated public docs, user-approved workspace
  excerpt, sanitized issue, or generated adversarial fixture;
- consent basis and retention class;
- redaction status and reviewer;
- allowed use such as eval-only, prompt-tuning, retrieval tuning, regression,
  adversarial safety, or future fine-tuning candidate;
- excluded use such as training-disallowed, export-disallowed, or
  provider-disallowed;
- checksum for each fixture and an immutable manifest checksum;
- expected behavior, rubric, and minimum assertions per fixture;
- deprecation or rollback note when a fixture is replaced.

Default-allowed data:

- synthetic task briefs created for testing;
- product docs authored for public or repository-local use;
- sanitized workflow state created in isolated temporary workspaces;
- adversarial fixtures that are clearly labeled as hostile data.

Default-blocked data unless separately approved:

- raw user prompts, chats, provider responses, screenshots, or recordings;
- secrets, tokens, API keys, credentials, private URLs, or auth headers;
- PII, customer data, regulated data, private repository code, or commercial
  confidential material;
- unredacted issue bodies, logs, traces, or stack traces from real workspaces;
- generated content that contains prompt-injection instructions intended to
  alter runtime rules.

## Evaluation Path

The local evaluation path should run in tiers:

| Tier                       | Purpose                                                                        | Provider                                | Data                         | Release Meaning            |
| -------------------------- | ------------------------------------------------------------------------------ | --------------------------------------- | ---------------------------- | -------------------------- |
| P0 deterministic contract  | Prove CLI, policy, redaction, provenance, and fail-closed behavior             | fake or mock OpenAI-compatible endpoint | synthetic fixtures           | PR-safe blocker            |
| P1 local Ollama smoke      | Prove a developer's local Ollama installation can satisfy the adapter contract | real loopback/private Ollama            | approved smoke fixtures only | opt-in confidence          |
| P1 eval harness            | Score candidate prompts/context packs against governed fixtures                | mock by default, optional Ollama        | manifest-approved eval set   | tuning evidence            |
| P2 adversarial/local model | Probe jailbreak, unsafe tool text, leakage, and hallucinated workflow state    | mock plus optional Ollama               | hostile fixtures             | security confidence        |
| Future training readiness  | Decide whether any fine-tuning path is permissible                             | none until approved                     | candidate manifest only      | design gate, not execution |

Evaluation results must record:

- task id and command;
- model provider, model name, endpoint class, and whether the endpoint was mock
  or real local;
- dataset manifest id and checksum;
- prompt/context pack version;
- pass/fail counts, rubric scores, and named failed fixtures;
- redaction status;
- retained artifacts and retention class;
- explicit statement that contract evidence is not real-model acceptance when
  the provider was fake or mocked.

## Tuning Path

The first improvement mechanism is prompt and context tuning, not model-weight
tuning. Prompt/context packs should be versioned separately from datasets and
models. A pack should include:

- pack id, version, owner, and linked task;
- intended roles and risk class;
- input context sources and context budget;
- safety preamble and refusal boundaries;
- expected tool/runtime capabilities;
- rollback pointer to the previous approved pack;
- eval manifest ids used for approval;
- release notes for behavior changes.

Changes that alter role behavior, provider routing, model selection, or evidence
semantics require Architect and Security review before use in provider-backed
phases. Changes that affect user-visible flows or release confidence require QA
evidence before promotion.

## Training And Fine-Tuning Readiness Gates

No fine-tuning or training may start until a separate child story satisfies all
of these gates:

1. Product Owner approves the use case, user value, non-goals, and rollback
   expectation.
2. Architect approves model boundary, adapter contract, artifact format, and
   release/versioning design.
3. Security and Compliance/Privacy approve data classification, consent,
   redaction, retention, access controls, and export restrictions.
4. QA approves deterministic eval, adversarial eval, regression coverage, and
   model comparison methodology.
5. Release Manager approves rollout, rollback, model registry, artifact
   retention, and support communication.
6. Dataset manifest proves every candidate fixture is training-allowed and
   reviewed.
7. Baseline eval shows prompt/context tuning is insufficient or documents why
   fine-tuning is justified despite the added risk.
8. Offline training environment is isolated from production credentials,
   customer data, and public egress unless an explicit trusted policy exists.
9. Model release manifest defines version, source dataset checksums, eval
   checksums, safety findings, known limitations, and rollback target.

Until those gates pass, any model-training command, dataset export, adapter
build, model upload, or weight publication is out of scope and should fail
closed in process and review.

## Safety And Privacy Controls

- Treat prompts, fixtures, model output, retrieved context, tool output, logs,
  and generated artifacts as untrusted.
- Redact secrets, auth headers, token-shaped values, private URLs, stack traces,
  internal paths, raw prompts, and provider errors before evidence capture.
- Keep dataset and eval artifacts local unless a future issue explicitly defines
  a trusted export policy.
- Require dataset owners and retention classes; remove stale eval fixtures when
  their consent or source validity expires.
- Keep public-network egress disabled for local-provider smokes.
- Block hosted-provider fallback when Ollama is offline or fails.
- Require human review for any proposed runtime-rule, security, release, or
  workflow-policy change generated by a model.
- Separate eval-only fixtures from training-eligible fixtures.
- Store provider secrets only in environment variables or local secret files;
  never in workflow state, docs, datasets, reports, or evidence.

## Offline Execution

The default evaluation mode must be offline and deterministic:

- use temporary workspaces;
- use fake or mock OpenAI-compatible provider endpoints by default;
- avoid hosted provider credentials;
- avoid network calls except loopback/private Ollama when explicitly enabled;
- capture sanitized command and JSON evidence;
- record deferred evidence when real local Ollama is unavailable.

`ORCHESTRA_OLLAMA_SMOKE=1` may enable a local real-model smoke only when the
developer has an approved loopback/private Ollama endpoint. That smoke proves
local installation compatibility and basic behavior; it does not replace
dataset-governed eval evidence or training readiness.

## Agent-Consumable CLI And Runtime Capabilities

Agents should consume existing governed capabilities instead of invoking
providers or shelling out to local model tools directly:

- `orchestra health --runtime <runtime> --json` for workspace/runtime preflight.
- `orchestra task list --json --status pending,blocked,in_progress` for
  managed task discovery.
- `orchestra validate --pre-run --task <ID> --json` for active context checks.
- `orchestra context --task <ID>` for task-scoped state.
- `orchestra model providers --json` for supported provider discovery.
- `orchestra model profile smoke --name <profile> --json` for sanitized
  provider configuration checks.
- `orchestra model connect --provider ollama ...` only when a trusted user or
  workflow explicitly approves provider-backed execution.
- `orchestra workflow run --task <ID> --profile <profile> --gates phase` only
  after task, estimate, workflow, and gate requirements are satisfied.
- `orchestra model provenance list --json` for provider/model evidence review.
- `orchestra evidence add` and `orchestra review` for explicit proof and
  handoff records.
- `npm run test:e2e:runtime:ollama` for the mock-by-default Ollama adapter
  smoke.

Agents must not call `ollama`, `curl`, provider SDKs, or model-training tools
directly for this path unless a future issue explicitly adds a reviewed command
contract and safety policy.

## Acceptance Criteria

- The local Ollama agent boundary is documented as private-only,
  provider-backed, policy-gated, and separate from runtime-native execution.
- Evaluation datasets require manifest provenance, source classification,
  consent basis, redaction status, allowed use, checksums, and retention.
- Prompt/context tuning is the first improvement path and is versioned with
  rollback and eval evidence.
- Training/fine-tuning remains blocked behind explicit Product, Architect,
  Security, Compliance/Privacy, QA, and Release gates.
- Offline execution defaults to fake or mock endpoints and temporary workspaces.
- Real local Ollama smoke is opt-in, loopback/private-only, and cannot satisfy
  training or hosted-provider acceptance.
- Tests distinguish deterministic contract evidence, real local smoke,
  adversarial fixtures, and future training-readiness evidence.
- Risks and non-goals are explicit.

## Test Plan

- Run markdown/docs formatting checks available in the repository.
- Run `npm run test:e2e:runtime:ollama` after any implementation child story
  that changes the Ollama adapter or provider-backed workflow contract.
- Add future unit tests for dataset manifest parsing and use-policy decisions
  before an eval harness is introduced.
- Add future E2E coverage proving public Ollama endpoints fail closed, fake or
  mock smokes are labeled contract evidence, and real local smokes record
  redacted provenance.
- Add future redaction tests for eval reports covering prompt text, provider
  errors, endpoint details, token-shaped strings, and private paths.
- Add future training-readiness tests that fail closed when a fixture lacks
  training-allowed provenance or required review.

## Risks

- Local models may hallucinate workflow state, unsafe commands, or false
  evidence; all model output must remain untrusted.
- Developers may overinterpret mock or fake provider contract evidence as real
  model acceptance.
- Private workspace data may leak into datasets if manifests and redaction
  reviews are skipped.
- Fine-tuning can memorize sensitive data or unsafe instructions if training
  eligibility is not strict.
- Public-network Ollama endpoints could create unintended data egress.
- Model and prompt drift can make previous eval scores stale unless datasets,
  prompt packs, and model versions are pinned together.
- Optional local Ollama availability can make evidence inconsistent across
  machines; unavailable local smoke must be recorded as deferred, not failed
  product evidence.

## Related Docs

- [Runtime LLM Flow](runtime-llm-flow.md)
- [Runtime Adapters](runtime-adapters.md)
- [End-to-End Test Batteries](e2e-test-batteries.md)