# Test Prompt 001 — Group 1: Basic API (HTTP Contract Tests) **Phase:** 1 **Focus:** Pure HTTP contract verification — API surface, field shapes, auth enforcement, CRUD persistence **Agents running tasks:** No — this phase does not ask agents to do intelligent work --- ## Your Role You are a test engineer executing Phase 1 of the VeilCLI test suite. Set up a clean workspace, start the server, and run HTTP contract tests against the live API. You decide how to implement the tests — curl, scripts, Node.js, a mix — whatever gets the job done cleanly. What matters is the quality of verification, not the implementation method. Do NOT just check HTTP status codes and call it a day. Each test below specifies what a **genuine pass** looks like. A test that returns HTTP 200 but doesn't verify the actual field values is not a passing test — it's a false positive. --- ## Environment - **VeilCLI source:** `/home/ixi/khacloud/drive/Plugins/VeilCli` - **How to invoke:** Try `veil` first (it is globally linked) - **Reference auth.json** (contains real API keys — you must copy this): `/home/ixi/khacloud/drive/Plugins/VeilCli/.veil/auth.json` - **API DOCS:** `/home/ixi/khacloud/drive/Plugins/VeilCli/docs/api/` check them to ensure your tests are right --- ## ⚠️ Non-Blocking Execution Rule **Any command that takes time must be run in the background and monitored by polling — never block waiting for it.** This applies to: - Starting the server (`veil start` → run in background, poll `GET /health` in a loop until ready) - Running test scripts — if a script can hang, run it with a timeout or in background and tail its output If a command blocks and gets stuck, you get stuck too. Background + poll is the pattern for everything time-sensitive in this phase. --- ## Step 1 — Create the Test Workspace Create the workspace at: ``` /home/ixi/khacloud/drive/Plugins/VeilCli_TESTS/workspace-test-001 ``` Inside it, create `.veil/settings.json` with the following intent: - Port **5151** (never 5050 — that's the default and may conflict with a running instance) - A secret value of your choice (you'll need it for the auth tests — write it down) - Reasonable iteration and duration limits for tests - Permissive tool permissions (allow all) Copy the reference `auth.json` into `.veil/auth.json`. Do **not** copy agents, memory, or anything else — the workspace must be clean. --- ## Step 2 — Start the Server From **inside** `/home/ixi/khacloud/drive/Plugins/VeilCli_TESTS/workspace-test-001`, run `veil start` in the background. The server reads workspace config from the current directory. Confirm the server is up by polling `GET http://localhost:5151/health`. Do not proceed until it responds. If it doesn't respond within 20 seconds, treat this as a fatal failure, capture the startup output, and stop. --- ## Step 3 — Run the Tests Run all tests below and record results. All requests go to `http://localhost:5151`. Include the `X-Veil-Secret: ` header on all requests **unless the test explicitly says not to** (the auth tests need to test the absence of it). --- ### Test 01 — Health & Status **What this catches:** `/health` is a liveness probe — it must be fast and require no database access. `/status` provides a runtime snapshot. If either is missing fields or counts don't change with state changes, observability is broken. **Verify:** - `GET /health` responds with HTTP 200. No auth header needed on this endpoint. Should respond immediately (under 1 second). - `GET /status` responds with a JSON body. Verify it contains all of: `uptime`, `cwd`, and numeric counts for agents, sessions, and tasks. All counts should be 0 or reflect actual state — not `null` or `undefined`. - Create one agent via the API (any minimal valid config). Then call `GET /status` again and verify the agent count increased by exactly 1. This confirms the status snapshot reflects live state, not cached data. --- ### Test 02 — Auth Enforcement **What this catches:** If the secret middleware has a bug (wrong header name, wrong comparison, only applied to some routes), requests that should be rejected get through silently. **Verify:** - Call `GET /agents` with **no auth header** → must return HTTP 401. Not 200, not 403, not 500. - Call `GET /agents` with the header `X-Veil-Secret: wrong-value` → must return HTTP 401. - Call `GET /agents` with the correct `X-Veil-Secret: ` → must return HTTP 200. - Repeat the unauthorized test on a **different endpoint** (e.g. `GET /sessions`) to confirm the middleware applies globally, not just to the agents route. - `GET /health` should return 200 even without an auth header — confirm this endpoint is intentionally unprotected. --- ### Test 03 — Agent CRUD **What this catches:** Agent creation, persistence, update, and deletion. The common failure mode: create returns 201, but the agent was never actually written to disk/DB — a subsequent GET returns nothing. **Verify:** *Create:* - `POST /agents` with a minimal valid agent config (name, model, at least one mode enabled). Expect HTTP 201. The response must contain a `name` field matching what you sent. If you receive 201 but no `name` field, that is a FAIL. *Read after create:* - `GET /agents` → response must be an array containing the agent you just created (match by name). An empty array here after a successful POST is a FAIL. - `GET /agents/:name` → response must include key config fields that match what you sent in the POST body. Verify at least `name` and `model` match exactly. *Update:* - `PUT /agents/:name` changing a specific field (e.g. `temperature` or `description`). Expect HTTP 200. - Immediately `GET /agents/:name` again → verify the field you changed now has the new value. If it still has the old value, the update was not persisted. *Delete:* - `DELETE /agents/:name` → expect HTTP 200 or 204. - `GET /agents/:name` → must return HTTP 404. If it returns 200 with data, the deletion failed silently. - `GET /agents` → the deleted agent must not appear in the list. --- ### Test 04 — Settings CRUD **What this catches:** Settings have a layered merge system (global → project → local → CLI flags). Read/write must work correctly, API keys must be redacted, and live updates must apply without a server restart. **Verify:** *Read:* - `GET /settings` → response must include top-level fields like `port`, `maxIterations`, `permissions`. Verify `port` is `5151` (matching what you set). - The response must NOT contain raw API key values. Any field representing an API key (e.g. inside `models.main.api_key`) must be redacted (e.g. `"***"` or omitted entirely). If a real key string appears in the response, that is a security FAIL. - `GET /settings?level=project` → should return only what is in your project-level `settings.json`, not merged defaults. - `GET /settings?level=merged` → should include defaults. The merged result should have more fields than the project-level result. *Live update:* - `PUT /settings` with a change to a safe field like `maxIterations` (use a recognizable value like `42`). - Immediately `GET /settings` and verify the change is reflected — no server restart required. If it still shows the old value, live reload is broken. --- ### Test 05 — Session CRUD **What this catches:** Sessions are the backbone of chat continuity. Pre-creating sessions (before any chat), reset, and deletion must all work correctly. Broken session management means broken history, broken resumption, and broken context. **Verify:** *Create:* - `POST /sessions` with at least an `agent` field pointing to an existing agent. Expect a response containing a session `id`. A response without an `id` field is a FAIL. *Read:* - `GET /sessions` → list must contain the created session. - `GET /sessions/:id` → verify the response contains: `id`, `agent`, `status`, and a message count field. Verify the agent name matches what you used to create it. *Reset:* - `POST /sessions/:id/reset` → expect HTTP 200. - After reset, `GET /sessions/:id` → verify message count is 0 and the session still exists (was not deleted). The session being gone after a reset is a FAIL. *Delete (soft):* - `DELETE /sessions/:id` → expect HTTP 200 or 204. - `GET /sessions/:id` → if soft delete is supported, status should reflect it. Note the behavior you observe. *Delete (hard):* - Create a second session, then `DELETE /sessions/:id?hard=true` → expect HTTP 200 or 204. - `GET /sessions/:id` must return HTTP 404. If the session is still retrievable, hard delete is broken. --- ### Test 06 — Models Endpoint **What this catches:** The models index is used internally for context window size limits and cost calculation. If the endpoint is broken or returns empty, token tracking and cost estimation silently fail. **Verify:** *List:* - `GET /models` → response must contain a non-empty array of models. An empty array is a FAIL — it means the local model index was never populated. - Pick any model from the list and verify it has at minimum these fields with non-null values: `id`, `name`, `context_length`, and a `pricing` object. - The response should also have an `updated_at` or similar timestamp indicating when the index was last fetched. *Single model lookup:* - From the list above, take a model and identify its provider and name from its `id` (typically `provider/model-name` format). - `GET /models/:provider/:name` → should return that specific model's details. A 404 here when the model exists in the list is a FAIL. --- ## Step 4 — Stop the Server After all tests are complete, stop the server cleanly (`veil stop` from the workspace directory or send SIGTERM). --- ## Step 5 — Write the Summary Create a file `summary.md` in `/home/ixi/khacloud/drive/Plugins/VeilCli_TESTS/workspace-test-001/` with the following structure: ```markdown # Test Run Summary — workspace-test-001 **Date:** **Port:** 5151 **Phase:** Group 1 — Basic API (HTTP Contract Tests) ## Results | Test | Status | Notes | |------|--------|-------| | Test 01: Health & Status | PASS / FAIL | | | Test 02: Auth Enforcement | PASS / FAIL | | | Test 03: Agent CRUD | PASS / FAIL | | | Test 04: Settings CRUD | PASS / FAIL | | | Test 05: Session CRUD | PASS / FAIL | | | Test 06: Models Endpoint | PASS / FAIL | | ## Failures For each FAIL, write: - Which test and which specific assertion failed - What value was expected - What value was actually returned - Whether this looks like a VeilCLI bug or a test setup issue ## Observations Anything unexpected you noticed that wasn't a hard FAIL but seems worth flagging — odd field names, inconsistent behavior, missing documentation cues, things that were confusing to test against, etc. ``` --- ## Notes on Avoiding False Positives - **Never accept HTTP status code alone as a pass.** Always read the response body and verify the fields that matter. - **Mutations must be confirmed.** After every PUT or DELETE, do a GET to verify the state actually changed. - **Counts must be exact.** If you create 1 agent and the count goes from 0 to 2, that's a bug. - **404 must be a real 404.** After deletion, if the endpoint returns 200 with an empty body or `{}`, that is NOT the same as 404 — it is a fail. - **Settings redaction must be verified.** Don't assume keys are redacted — actively check the response for strings that look like real API keys. - If something fails, **do not work around it** — record it as a failure and move on to the next test. The goal is an accurate picture of what works and what doesn't.