# Test Prompt 002 — Group 2: Basic Flows (End-to-End Happy Paths) **Phase:** 2 **Focus:** End-to-end flows that simulate real user behavior — chat lifecycle, async task execution, SSE streaming, session resumption **Agents running tasks:** Yes — agents will be doing real work in this phase --- ## Your Role You are a test engineer executing Phase 2 of the VeilCLI test suite. This phase goes beyond HTTP contract tests — agents must actually run, produce output, and you verify that the full pipeline (request → agentic loop → tool execution → response) works correctly end-to-end. These are the tests that catch runtime failures that the HTTP surface hides. A 200 response means nothing if the agent returned an empty output, the session wasn't persisted, the SSE stream never sent any chunks, or the tool pipeline silently dropped the result. --- ## Environment - **VeilCLI source:** `/home/ixi/khacloud/drive/Plugins/VeilCli` - **How to invoke:** `veil` (globally linked) - **Reference auth.json:** `/home/ixi/khacloud/drive/Plugins/VeilCli/.veil/auth.json` - **API DOCS:** `/home/ixi/khacloud/drive/Plugins/VeilCli/docs/api/` check them to ensure your tests are right - **Agent schema & examples:** `/home/ixi/khacloud/drive/Plugins/VeilCli/schemas/agent.json` and `/home/ixi/khacloud/drive/Plugins/VeilCli/examples/` --- ## ⚠️ Non-Blocking Execution Rule **Any command that takes time must be run in the background and monitored by polling — never block waiting for it.** This applies to: - Starting the server (`veil start` → run in background, poll `GET /health` in a loop until ready) - Running test scripts — if a script can hang, run it with a timeout or in background and tail its output If a command blocks and gets stuck, you get stuck too. Background + poll is the pattern for everything time-sensitive in this phase. --- ## Step 1 — Create the Test Workspace Create the workspace at: ``` /home/ixi/khacloud/drive/Plugins/VeilCli_TESTS/workspace-test-002 ``` Inside it, create `.veil/settings.json`: - Port **5252** - A test secret of your choice - Reasonable iteration and duration limits (agents need enough room to complete short tasks) - Permissive tool permissions Copy the reference `auth.json` into `.veil/auth.json`. --- ## Step 2 — Create the Test Agents You need two agents for this phase. Create them as agent folders inside `.veil/agents/`. Check the schema and examples for valid `agent.json` structure. **Agent 1 — `chat-basic`** - Chat mode enabled - No tools needed (this agent just talks, doesn't use tools) - Memory disabled (keep it simple) - Use the default/main model from auth.json **Agent 2 — `task-runner`** - Task mode enabled - Must have file I/O tools available: at minimum `write_file`, `read_file`, `list_dir` - Memory disabled - Use the default/main model Give each agent a brief `AGENT.md` so it has a clear identity and doesn't confuse itself. Keep the prompts minimal — just name and role. --- ## Step 3 — Start the Server From inside `workspace-test-002`, run `veil start` in the background. Confirm `GET http://localhost:5252/health` responds before proceeding. If it doesn't come up within 20 seconds, that's a fatal failure — capture the server output and stop. --- ## Step 4 — Run the Tests All requests to `http://localhost:5252`. Include the `X-Veil-Secret` header on all requests. --- ### Test 01 — Chat Happy Path (Multi-Turn) **What this catches:** The most critical regression in VeilCLI. If a new user creates an agent and sends a message and gets nothing back — or can't continue a conversation — the product is broken. Also catches the silent failure where the session isn't actually persisted after a successful response. **Setup:** Use the `chat-basic` agent. **Verify:** *First message:* - `POST /agents/chat-basic/chat` with a simple conversational message. Expect HTTP 200. - The response must contain a non-empty `message` field. An empty string is a FAIL. - The response must contain a `sessionId`. Missing sessionId means the conversation cannot be continued. - Verify token counts on the response — both input and output tokens must be non-zero. *Second message (session continuation):* - Send a second message using the `sessionId` from the first response. Ask something that only makes sense if the agent read the first message (e.g. reference something specific you said in message 1). - Response must be non-empty. - `GET /sessions/:id/messages` — the message history must contain exactly 2 user turns and 2 assistant turns, in the correct order (user → assistant → user → assistant). If the second message created a new session instead of continuing, message count will be wrong. - Verify each assistant message has non-zero output_tokens. Verify each user message has non-zero input_tokens. --- ### Test 02 — Async Task Full Lifecycle **What this catches:** Task mode is async — the status must transition correctly from pending → processing → finished, output must be non-empty, and the event log must capture the full execution trace. Previous test approaches would check the 202 response and stop there, missing silent failures in the execution loop. **Setup:** Use the `task-runner` agent. **The task to give the agent:** Ask it to write a specific, recognizable string (e.g. `"VEIL_TASK_MARKER_002"`) into a file named `task-output.txt` inside the workspace directory, then confirm what it wrote. This task is deliberately designed to: 1. Force tool use (`write_file`) 2. Leave a verifiable side effect on disk 3. Be short enough to complete quickly **Verify:** *Creation:* - `POST /agents/task-runner/task` → expect HTTP 202. Response must contain a `taskId`. The initial status must be `pending` (not already `finished` — that would mean it ran synchronously which is wrong). *Polling to completion:* - Poll `GET /tasks/:id` at a reasonable interval until status reaches a terminal state (`finished`, `failed`, or `canceled`). Set a polling timeout of 90 seconds. If it times out, record the last known status and mark as FAIL. - Final status must be `finished`. If it's `failed`, the output/error field should explain why — record it. *Output verification:* - The task's `output` field must be non-empty. An empty output with `finished` status is a silent failure — the agent "completed" but produced nothing. - Verify token_input and token_output on the task record are both non-zero. - Verify `iterations` on the task record is at least 1. *Event trace:* - `GET /tasks/:id/events` — the event list must not be empty. - Verify at least one `status.change` event exists showing the transition from `pending` to `processing`. - Verify at least one `tool.start` event for `write_file` exists — this confirms the agent actually attempted to use the tool, not just described what it would do. - Verify the corresponding `tool.end` event for `write_file` exists and does not contain an error result. *Filesystem side effect:* - Check that `task-output.txt` actually exists in the workspace. Its content must contain the marker string you specified. This is the definitive verification that the tool pipeline worked end-to-end — not just that the agent said it ran the tool. --- ### Test 03 — Chat SSE Streaming **What this catches:** SSE (Server-Sent Events) is an entirely separate code path from the regular JSON chat response. It's the primary interface for any streaming UI client. If it's broken, the UI shows nothing while non-streaming works fine — and a simple HTTP status check would miss this. **Setup:** Use the `chat-basic` agent. **Verify:** *Response type:* - `POST /agents/chat-basic/chat` with `"sse": true` in the request body. The response must have `Content-Type: text/event-stream`. If it returns `application/json`, the SSE code path is not executing. *Stream content:* - Consume the event stream until a `done` event is received (or a timeout, 60 seconds max). - There must be at least one `chunk` event (or equivalent token/delta event) received before the `done`. A stream that goes straight to `done` with no chunks means content was never streamed — it was buffered and sent all at once (or not at all). - The `done` event must contain: a final `message` (non-empty) and token usage data (`tokenUsage` or equivalent). A `done` event with an empty message is a FAIL. *Functional equivalence:* - The final assembled content from all chunks should be meaningfully similar to what a non-SSE chat response would return for the same prompt. You don't need to compare them exactly — but if the SSE version returns a completely empty or single-character response while non-SSE returns a full paragraph, something is wrong. --- ### Test 04 — Session Resumption **What this catches:** A user closes their client, comes back later, and resumes. The session must be loadable, the history must be intact, and after a reset the agent must start fresh. This catches two distinct failure modes: history not persisting (messages lost on retrieval) and reset not working (messages still present after clear). **Setup:** Use the `chat-basic` agent. **Verify:** *History persistence:* - Start a conversation with 3 message exchanges (3 user turns, 3 assistant responses). Use the same `sessionId` throughout. - After the third exchange, `GET /sessions/:id/messages` → the history must contain all 6 messages (3 user + 3 assistant) in the correct chronological order. - Verify the `role` field on each message is either `user` or `assistant`. No message should have a null or missing role. *Cold resumption:* - Without using the sessionId for a few seconds (simulate reconnect), send a new message using the same `sessionId`. - This must work as a continuation — no error about session not found, no empty response. - After this 4th exchange, message count must be 8 (4 user + 4 assistant). If it resets to 2, the session lookup on resume is creating a new session. *Reset:* - `POST /sessions/:id/reset` → expect HTTP 200. - `GET /sessions/:id/messages` → message count must be 0. The session itself must still exist (status is not `deleted`). - Send one more message on the same session after reset. It must work normally as a fresh start. The agent should not reference anything from the prior conversation — if it does, the reset didn't actually clear the context used in the prompt. --- ## Step 5 — Stop the Server Stop the server cleanly from the workspace directory after all tests complete. --- ## Step 6 — Write the Summary Create `summary.md` in `workspace-test-002/`: ```markdown # Test Run Summary — workspace-test-002 **Date:** **Port:** 5252 **Phase:** Group 2 — Basic Flows ## Results | Test | Status | Notes | |------|--------|-------| | Test 01: Chat Happy Path | PASS / FAIL | | | Test 02: Async Task Lifecycle | PASS / FAIL | | | Test 03: SSE Streaming | PASS / FAIL | | | Test 04: Session Resumption | PASS / FAIL | | ## Failures For each FAIL: - Which test and specific assertion - Expected vs actual - Whether it looks like a VeilCLI bug or test setup issue ## Observations Anything unexpected — agent behavior oddities, timing issues, fields that behaved differently than the docs described, things that were harder to verify than expected. ```