# Pi Web Access — Manual End-to-End Test Plan

This checklist exercises every registered tool, command, opt-in feature, and safety control without using a production repository, secret, or copyrighted media. The opt-in flags are already enabled in `~/.pi/pi-web-access.json`; start with `/reload` so the loaded extension rereads that file.

## 1. One-time setup

### Required local tools (Ubuntu)

```bash
sudo apt update
sudo apt install -y poppler-utils ffmpeg git curl gh yt-dlp jq
# If your Ubuntu release does not provide yt-dlp, install the current release instead:
# sudo apt install -y pipx && pipx ensurepath && pipx install yt-dlp

gh auth login                    # only for PR tests
```

`poppler-utils` provides `pdftotext`. Open a new shell after `pipx ensurepath` so `yt-dlp` is on `PATH`.

Confirm them:

```bash
command -v git pdftotext ffmpeg yt-dlp gh curl
pdftotext -v
ffmpeg -version | head -1
gh --version
gh auth status                  # required only for PR tests
```

### Load the extension and create its Ubuntu configuration

On Ubuntu, the configuration is per Linux user at `~/.pi/pi-web-access.json`; it is not shared with the macOS configuration created earlier.

1. Place/load the extension using its normal Pi installation method, start Pi in this project, and verify that the `web_fetch`, `web_fetch_many`, `web_get_content`, `github_repo`, `video_understand`, and `web_search` tools are present.
2. Run `/web-config`. On a new Ubuntu account, use the menus below to create the configuration; select **Yes** for every desired toggle.
   - **Core toggles:** enable persistent cache, per-domain rate limiting, and request logging.
   - **Limits and cache:** use the safe test defaults: cache TTL `300000`, requests/minute `10`, timeout `10000`, GitHub maximum repository size `350`, and the default user agent.
   - **Domain policy and security:** leave whitelist disabled and empty, retain the blacklist. The whitelist/blacklist tests later deliberately change these settings and then restore them.
   - **Optional features:** enable GitHub, video understanding, YouTube download, PDF extraction, alternate-user-agent fallback, and Brave/Parallel search.
   - **Search provider credentials:** paste the disposable Brave and/or Parallel key. A blank field intentionally keeps the existing key unchanged.
   - **Video provider credentials:** select **Gemini** and paste the disposable Gemini key. Select **None** only when testing the no-provider failure path.
3. Select **Save changes**. Limits apply immediately and optional feature settings are read for each tool call; `/reload` is not required.
4. Verify file permissions and non-secret settings without printing keys:
   ```bash
   stat -c '%a %n' ~/.pi/pi-web-access.json
   # expected: 600 /home/YOUR_USER/.pi/pi-web-access.json
   jq 'del(.braveApiKey, .parallelApiKey, .videoApiKey)' ~/.pi/pi-web-access.json
   ```
   If the mode is not `600`, fix it with `chmod 600 ~/.pi/pi-web-access.json`.

### Provider-specific configuration and preflight

#### Gemini

1. Sign in to Google AI Studio and create a key at <https://aistudio.google.com/apikey>.
2. Confirm that the Gemini API no-cost tier is available for your account/region at <https://ai.google.dev/gemini-api/docs/pricing>.
3. In `/web-config` → **Video provider credentials**, choose **Gemini**, enter the key, and select **Save changes**.
4. Use only a small local MP4 within the configured 1–18 MiB video limit. The extension sends the video bytes and prompt to Gemini, so do not use confidential footage.

Custom video providers are a programmatic extension API. The built-in configuration UI intentionally exposes only Gemini.

#### Brave and Parallel

1. Create separate test keys, set spending/usage alerts in each provider dashboard, and paste them only into `/web-config`.
2. For Brave, use its browser documentation/signup flow: <https://api.search.brave.com/app/documentation>. Verify the current trial/free quota there; it changes over time.
3. For Parallel, create a key at <https://platform.parallel.ai>, review <https://docs.parallel.ai/getting-started/overview>, and check its current free allowance at <https://parallel.ai/pricing>.
4. Configure both keys to test `provider:"auto"` fallback. Configure one key at a time to test each explicit provider and the missing-key error.
5. If either provider returns an API-version, authorization, or quota error, record the exact response in the report. Do not replace the extension endpoint or send a fake key just to make the test pass.

#### GitHub and PR authentication

1. Create a private or public throwaway repository in your own account, e.g. `pi-web-access-sandbox`, with a README and an initial commit.
2. Authenticate the Ubuntu machine:
   ```bash
   gh auth login --web --git-protocol https
   gh auth status
   git config --global user.name 'Pi Web Access Test'
   git config --global user.email 'YOUR_TEST_EMAIL@example.invalid'
   ```
3. Enable GitHub in `/web-config`, select **Save changes**, and first run a read-only public-fixture test. Use the sandbox only for edit/branch/PR cases.
4. The extension uses private managed temporary clones. They are removed by `/web-cleanup` and session shutdown; do not depend on a fixed filesystem path.

#### PDF, frames, and YouTube

- `pdftotext` must be the `poppler-utils` binary; test it with `pdftotext TEST.pdf -` after downloading a permitted PDF.
- `ffmpeg` must be on `PATH` for frame extraction. The plan uses an MP4 under `/tmp` so no special Linux permissions are needed.
- `yt-dlp` must be on `PATH` for the enabled YouTube-download path. Use an owned or explicitly downloadable video only. If the test is metadata-only, temporarily disable YouTube download and reload.

### Test-only accounts and keys

Do **not** put a production credential in the extension config. Use `/web-config` to set the following only for the test session, then preserve the config file mode (`chmod 600 ~/.pi/pi-web-access.json`).

| Capability | What to configure | Free testing option |
|---|---|---|
| Gemini video analysis | Select **Gemini** and enter a Gemini API key | Google AI Studio has a no-cost tier with limits: <https://aistudio.google.com/apikey> and <https://ai.google.dev/gemini-api/docs/pricing>. |
| Brave search | Enter a Brave Search API key | Create a key at Brave Search API and check the current free/trial allowance before use: <https://api.search.brave.com/app/documentation>. The documentation site may reject automated requests, so open this link in a browser. |
| Parallel search | Enter a Parallel API key | Parallel currently advertises up to 5,000 requests/month free plus monthly credits; verify before use: <https://parallel.ai/pricing>. Create a key at <https://platform.parallel.ai>. |
| GitHub PR | A separate GitHub account/repository and `gh auth login` | GitHub accounts and public repositories are free. Create a repository named, for example, `pi-web-access-sandbox`; never use a real project. |

Free/public test content:

- HTML / links: <https://example.com>
- JSON: <https://api.github.com>
- Public test PDF: <https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf>
- Small downloadable, no-license-restriction sample videos: <https://samplelib.com/sample-mp4.html>. Use the 1 MB `sample-5s-360p.mp4`; Gemini inline analysis is limited by this extension to 18 MiB.
- Public GitHub fixture: <https://github.com/octocat/Hello-World>
- A YouTube test must be media you own or are permitted to download. Do not use a random copyrighted video merely to test `yt-dlp`.

## 2. Start and verify configuration

1. In Pi, run `/reload`.
2. Run `/web-status`. Expected: it lists `github, video, pdf, fallback, search` as optional features.
3. Run `/web-config` and verify all six feature toggles are enabled. Configure only the test keys above. Choose **Gemini** for the video provider if testing analysis.
4. After every configuration change, run `/reload` again. The extension constructs its `ConfigManager` at load time.

Record the date, Pi version, Node version, and the result of each checklist item. Run `/web-cleanup` at the end.

## 3. Core retrieval (`web_fetch`)

Ask Pi to call each tool with the exact arguments shown. Mark a case passed only when the expected outcome occurs.

| ID | Tool input | Expected result |
|---|---|---|
| WF-01 | `web_fetch({url: "https://example.com", extract: "text"})` | Readable text/Markdown containing **Example Domain**. |
| WF-02 | `web_fetch({url: "https://example.com", extract: "html"})` | Raw HTML is returned. |
| WF-03 | `web_fetch({url: "https://api.github.com", extract: "json", max_length: 1000})` | Pretty-printed JSON, bounded near 1,000 characters. |
| WF-04 | `web_fetch({url: "https://example.com", extract: "links"})` | The IANA link appears. |
| WF-05 | `web_fetch({url: "https://example.com", extract: "headers"})` | `# Example Domain` appears. |
| WF-06 | Repeat WF-01 without `bypass_cache`. | Result identifies itself as cached. |
| WF-07 | Repeat WF-01 with `bypass_cache: true`. | Fetch succeeds and is not labeled cached. |
| WF-08 | `web_fetch({url: "https://localhost"})` | Fails with a blocked internal-hostname/SSRF error. |
| WF-09 | `web_fetch({url: "https://example.com/does-not-exist"})` | Fails with HTTP 404; with fallback enabled the diagnostic contains both direct and alternative attempts. |

Optional controls (restore the original setting immediately after each):

- **Whitelist:** enable it, set `whitelist` to `example.com`, verify WF-01 passes and `api.github.com` is denied; then disable it.
- **Blacklist:** add `example.com`, verify WF-01 is denied; remove it afterward.
- **Rate limit:** set max requests/minute to 1; fetch two distinct uncached `example.com` URLs with `bypass_cache: true`; the second should be rate-limited. Restore 10.
- **Timeout and redirects:** use a public endpoint you control. Confirm a slow response aborts at the configured timeout and every redirect target is public. Do not point this test at a private address.
- **Authentication:** use only a disposable basic-auth test endpoint/account. Verify success with the `auth` object and failure without it; never paste an API key into chat output.

## 4. Multi-fetch and cached-content tools

| ID | Tool input | Expected result |
|---|---|---|
| WM-01 | `web_fetch_many({urls:["https://example.com","https://example.com/does-not-exist"], extract:"text"})` | First result succeeds; second is reported as an individual error, not a total failure. |
| WC-01 | After WF-01, `web_get_content({url:"https://example.com", extract:"text", offset:0, limit:100})` | Returns only the requested first slice. |
| WC-02 | `web_get_content({url:"https://example.com", extract:"text", find_text:"documentation", limit:300})` | Returns a matching passage and a positive match count. |
| WC-03 | Request an uncached URL/mode. | Fails with the clear “Run web_fetch first” message. |

## 5. PDF extraction (`web_fetch`)

1. Ensure `pdftotext` is installed and `/reload` Pi.
2. Call `web_fetch({url:"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", max_length:1000})`.
   - Expected: a `PDF content from` result with `application/pdf` details.
3. Repeat with `page_start: 1, page_end: 1`; expect the first page only.
4. Call with `page_start: 2, page_end: 1`; expect a validation error saying the end cannot precede the start.
5. Temporarily run without `pdftotext` (or use a machine without it) to verify the actionable missing-binary error; reinstall afterward.

## 6. GitHub (`github_repo`)

### Read-only public fixture

Use `https://github.com/octocat/Hello-World` and run:

| ID | Action | Expected result |
|---|---|---|
| GH-01 | `tree` | A small tree and README preview. |
| GH-02 | `read` with `path:"README"` | `Hello World!` is returned. |
| GH-03 | `history` | At least one commit line. |
| GH-04 | `branches` | `master`/`main` and/or remote branch lines. |
| GH-05 | `read` with `path:"../../etc/passwd"` | Path-escape rejection; no local file is read. |

### Destructive sandbox only

For `switch`, `edit`, `pull_requests`, and `create_pull_request`, use only `YOUR_ACCOUNT/pi-web-access-sandbox`:

1. Call `tree` once so the extension creates a managed session clone.
2. Call `edit` on a disposable file such as `extension-test.txt`, then `read` it to verify the exact content. Confirm the prompt; do not use a production repository.
3. To test `switch` or `create_pull_request`, prepare and push a sandbox branch outside the extension first, then use a GitHub URL that selects that branch (for example a `/tree/BRANCH` URL). Confirm the prompt before the action.
4. Call `pull_requests`; it should list PRs when `gh` is authenticated.

The extension does **not** create branches, commits, or pushes automatically. `create_pull_request` only opens a PR for an already-pushed sandbox branch.

## 7. Search (`web_search`)

Use a harmless query such as `Pi Coding Agent` after configuring keys.

| ID | Tool input | Expected result |
|---|---|---|
| WS-01 | `web_search({query:"Pi Coding Agent", provider:"brave"})` | Brave results, titles, URLs, and descriptions. |
| WS-02 | `web_search({query:"Pi Coding Agent", provider:"parallel"})` | Parallel results. |
| WS-03 | `web_search({query:"Pi Coding Agent", provider:"auto"})` | Uses Brave first, then Parallel if Brave fails. |
| WS-04 | Add `domain_filter:["github.com"]`. | Every returned hostname is `github.com` or a subdomain. |
| WS-05 | Add `domain_filter:["-github.com"]`. | No returned hostname is `github.com` or a subdomain. |
| WS-06 | Temporarily remove both keys. | Clear “not configured” error and no unauthenticated request succeeds. Restore the test keys. |

Check provider dashboards after testing so the free-tier consumption is understood. Free quotas and offers can change.

## 8. Video (`video_understand`)

### Local Gemini analysis and frames

1. Download a permitted small sample locally:
   ```bash
   curl -L https://samplelib.com/lib/preview/mp4/sample-5s-360p.mp4 -o /tmp/pi-web-access-sample.mp4
   ls -lh /tmp/pi-web-access-sample.mp4
   ```
2. In `/web-config`, select **Gemini**, enter the test API key, then `/reload`.
3. Call:
   ```text
   video_understand({
     url:"/tmp/pi-web-access-sample.mp4",
     question:"Describe the scene in one sentence.",
     timestamp:"0",
     extract_transcript:true,
     extract_frames:2
   })
   ```
4. Expected: Gemini answer plus two local frame paths. Open the frame files to confirm they exist and look like the sample video.
5. Test the 18 MiB protection using a permitted file above that size; expect the inline-size-limit error before an API call.
6. Test an invalid timestamp such as `bad:time`; expect an invalid-timestamp error from frame extraction.

### YouTube download path

With `youtubeDownloadEnabled` on and `yt-dlp` installed, use only a video you own or are explicitly licensed to download. Call `video_understand` with its YouTube URL. Expected: download, analysis, and optional frame extraction. Delete downloaded temporary media afterward. To test metadata-only behavior instead, temporarily turn **off** YouTube download and call the same URL; expected output is oEmbed title/channel metadata and no media download.

## 9. Commands, persistence, and cleanup

1. Run `/web-status` before and after several fetches; cache count should increase.
2. Restart Pi and repeat WC-01. With cache enabled and an unexpired entry, persistent cache should be available.
3. Run `/web-cleanup`; `/web-status` should report zero cache entries.
4. Repeat WC-01; it must now report no cached content.
5. Confirm `~/.pi/pi-web-access.json` is mode 600 and contains only intended test keys/settings. Remove test API keys when finished.

## 10. Pass criteria and report template

The extension passes a manual run when all applicable **Expected result** statements pass, known missing dependencies are resolved before the corresponding positive tests, and no test reads a private URL or modifies a non-sandbox repository.

```text
Date / Pi version / Node version:
Configuration reloaded: PASS | FAIL
WF: __/__   WM/WC: __/__   PDF: __/__   GH: __/__
Search: __/__   Video: __/__   Commands/cleanup: __/__
Free-tier usage observed:
Failures (case ID, exact error, reproduction steps):
Cleanup completed and test keys removed: YES | NO
```