---
name: url-get
mcp:
  command: node
  args:
    - ${PLATFORM_ROOT}/lib/mcp-spawn-tee/dist/index.js
    - ${PLATFORM_ROOT}/plugins/url-get/mcp/dist/index.js
  env:
    MCP_SPAWN_TEE_NAME: url-get
    LOG_DIR: ${LOG_DIR}
    PLATFORM_ROOT: ${PLATFORM_ROOT}
    ACCOUNT_ID: ${ACCOUNT_ID}
    SESSION_ID: ${SESSION_ID}
mcp-manifest: auto
description: "Faithful page retrieval. Provides url-get — fetches a server-rendered web page over HTTP, cleans the HTML to markdown with no model in the path (so there is nothing to refuse on copyright), and writes that verbatim copy to an account-scoped reference file. Returns { text, referencePath }: text is the cleaned markdown capped at 100k chars, referencePath is the full verbatim copy on disk. There is no subprocess inference and no summary — a caller that needs a summary invokes url-get from a delegated subagent that holds the full text in its own context and returns only the summary to the main agent. For a large page the agent greps the reference file for passages and librarian reads it directly to ingest. This is the tool to use instead of native WebFetch when a faithful copy of a page is needed (e.g. ingesting the operator's own published writing): WebFetch refuses to reproduce copyrighted text. Server-rendered HTML only — no JavaScript execution (an empty text with textLength=0 means a JS-rendered shell — retry with the browser MCP). A bot-challenge or empty page fails with an explicit error and writes no reference file."
tools:
  - name: url-get
    publicAllowlist: false
    adminAllowlist: true
    riskClass: read
always: true
embed: false
---

# url-get

## What it does

A single MCP tool, `url-get`, takes a URL and returns one cleaned artefact in
two surfaces, with no model anywhere in the path:

- **Verbatim** — an HTTP GET followed by HTML→markdown cleanup, written to an
  account-scoped reference file at `$ACCOUNT_DIR/url-get/<url-hash>.md`. No model
  sits in this path, so there is nothing for a copyright refusal to act on. This
  file is the ground truth.
- **Inline text** — the same cleaned markdown returned in `text`, capped at 100k
  chars so a large page does not flood the caller's context. No inference runs:
  url-get does not summarise.

`{ text, referencePath }` returns to the conversation. The full verbatim copy is
never injected wholesale beyond the cap: the agent greps the reference file when
it needs a specific passage, and `librarian` reads the file directly when the
operator chooses to ingest it. When a summary is wanted, the caller invokes
url-get from a delegated subagent — the subagent holds the full `text` in its own
context, summarises there, and returns only the summary to the main agent. This
keeps both the verbatim text and the summarisation model out of the main
conversation, and keeps `claude --print` out of url-get entirely.

## Why it exists

The only built-in page-retrieval tool is Claude's native WebFetch, whose refusal
to reproduce copyrighted text lives in an internal post-fetch model — not in the
source site and not in the outer prompt, so reframing cannot move it. A knowledge
worker ingesting their own published writing (e.g. a Substack post) hits the same
wall, because WebFetch cannot verify authorship. WebFetch is also structurally
wrong for ingestion even when it does not refuse: it returns only a summary, never
the verbatim source. `url-get` keeps WebFetch's useful behaviour (HTML cleanup),
adds the verbatim artefact, and removes the model from the path entirely —
summarisation, when needed, is the delegated subagent's job, not url-get's.

## Surface boundaries

- **Server-rendered HTML only.** `url-get` does not execute JavaScript — the same
  coverage limit as a plain fetch. JS-rendered pages are handled by the
  pre-installed browser MCP.
- **No graph write.** `url-get` writes a reference file and returns cleaned text;
  it never touches Neo4j. Ingestion is `librarian`'s job, reading the reference file.
- **Verbatim on disk is not size-capped** — the reference file holds the full
  cleaned page. The inline `text` is capped at 100k chars only to bound what
  enters the caller's context; the reference file remains the uncapped source.
- **No caching.** Every call fetches fresh.
- A page that returns a bot-challenge interstitial or an empty body fails with an
  explicit error (`challenge` / `empty`) and writes no reference file — never a
  reference file full of challenge-page HTML.

## Failure modes

`url-get` returns an error envelope (never throws to the caller) for: `fetch`
(network/DNS/TLS/timeout), `http` (non-2xx status), `non-html` (unsupported
content-type), `challenge` (bot interstitial), `empty` (no readable text), and
`no-account-dir` (storage unresolvable). There is no inference step to fail: once
the reference file is on disk the call succeeds and returns its cleaned `text`.
An HTTP-200 page with no extractable body returns `text: ""` with `textLength: 0`
— the JavaScript-rendered-shell signal — and the caller retries with the browser MCP.

## Enabling

`always: true` — the plugin auto-installs on every brand and is reachable by the
spawned PTY agent on device; it is never pulled mid-session. The tool is
admin-allowlisted (`adminAllowlist: true`) and is on the `tools:` list of the
`research-assistant`, `librarian`, and `personal-assistant` specialists. It is
not exposed to public agents (`publicAllowlist: false`).