--- name: web-read description: | Web-page reading skill — open a URL in a headless browser (Playwright Python) and get back the page's **main text**: title, headings, paragraphs, lists, code blocks and tables, as Markdown / plain text / structured JSON. Boilerplate (nav, sidebar, comments, ads, footer) is stripped by a Readability-style pass, and JS-rendered pages work because a real browser runs the page first. Text only. For a still screenshot (`.png` / `.jpg`) use web-screenshot / `web_screenshot`; for a recording (`.mp4` / `.webm`) use web-record / `web_record`. Use this skill immediately whenever the user asks for any of: - Read this link / what does this page say / summarize this article - Fetch page content, extract the article text, get the text of a URL - Use a web page as source material for a script, outline, or video - Read a README / docs page / changelog / blog post - Pull the code samples or the table out of a page - Check what is behind a link before acting on it Even when the user does not say "read", any request that needs the *content* of a URL (rather than a picture of it) should route here. triggers: - Read this link / what does this page say / summarize this article - Fetch page content, extract article text, get the text of a URL - Use a web page as source material for a script or video - Read a README / docs page / changelog / blog post - Pull code samples or tables out of a page --- # Web Read Skill (`web_read`) Turns a URL into text. Entry script **`read_page.py`**, pure Python, prints to **stdout**. > **Text only.** A screenshot is `web_screenshot`, a recording is `web_record`. Those two produce *files*; this one produces *content you can reason about*. **Script location**: this skill has no `scripts/` of its own — it reuses `read_page.py` and the `_media_screenshot/` package from the web-screenshot directory (`skill.json`'s `entry.scriptPath` points relatively at `../web-screenshot/scripts/read_page.py`). Everywhere the commands below say **``**, substitute: ``` /../web-screenshot/scripts/read_page.py ``` > Path convention: read the system-injected `Base directory for this skill: ` as ``. Never hardcode an absolute path. ## Prerequisites - **Python 3.9+** - **The `playwright` pip package + the chromium engine**: the first run **bootstraps automatically** (`pip install playwright` + `playwright install chromium`). ## Basic usage ```bash python3 --url "https://example.com/article/123" ``` Prints a Markdown document: an `# title` line, a `site · author · date · url` line, then the body. A one-line extraction diagnostic (`blocks / chars / container`) goes to **stderr**, so piping stdout gives you clean text. GitHub repository pages also include Stars / Forks / Watchers found on that page, captured before sidebar cleanup. JSON exposes them as `metadata.repository`; Markdown and plain text include a statistics line. Counts retain the page's precision (e.g. `47.2k`); missing counters are unknown, not zero. Reuse a successful read for both the script and template fields. Re-read only for a specific missing required fact, an extraction failure/truncation, or an explicit request for fresh data. Change the relevant selector/limit/wait setting for corrective reads. Do not repeatedly fetch the same page for optional popularity text: reuse returned counters or use factual non-numeric wording. ## Output formats ```bash # Markdown (default) — headings, lists, ``` code fences, | tables | python3 --url "https://docs.python.org/3/tutorial/introduction.html" # Plain text — no markup, for TTS or keyword work python3 --url "https://example.com/post" --format text # JSON — typed blocks + metadata, for programmatic consumption python3 --url "https://example.com/post" --format json ``` The JSON shape is `{url, status, metadata{title,byline,siteName,publishedTime,description,lang}, container, charCount, blocks[]}`, where each block is one of `heading` / `paragraph` / `quote` / `code` / `list` / `table` / `image` / `rule`. JSON is **never truncated** (half a JSON document is not a JSON document) — cap it with `--selector` instead. ## Length control (read this before pointing it at a long page) `--max-chars` defaults to **20000** and cuts on a block boundary, appending an explicit `[truncated] 已显示 N / M 字符` notice. Nothing is silently dropped. ```bash # Keep the whole document on disk, read a bounded slice now python3 \ --url "https://example.com/very-long-guide" \ --max-chars 8000 \ --output "./guide.md" # No cap at all python3 --url "https://example.com/post" --max-chars 0 ``` ## When the automatic extraction misses The container is picked by paragraph-density scoring, which is right on ordinary article/docs/blog pages and can miss on unusual layouts. In order of what to try: An explicit `--selector` opts out of boilerplate pruning inside that scope, so a sidebar, button, or counter can be read intentionally. Non-content elements such as scripts and styles are still removed. ```bash # 1. Content renders late (SPA): wait for the real element python3 --url "https://app.example.com/doc/1" --wait-for-selector "article.body" # 2. Still short: give it a fixed settle window python3 --url "https://example.com/x" --settle-ms 3000 # 3. Wrong part of the page: name the container yourself python3 --url "https://example.com/x" --selector "#main-content" # 4. Behind a login python3 --url "https://example.com/x" --storage-state "./auth.json" python3 --url "https://example.com/x" --cookies '[{"name":"sid","value":"…","domain":"example.com","path":"/"}]' # 5. Site serves headless browsers a stub page python3 --url "https://example.com/x" --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36" ``` The stderr diagnostics tell you which case you are in: `⚠️ 整页有 N 字符，却只抽到 M` means the wrong container was chosen (→ `--selector`), while `⚠️ 这个页面几乎没有文本` means the page itself never rendered text (→ waiting, cookies, or user-agent). ## Links and images Both are dropped by default, because they are noise for a summarize/rewrite task and they inflate the character budget. ```bash # Keep links as [text](url) — when you need to follow them python3 --url "https://example.com/index" --include-links # Keep images as ![alt](src) — when harvesting illustration URLs python3 --url "https://example.com/post" --include-images ``` ## Private addresses are refused `web_read` hands page text to a model, so by default it refuses URLs that resolve to private / loopback / link-local addresses (`localhost`, `10.*`, `169.254.169.254`, …). Reading an intranet page or a local dev server on purpose: ```bash WEB_CAPTURE_ALLOW_PRIVATE_HOSTS=1 python3 --url "http://localhost:5173/" ``` ## Full flag list | Flag | Meaning | |---|---| | `-u, --url` | Target URL (required, http/https) | | `--format` | `markdown` (default) / `text` / `json` | | `--max-chars` | stdout cap, block-aligned (default 20000, `0` = unlimited) | | `--selector` | Extract only inside this CSS selector | | `--include-links` / `--include-images` | Keep `[text](url)` / `![alt](src)` | | `-o, --output` | Write the **full** text to a file (stdout stays capped) | | `--settle-ms` | Extra wait before extracting | | `--quiet` | Suppress the stderr diagnostic line | | `-b, --browser` | `chromium` (default) / `firefox` / `webkit` | | `--device`, `--viewport`, `--color-scheme`, `--user-agent` | Emulation | | `--wait-for-selector`, `--wait-for-timeout`, `--timeout` | Waiting | | `--storage-state`, `--cookies`, `--ignore-https-errors` | Session | ## Exit codes `0` on success — including a page that legitimately has little text. Non-zero only for a refused URL, an unresolvable selector, or a navigation failure; each prints a single actionable line rather than a Python traceback.