# MCP Scraper MCP Scraper is an MCP server for live web intelligence tools backed by `https://mcpscraper.dev`. ## Install Use the MCPB Desktop Extension for the branded Claude Desktop install, or use the npm package from any MCP client that can run stdio commands. MCP Scraper ships one stdio entrypoint plus human-facing helper CLIs: - `mcp-scraper` — the single MCP server with every tool: live web intelligence (SERP, PAA, site extraction, YouTube, Facebook ads and organic video transcripts, Maps, directory, rank tracker blueprint, credits) plus agent-controlled direct/no-proxy hosted browser sessions (screenshots, clicks, typing, scrolling, watch URLs, replay links, MP4 replay download, and saved profile/login setup for authenticated AI visibility workflows). It is context-aware: in a human terminal it prints the branded ASCII install card; in an MCP client it runs as a protocol-clean stdio server. This is the entrypoint used by the MCPB Desktop Extension. - `mcp-scraper-install` — explicit alias for the human-facing terminal installer card with the branded ASCII intro and copyable install commands. - `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, SEO workflow runs, and HTML reports. This command is safe to print because it is not an MCP stdio server. ### Terminal installer Run the command when you want the designed terminal install experience: ```bash npx -y -p mcp-scraper@latest mcp-scraper ``` In a human terminal, it prints the MCP Scraper banner, loaded tool groups, Desktop Extension download, Claude Code command, and Codex config. When launched by an MCP client, the same command detects non-interactive stdio and writes only valid JSON-RPC to stdout. The explicit installer alias still works: ```bash npx -y -p mcp-scraper@latest mcp-scraper-install ``` ### Human CLI Run setup checks and generate agent wiring: ```bash npx -y -p mcp-scraper@latest mcp-scraper-cli doctor MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude --apply npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet ``` `agent install claude --apply` upserts the Claude Code user-scope `mcp-scraper` entry to `npx -y --package mcp-scraper@latest mcp-scraper`. Fully exit Claude Code and open a new Claude terminal after applying; MCP servers are attached when Claude starts. The registered command uses the long `--package` flag deliberately. Claude Code's `mcp add` leaks short flags that appear after `--` back into its own option parsing, so a registered `-p` makes it reject its own `--scope`/`-s` argument with a misleading `unknown option` error. If the registration ever fails, the previous entry is captured beforehand and restored automatically. Check usage and upgrade concurrency from a normal terminal: ```bash MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli billing concurrency info MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli billing concurrency checkout ``` Each account has at least 1 base concurrent operation. Each `$5/month` concurrency pack adds 2 browsers on any plan. If an MCP tool hits `concurrency_limit_exceeded`, the error includes `active`, `limit`, `upgrade_url`, and the `mcp-scraper-cli billing concurrency checkout` command so the AI can explain the next step without guessing. Run workflow reports: ```bash npx -y -p mcp-scraper@latest mcp-scraper-cli workflow list npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run agent-packet --keyword "roof repair Denver" --domain example.com npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run local-competitive-audit --query roofers --state TN --min-pop 100000 --per-city 20 --hydrate-top 5 --reviews 50 npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run map-comparison --query roofers --location "Denver, CO" --per-city 20 --hydrate-top 5 npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run serp-comparison --keyword "roof repair Denver" --domain example.com --extract-top 5 npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run paa-expansion-brief --keyword "roof repair cost" --max-questions 80 npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run ai-overview-language --keyword "best roof repair company" --domain example.com npx -y -p mcp-scraper@latest mcp-scraper-cli report open last ``` Workflow runs save `manifest.json`, `report.html`, CSVs, Markdown, and evidence JSON under `MCP_SCRAPER_OUTPUT_DIR/workflows` or `~/Downloads/mcp-scraper/workflows`. High-level workflow IDs are `directory`, `agent-packet`, `local-competitive-audit`, `map-comparison`, `serp-comparison`, `paa-expansion-brief`, and `ai-overview-language`. Run hosted workflow schedules and hosted run history: ```bash npx -y -p mcp-scraper@latest mcp-scraper-cli schedule create local-competitive-audit --weekly --query roofers --state TN --min-pop 100000 --per-city 20 --webhook https://example.com/mcp-scraper-hook npx -y -p mcp-scraper@latest mcp-scraper-cli schedule list npx -y -p mcp-scraper@latest mcp-scraper-cli schedule run npx -y -p mcp-scraper@latest mcp-scraper-cli schedule pause npx -y -p mcp-scraper@latest mcp-scraper-cli schedule resume npx -y -p mcp-scraper@latest mcp-scraper-cli runs list npx -y -p mcp-scraper@latest mcp-scraper-cli runs status npx -y -p mcp-scraper@latest mcp-scraper-cli runs download ``` The hosted workflow API is mounted under `/workflows`: `GET /definitions`, `POST /run`, `GET /runs`, `GET /runs/:id`, `GET /runs/:id/artifacts/:artifactId`, `POST /schedules`, `GET /schedules`, `PATCH /schedules/:id`, `DELETE /schedules/:id`, and `POST /schedules/:id/run`. Scheduled dispatch is handled by `/workflows/cron/dispatch` and the main `/cron/tick` route when `CRON_SECRET` is configured. Current hosted artifacts are filesystem-backed for immediate run retrieval; durable public rollout should add S3/R2-compatible object storage. ## Build or enrich lead lists There are two neighboring lead journeys: - To **build a new roofing list** from a niche and city, use `workflow_run` with `workflowId: "get-leads"`, `query: "roofers"`, and a separate `location` such as `"Fort Collins, CO"`. - To **enrich an existing list** of named businesses or people, use `lead_list_enrich`. It preserves the supplied columns and appends public website, phone, email, social, contact-form, identity, provenance, and warning fields. For business rows, its optional `peopleDiscovery` mode can also find role-bearing owners or leadership and company-matched LinkedIn profile candidates. The shortest input path depends on what the current client can actually send: | Client journey | Input path | |---|---| | Codex or Claude Code with pasted rows | Normalize the pasted table, then call `lead_list_enrich` directly with `source.kind: "rows"`. | | Codex or Claude Code with a local CSV/XLSX file | Call `lead_list_upload_start`, PUT the local bytes to its signed URL, then call `lead_list_import`, `lead_list_enrich`, and `lead_list_enrich_status`. The MCP never reads the local path. | | ChatGPT or Claude app that exposes attachment contents | Express the attachment as mapped rows for direct enrichment, or bounded CSV text for `lead_list_import`. | | ChatGPT or Claude app without attachment-byte forwarding | Call `lead_list_upload_start` and give the user its upload-page link; after upload, import by `uploadId`, enrich, and poll status. | For a few pasted rows, start directly: ```json { "source": { "kind": "rows", "rows": [ { "Business": "Example Roofing", "City": "Fort Collins", "State": "CO" } ] }, "columnMap": { "name": "Business", "city": "City", "region": "State" }, "defaultEntityType": "business", "resolveMissingUrls": true, "peopleDiscovery": "owners", "peopleQueryTemplates": [ "{business} owner of company {city} {region}", "\"{business}\" founder {city} {region}" ], "maxPeoplePerLead": 2, "outputFormats": ["csv", "xlsx"], "idempotencyKey": "roofers-fort-collins-001" } ``` For CSV text, inspect it before any paid web work: ```json { "source": { "kind": "csv_text", "csvText": "Name,City,State\nJane Agent,Denver,CO" }, "idempotencyKey": "realtors-import-001" } ``` Review the returned headers, samples, and suggested map. Then call `lead_list_enrich` with `source: { "kind": "lead_list", "leadListId": "lst_..." }` and an explicit `columnMap`. Poll `lead_list_enrich_status` with the returned `jobId` until it returns a terminal status, then use its owner-scoped CSV and XLSX resource links. Reuse an idempotency key only when retrying the same semantic request; use a new key for a new upload, import, or enrichment job. Uploads accept CSV, TSV, and XLSX up to 25 MiB. Inline rows and CSV text are bounded to 500 KiB; enrichment accepts at most 250 rows. Incomplete uploads expire after 24 hours, while imported lists and result files are retained seven days. The generated XLSX contains `Enriched Leads` and `Run Summary`; input styles, macros, charts, comments, and formulas are not preserved or executed. Every discovered contact includes source provenance. Email scope is explicit (`person`, `team`, `office`, or `unknown`), so a brokerage inbox is not represented as a realtor's direct address. Public occurrence is not deliverability verification, consent, or permission to contact someone. Associated- people discovery defaults off. When enabled, the calling model can supply up to three ordered query templates with one role intent each; every role query reads AI Overview plus organic evidence, and the worker stops once the requested people bound is filled. It performs at most one LinkedIn result search per retained person and does not open LinkedIn profiles. Role evidence, AI citations, and profile-match evidence remain separate, and indexed public occurrences are not treated as verified current employment. ### Claude Desktop MCPB Build the branded one-click bundle: ```bash npm run build:mcpb ``` The generated bundle is written to `build/mcpb/mcp-scraper-.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.66.6`, SHA-256 `828f1685c159314275c2ffd1ac437cc466801e1b0d4f43f09a65073581cb7db1`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, API-key configuration field, and manually curated current-release message from the bundle manifest. The MCPB install exposes every tool — web-intelligence plus all `browser_*` tools — through the one `mcp-scraper` server. ### Raw stdio config Claude Desktop: ```json { "mcpServers": { "mcp-scraper": { "command": "npx", "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper"], "env": { "MCP_SCRAPER_API_KEY": "sk_live_your_key", "BROWSER_AGENT_PROFILE_NAME": "work-accounts" } } } } ``` `npx -y mcp-scraper@latest` runs the same single server (the package's default bin), so existing configs get every tool, including browser tools, after they pull the latest package. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process. For customer AI visibility setup, use the hosted profile connection flow: run `browser_profile_connect`, give the user the returned `watch_url`, have them complete login, then poll `browser_profile_list` until the connection is authenticated. One profile can hold many logins — call `browser_profile_connect` again with the same profile and a different domain to stack accounts. Browser hosted sessions are direct/no-proxy by default. They do not inherit the configured SERP/Maps proxy. Use proxies for SERP/Maps tools when those tools need location or anti-bot handling, not for the interactive login browser. `BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions. Claude Code one-command setup: ```bash MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude --apply ``` Then fully exit Claude Code, open a new Claude terminal, and verify: ```bash claude mcp list ``` Manual Claude Code command: ```bash claude mcp add mcp-scraper --scope user --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y -p mcp-scraper@latest mcp-scraper ``` Codex config: ```toml [mcp_servers.mcp-scraper] command = "npx" args = ["-y", "-p", "mcp-scraper@latest", "mcp-scraper"] env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" } ``` ## Tools ### Web-intelligence tools - `harvest_paa` - `search_serp` - `extract_url` — extract normal or Wayback-replayed page copy; Wayback results omit playback chrome and can include a timestamp-matched featured image. Set `preserveMedia:true` to union static and rendered/lazy media, collapse responsive variants, attach up to `maxInlineImages` AI-readable images, and receive an owner-scoped ZIP manifest readable with `archive_read`. Branding output ranks the site logo separately from evidence-bounded proof images such as certifications, awards, memberships, partner/customer marks, and press mentions. - `map_site_urls` - `map_wayback_snapshots` — count and inventory Wayback captures across an inclusive date range without downloading page bodies. Supports exact pages, prefixes, hosts, domains, or selected URLs; reports exact versus lower-bound counts, unique URLs/content digests, monthly coverage, missing months, and optional timestamp rows. - `extract_site` — crawl a live site, batch one archived site snapshot from a Wayback replay URL, or pass a `wayback` plan for whole-site, single-page, or selected-page timelines across explicit months or a `from`/`to` range. Timeline ZIPs include month folders and a capture matrix. - `archive_read` — list files in a bounded public ZIP, read one selected UTF-8 text file in byte windows, or preserve the complete selected file in the tenant Library vault. Use it with the `bundleUrl` returned by `check_site_export`. - `youtube_harvest` - `youtube_transcribe` - `facebook_ad_search` - `facebook_page_intel` - `facebook_ad_transcribe` — transcribe a direct Facebook ad video URL returned by `facebook_page_intel`. - `facebook_video_transcribe` — transcribe an organic Facebook reel, video, watch, post, or share URL, including `fb.watch` links. The tool renders the page, extracts the best matching public Facebook CDN MP4 URL, then returns transcript text, timestamped chunks, selected quality, video metadata, and the extracted MP4 URL for follow-up download. - `instagram_profile_content` — discover Instagram profile grid content links for a handle or profile URL, optionally through a saved hosted browser `profile` for authenticated access. Returns collected post/reel/tv URLs, profile counts, type counts, shortcodes, browser details, pagination attempts, stop reason, and limitations. - `instagram_media_download` — extract and download one Instagram post/reel/tv URL, optionally through a saved hosted browser `profile` for authenticated access. Returns text/caption, image URL/downloads, selected video/audio MP4 tracks, optional muxed MP4 when `ffmpeg` is available, optional transcript, and browser details. - `maps_search` — search Google's localized local-results list for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. It opens the rendered business card, reads the profile dialog, then closes it before continuing to the next ranked card. Set `includeServices: true` to return services and areas served without collecting review cards. `maxResults` defaults to 10 and is capped at 50. - `maps_place_intel` — hydrate one known/named Google Maps business with profile details, entity IDs/CID, services, service areas, review aggregates/cards, and optional photos. Set `includeImages:true`, choose `imageScope:"owner"` or `"all"`, and tune `maxImages`; results include ownership evidence, completion state, bounded AI image blocks, and an owner-scoped ZIP manifest readable with `archive_read`. - `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus localized Google business searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." Supply the business category, state, and market limits; MCP Scraper manages search transport and retry behavior internally. The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `review_count`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups. - `workflow_list` — list higher-level workflow IDs plus AI-facing recipes for market analysis, ICP research, forum/review acquisition, brand design briefings, CRO audits, positioning briefs, content gaps, and AI search visibility audits. - `workflow_suggest` — route a high-level business goal to the right workflow/tool chain before spending credits. - `workflow_run` — run hosted workflows such as `agent-packet`, `local-competitive-audit`, `map-comparison`, `serp-comparison`, `paa-expansion-brief`, and `ai-overview-language`; returns run metadata, summary, and artifact IDs. - `workflow_status` — reopen a workflow run and list its current status and artifacts. - `workflow_artifact_read` — pull generated workflow artifacts such as `evidence.json`, CSVs, Markdown briefs, and reports back into MCP context. - `editorial_reading_room_guide` — load the reusable editorial workflow, 100-article content contract, image/provenance rules, or compact example before turning dense supplied material into a reading surface. - `create_editorial_reading_room` — render up to 100 fully authored, source-grounded articles into one self-contained mobile-first HTML reading room with contents, navigation, search, jump links, progress, text sizing, evening mode, structured card/hero images, Markdown body images, collection/article Open Graph images, and visible provenance. Static files emit collection social metadata and update article metadata in-browser; crawler-perfect article unfurls require a host that renders article-specific head tags. Hosted clients receive a private seven-day artifact; local stdio clients receive an openable file under the MCP Scraper output directory. - `renew_editorial_reading_room_download` — issue a fresh signed URL for an unexpired private reading-room artifact. - `commons_prepare_publication` / `commons_validate_publication` — check a subscriber-chosen publication subdomain and validate a claim or complete source-grounded editorial edition without writing. - `commons_claim_publication` — idempotently claim one globally unique `https://{name}.transparent-commons.cc` publication for the authenticated subscriber. - `commons_publish_editorial` — publish a validated editorial reading-room edition to the caller-owned publication and return its permanent root, archive, and edition URLs. - `commons_get_publication` — recover the caller-owned or public publication metadata and current edition revisions for later edits and sharing. - `report_artifact_read` — read owner-scoped text and JSONL artifacts through the authenticated MCP connection when a model sandbox cannot open the optional signed download URL. Continue with `nextOffset` until it is null; ZIP archives use `archive_read`. - `rank_tracker_workflow` — generate a database schema, cron/heartbeat plan, ingestion workflow, metrics list, and implementation prompt for building rank trackers. It has modes for Maps rankings via `directory_workflow`/`maps_search`, organic rankings via `search_serp`, AI Overview citation tracking, and PAA source presence tracking. This planning tool does not spend credits. - `credits_info` ### Connected-account tools - `list_service_connections` — list this caller's tenant-owned Nango OAuth and official remote MCP connections, including verified provider-side account email/name when exposed, exact live reads, gated actions, permanently blocked administrative tools, credential transport, and schema-discovery metadata. Provider identity is distinct from the MCP Scraper login, and connections are never shared between customers. - `describe_service_connection_tool` — fetch the sanitized live MCP Tool definition for one tool listed on one tenant-owned connection, including its current callability, input schema, optional output schema, safe annotations, and schema hash. Use this before constructing provider-native arguments; provider functions stay behind the generic bridges instead of becoming dozens of permanent top-level tools. - `export_connected_service_data` — fetch a fresh Gmail, Google Calendar, Google Search Console, Zoom, Slack, Resend, or Meta dataset in one MCP call. Slack's `slack_channel_messages` dataset accepts a `channelId`, paginates top-level history, fetches threaded replies in bounded parallel batches, preserves file metadata, honors retry delays, and supports `allTime:true`; it never joins or changes the channel. Zoom's `zoom_transcripts` dataset resolves VTT files from recording metadata and downloads them through the authenticated connection without looping the separately rate-limited `get-meeting-transcript` function. Search Console's `search_console_performance` dataset walks accessible properties and bounded live Search Analytics pages with continuation. Small exports return inline; larger exports become private JSONL retained for seven days with exact `report_artifact_read` arguments plus an optional 15-minute human download URL. For relationship work, gather source evidence first: inspect existing People records, resolve the exact provider account, preserve provider provenance when writing a linked Communication, and never treat an export as permission to mutate the source account. - `export_search_console_table_data` — filter up to 50,000 Search Console rows already persisted by a scheduled `connection_sync` and create a private renewable JSONL artifact without calling Google again. Get the typed `gsc_performance_*` table name from `list_service_connections`, inspect it with `table-describe`, and use the same filters with `table-query` for interactive analysis. - `renew_connected_data_download` — issue a fresh 15-minute signed URL for an unexpired private export artifact without pulling the provider again. - `read_service_connection` — run one small live read by exact allowlisted name across Nango OAuth or official remote MCP connections, including bounded Google Drive inventory, change, Doc, Sheet, and text-file tools. Do not loop it over a time range when `export_connected_service_data` supports that provider's collection. - `import_service_connection_to_memory` — run one exact approved read and upsert the bounded, redacted result into an existing ordinary Memory vault at a stable server-generated path. The snapshot is embedded for semantic search and explicitly marked as untrusted provider data. This is not account-wide pagination or continuous sync. - `call_service_connection_action` — run one exact mutation only after the account action switch is enabled. Google Drive, Search Console, Resend, GitHub, Gmail, Calendar, Slack, Zoom, Facebook, Google Analytics, X, and YouTube actions remain live-inventory, provider-policy, tenant, and audit gated; credential, OAuth-grant, and raw editor-session administration is never exposed. - `set_scheduled_action_connections` — bind exact tenant connections and exact tool grants to one scheduled action. Deterministic connection syncs are model-free and one schedule occurrence counts as one metered event even when provider pagination spans many pages. Connecting or reconnecting a provider and calling direct connected-account reads, gated writes, supported exports, or bounded Memory imports do not consume additional Credits. Scheduled occurrences cost 75 Credits each. Agent-mode runs also add 1.5× the model provider's actual reported cost; deterministic connection syncs pay only the 75-Credit run base. Separately metered scraper, browser, and media operations keep their normal rate-card charges. Scheduling has no separate subscription or monthly run quota. GitHub uses the same provider-neutral read/action bridges: 27 live reads cover repository discovery, issues, pull requests, commits, releases, and workflows; 20 live mutations remain account- and schedule-gated. The 47 Nango functions and expanded reconnect scopes are deployed. The current test token remains `public_repo` until owners approve its two pending organization requests, so private-repository and workflow parity is not yet claimed. Google Drive Phase 1 exposes ten bounded reads plus the separately gated `create-drive-folder` and `copy-drive-file` actions through those same bridges and through per-run Mastra toolsets. It is interactive tool support, not the later deterministic bulk-export/`connection_sync` phase; large binary delivery and broader Drive mutations remain unreleased. Google Search Console exposes eight bounded reads and eight gated property and sitemap mutations, including API-only batches for URL inspection, Search Analytics comparison, property management, and sitemap management. Batch calls return per-item receipts, run quota-consciously, and do not require database persistence; destructive batches default to dry-run previews and require explicit confirmation. Its optional deterministic `connection_sync` path backfills Search Analytics in bounded pages, persists raw records and checkpoints, mirrors normalized records into the selected Memory vault, and upserts performance rows into one indexed tenant-owned table per connection. The table supports exact/range/substring/in-list filtering by site, date, query, page, country, device, clicks, impressions, CTR, and position. Scheduled agent runs can instead receive only the exact Search Console tools granted to that schedule. ### Memory discovery and safe editing - Start unknown-location recall with `memory-search`. It uses Hybrid Smart RAG and returns ranked excerpts, not complete notes. - Read strong candidates with `memory-get` before relying on, summarizing, linking, or editing them. - Use `memory-list` for an exhaustive inventory. Pass `allVaults:true` for every note and folder across the whole entitled Memory account; omit it for one logical vault. Account-wide results include aggregate totals and complete per-vault note/folder groups, so a default-vault count is never mistaken for the account total. - Use `list-memory-tags` for the complete account-wide canonical tag vocabulary, aliases, usage counts, and per-vault distribution. - `memory-put` replaces the complete note body. For edits, read the note, merge the requested change into the full body, write with `baseRevision`, then read back and verify it. ### Browser-agent tools - `serp_identity_create` — create an account-scoped Google search identity that binds one saved browser profile to one reusable static ISP exit, opens `google.com`, and returns a takeover URL so the user can clear consent or CAPTCHA before automation reuses it. - `serp_identity_list` — list the caller's named search identities and their readiness and last-used state without exposing provider resource IDs. - `serp_identity_delete` — delete a named search identity and its hosted profile and static ISP proxy. - `browser_open` — open a direct/no-proxy hosted browser session, or pass `serp_identity` to load that identity's saved profile and fixed exit. Returns a human `watch_url`. - `browser_profile_connect` — save a site login to a profile and return a user takeover `watch_url` on mcpscraper.dev. The user signs in through that URL, then `browser_profile_list` verifies it is authenticated. One profile holds many logins — call again with the same profile and a different domain to stack accounts. Use this again whenever a login disconnects. - `browser_profile_list` — list every login saved in a profile with its auth status, account, and note. Poll it after `browser_profile_connect` until a login reads AUTHENTICATED, then launch with `browser_open`. - `browser_screenshot` — capture a screenshot plus visible text and clickable element center coordinates and DOM bounds. - `browser_read` — read the current page text and elements with center coordinates and DOM bounds, without an image. - `browser_locate` — locate exact visible DOM elements or text ranges and return screenshot-pixel bounds. - `browser_goto` - `browser_click` - `browser_type` - `browser_scroll` - `browser_press` - `browser_replay_start` — start an MP4 replay. Returns `replay_id`, `view_url`, and `download_url` when available. - `browser_replay_stop` — stop a replay. Returns the final `view_url` and `download_url`. - `browser_list_replays` — list replay videos for a session. - `browser_replay_download` — download and save the replay MP4 under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. - `browser_replay_mark` — while recording, locate a DOM target and return a replay-timed annotation object. - `browser_replay_annotate` — download a replay MP4, render timed boxes, circles, underlines, arrows, and labels using annotation objects from `browser_replay_mark` or exact bounds from `browser_locate`, and save a new annotated MP4 under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. - `browser_capture_fanout` — capture ChatGPT/Claude AI-search fan-out from an open logged-in hosted session. Every client receives the complete structured capture inline. Installed stdio/MCPB clients can use `export=true` for durable `fanout.json`, query/source/citation/domain/snippet CSVs, TSV, and `report.html` under `MCP_SCRAPER_OUTPUT_DIR/fanout`; hosted OAuth clients receive `exports: null` and use the inline data. - `browser_close` - `browser_list_sessions` For accurate annotated videos, do not guess annotation times from a script. Start the replay, navigate until each target is visible and stable, call `browser_replay_mark` for each callout, then stop the replay and pass the returned annotations to `browser_replay_annotate` with the returned `source_width` and `source_height`. For Google SERP and Maps tools, callers provide the query, two-letter country code (`gl`), language (`hl`), device when relevant, and an optional city or region. MCP Scraper owns transport selection, anti-bot handling, and bounded retries internally; those implementation controls and receipts are not part of the public tool contract. The `mcp-scraper` server (and the MCPB bundle, which runs it) exposes both sections through one MCP server. All MCP tools return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step, plus readable text content for compatibility. Runtime `tools/list` omits output schemas so strict clients can register the complete catalog; the generated developer manifest retains every canonical output schema for validation and typed SDK generation. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects. The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The unified server exposes 259 tools: 158 scraper, browser, workflow, billing, and connected-service tools plus 101 durable-memory tools. The scraper-side inventory includes rendered site-content similarity, governed Local Sourcebook tools, Transparent Commons entity, linkset, and publication tools, direct site-export text and image readers, the searchable News Publisher and Editorial Reading Room template workflow, and 24 X-Ray tools for pixel administration, reporting, attribution journeys, campaign links, exports, and scheduled reports. Successful evidence-compiled Local Sourcebook revisions publish automatically to their canonical `localsourcebook.com` category profile and review URLs; administrator controls handle exceptional rejection or unpublishing. Release verification compares the exact local and hosted tool-name sets, not only the count. For contract parity, stdio and MCPB memory calls invoke the matching public tool on the hosted MCP Scraper `/mcp` endpoint. The hosted aggregate runtime owns MCP Scraper-specific billing, scheduling, credential, and in-process cutover policy; its internal `/memory/mcp-call` bridge is a fallback to the standalone memory service, not the public stdio execution path. Direct `mcp-memory` OAuth and stdio clients continue to use `memory.mcpscraper.dev` and must be verified as a separate dependent release surface. ## Resources The `mcp-scraper` NPX stdio server also exposes saved reports as MCP resources: `resources/list` returns the most recent Markdown reports from your output directory as `report://` URIs, and `resources/read` returns their content — so an MCP client can pull prior research into context without re-scraping or spending credits. The hosted endpoint does not expose resources (it saves no files). ## Environment - `MCP_SCRAPER_API_KEY` is required. - `MCP_SCRAPER_BASE_URL` is optional and defaults to `https://mcpscraper.dev`. - `MCP_SCRAPER_OUTPUT_DIR` is optional and defaults to `~/Downloads/mcp-scraper`. - `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files. - `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs. - `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `mcp-scraper` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`. - `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup behavior. It persists cookies and storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`. Every web intelligence tool call made through `mcp-scraper` saves a full Markdown report to disk by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. AI fan-out captures are always returned inline; only installed stdio/MCPB clients write optional `export=true` files under `MCP_SCRAPER_OUTPUT_DIR/fanout`, returning relative paths. Hosted clients always receive `exports: null`. ## Updating Existing Installs Hosted API and website changes deploy immediately to `https://mcpscraper.dev`. Stdio MCP changes require publishing a new npm package version and restarting the MCP client. Running MCP server processes do not hot-update, and tool names/descriptions are loaded when the server process starts. Recommended config for update-friendly installs: ```bash npx -y -p mcp-scraper@latest mcp-scraper ``` This is context-aware: in a normal terminal it prints the visible installer and ASCII card; in an MCP client it runs as the silent stdio server. Use `--stdio` or `MCP_SCRAPER_FORCE_STDIO=1` if you need to force server mode from a terminal. `npx -y mcp-scraper@latest` (the package default bin) runs the same server. If a user configured `mcp-scraper@0.2.0`, installed globally with `npm install -g mcp-scraper`, or installed it as a project dependency, they will stay on that version until they update the config or reinstall: ```bash npm update -g mcp-scraper npm install mcp-scraper@latest ``` Users who do not update can keep using the tools their installed package already advertises, but they will not see newly added stdio tools, schemas, or AI-facing descriptions. For example, a client running an older package cannot call `rank_tracker_workflow`, `directory_workflow`, connected-service actions, or browser tools through stdio even if the hosted API already supports adjacent endpoints. ## Branded One-Click Installs Raw `npx` MCP server installs are command/config based. `mcp-scraper` is context-aware: terminal TTY prints onboarding text and the ASCII card; MCP-client stdio pipes stay protocol-clean. Do not print marketing text to stdout from an active MCP stdio session; stdout is reserved for JSON-RPC protocol messages. For a branded Claude Desktop install, package MCP Scraper as an MCPB Desktop Extension. The repository builds one MCPB bundle with a generated icon, `manifest.json`, one self-contained server file, and `user_config` fields for API-key setup, API URL, and output folder. The build rejects `node_modules`, source maps, oversized archives, and unexpected file growth. ```bash npm run build:mcpb ``` The bundle runs `mcp-scraper` internally, so the user installs `MCP Scraper` once and gets web-intelligence tools plus live browser tools in one MCP server. ## Development - [MCP Tool Quality Spec](docs/mcp-tool-quality-spec.md) defines the shipping bar for model-facing tool names, descriptions, schemas, structured outputs, errors, packaging, and deployment.