# url-metadata

Fetch a URL and scrape its metadata using Node.js or the browser. Optional mode to parse metadata from HTML strings or `Response` objects instead. You can also request just the fields you need for smaller responses and less extraction work, tuned for your use-case. First-class configurable security options available. Content is returned raw by design; sanitize your own output (so you control processing efficiency).

---
<div>
  👉 <i><strong>Looking for a quick hosted solution?</i> <a href="https://minifetch.com">Minifetch</a></strong> is a toolkit for web devs and AI agents built on top of this package by the same author. Get started free:
  <a href="https://www.npmjs.com/package/minifetch-api">npm install minifetch-api</a>
</div>

---

## **Features**
- **proxy mode** for routing thru unblocking services (optional)
- **headless javascript rendering** (in proxy mode)
- **screenshots** (in proxy mode)
- **parser mode** - pass in an html string or Response object (optional)
- automatic charset detection & decoding (optional)
- [x402](https://www.x402.org/) errors return payment requirements
- 📫 Actively monitored [Feedback Form](https://forms.gle/UyXmzp596ZYzWCEH8) | [Discord](https://discord.gg/BqVBeeGsc5) | [GitHub](https://github.com/laurengarcia/url-metadata)

## **Extracts:**
- redirects
- response headers
- performance metrics
- meta tags
- hreflang
- favicons
- citations, per the Google Scholar spec
- [Open Graph Protocol (og:) Tags](http://ogp.me/)
- [Twitter Card Tags](https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup)
- [JSON-LD](https://moz.com/blog/json-ld-for-beginners)
- h1-h6 tags
- img tags
- the full response body as a string of html (optional)

### Performance & Token-Efficiency
- Use `fields` option to request only what you need
- Shrinks the response size *and* skips the extra extraction work
- Use `omitEmpty: true` option to drop empty fields
- Customize for your specific use-case

### **🔒 Security** - Protects against:
- Infinite redirect loops: `maxRedirects` option defaults to 10.
- SSRF attacks via [request-filtering-agent](https://www.npmjs.com/package/request-filtering-agent) in Node.js (custom options also available)
- Memory-exhaustion attacks (gzip bombs, oversized responses): set `size` option. Pair with `timeout` to prevent slow/ connection-holding responses.
- Leaking sensitive headers (auth, cookies) if target URL redirects to a diff host.

More details below. To report a bug or request a feature please open an issue or pull request in [GitHub](https://github.com/laurengarcia/url-metadata). Please read the `Troubleshooting` section below *before* filing a bug.


## Install
Works with Node.js versions `^20.19` or `>=22.12.0`, or in the browser when bundled. Try version 5.12.0 if you need backward compatibility with Node.js `>=18.17`. Example build configs available in the [GitHub repo](https://github.com/laurengarcia/url-metadata) `/example-*` dirs: Next.js, Vite and Webpack (see `/example-typescript`).

```
npm install url-metadata --save
```

## Usage
In your project file:
```js
// Use 'import' in .mjs/ .ts files or if your package.json
// has "type": "module", otherwise use 'require'.
// This package supports both:
import urlMetadata from 'url-metadata';
// const urlMetadata = require('url-metadata');

async function getMetadata(url) {
  try {
    return await urlMetadata(url); // pass options as 2nd argument
  } catch (err) {
    console.error(err);
  }
}

const metadata = await getMetadata('https://en.wikipedia.org/wiki/WHATWG');
```

### Options & Defaults
To override the default options, pass in a second options argument. The default options are the values below.
```javascript
const options = {

  // Customize the default fetch request headers:
  requestHeaders: {
    'User-Agent': 'url-metadata (+https://www.npmjs.com/package/url-metadata)',
    From: 'example@example.com'
  },

  // Route the fetch through a proxy/ unblocking service.
  // In proxy mode, other fetch-related options apply to the proxy
  // fetch, not the target URL. See  "Proxy Mode" section below.
  // Presence of proxyUrl alone triggers proxy mode.
  proxyUrl: undefined,
  // Optional vendor-specific query params passed thru as-is,
  // exactly as named in their docs. Get residential proxies,
  // headless javascript rendering, and screenshots.
  proxyParams: undefined,

  // Alternate use-case: pass `Response` object to be parsed
  // See example usage below.
  parseResponseObject: undefined,

  // Request only what you need. Accepts atomic field names or named groups:
  // ['network', 'meta', 'og', 'twitter']
  // Also accepts specific meta tags with the "meta:<name>" syntax, ex:
  // ['meta:description'] or ['meta:og:url']
  // See "Performance Tuning" section below.
  fields: undefined,

  // Drop empty fields (undefined, null, '', [], {}) from the returned
  // object. Shallow: only top-level fields are removed.
  omitEmpty: false,

  // (Node.js only)
  // To prevent SSRF attacks, the default option blocks fetch
  // requests to private network & reserved IP addresses via:
  // https://www.npmjs.com/package/request-filtering-agent
  // Browser security policies prevent SSRF automatically.
  requestFilteringAgentOptions: undefined,

  // (Node.js only)
  // Pass in your own custom `agent` to override the
  // built-in request filtering agent above
  // https://www.npmjs.com/package/node-fetch/v/2.7.0#custom-agent
  agent: undefined,

  // Maximum redirects in request chain, defaults to 10
  maxRedirects: 10,

  // `fetch` timeout in milliseconds, default is 10 seconds.
  // Time-bounds slow and connection-holding (Slowloris-class) responses.
  // Auto-bumped to 60 seconds in proxy mode (see "Proxy mode" below)
  // unless you explicitly pass your own `timeout`.
  timeout: 10000,

  // (Node.js only) max size of response in bytes (decompressed).
  // Aborts before limit is exceeded so oversized upstreams can't
  // exhaust process memory. Pair with `timeout` option above.
  // Default set to 0 disables max size:
  size: 0,

  // (Node.js only) compression defaults to true
  // Support gzip/deflate content encoding, set `false` to disable
  compress: true,

  // Force-rewrites favicon and img url strings returned to use https://,
  // valid for images with absolute & protocol-relative urls.
  // Relative urls pass thru untouched:
  ensureSecureImageRequest: true,

  // Charset to decode response with (ex: 'auto', 'utf-8', 'EUC-JP')
  // defaults to auto-detect in `Content-Type` header or meta tag
  // if none found, default `auto` option falls back to `utf-8`
  // override by passing in charset here (ex: 'windows-1251'):
  decode: 'auto',

  // (Browser only) `fetch` API cache setting
  cache: 'no-cache',

  // (Browser only) `fetch` API mode (ex: 'cors', 'same-origin', etc)
  mode: 'cors',

  // Number of characters to truncate description to
  descriptionLength: 750,

  // Include raw response body as string
  includeResponseBody: false
};

// Options usage:
const metadata = await urlMetadata(url, options);

// Alternately, parse a Response object instead:
try {
  // fetch the URL in your own code
  const response = await fetch('https://en.wikipedia.org/wiki/WHATWG');
  // ...do your own thing
  // then pass the `response` object to be parsed for metadata:
  const metadata = await urlMetadata(null, {
    parseResponseObject: response
  });
  console.log(metadata);
} catch (err) {
  console.error(err);
}

// Similarly, if you have a string of html you can create
// a Response object and pass the html string into it:
const html = `
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>Metadata page</title>
    <meta name="author" content="foobar">
    <meta name="keywords" content="HTML, CSS, JavaScript">
  </head>
  <body>
    <h1>Metadata page</h1>
  </body>
</html>
`;
const response = new Response(html, {
  headers: {
    'Content-Type': 'text/html'
  }
});
const metadata = await urlMetadata(null, {
  parseResponseObject: response
});
console.log(metadata);
```

### Proxy mode for blocked web pages (403 errors)
This package is vendor-neutral. Any proxy (unblocking) service works via `proxyUrl` + `proxyParams`. Two vendors are documented below, affiliate links support the author. Want another added? Ask in the [Discord support channel](https://discord.gg/BqVBeeGsc5).

**More code examples** and notes available in our [Github test suite](https://github.com/laurengarcia/url-metadata/blob/master/test/proxy.test.js).

`proxyUrl` triggers proxy mode. Proxy calls route through a third party doing its own upstream fetch, which takes longer than a direct fetch — so `options.timeout` defaults to 60 seconds instead of the usual 10, or you can customize.

`proxyParams` is a flat passthru, sent verbatim as query params exactly as named in your vendor's docs - no allowlist, no translation on our side. Some vendors authenticate via a header instead of a query param (ex: an `x-api-key` header) — for those, pass it in `requestHeaders` instead and skip `proxyParams` entirely if the vendor needs no other params.

In proxy mode, `requestHeaders` are sent to the proxy vendor, not the target URL — the vendor makes its own request to the target URL and won't include them unless it has a passthrough param for it (ex: ScraperAPI's `keep_headers: true`).

**Note: HTML-only.** Some vendors offer structured-data or merchant-specific endpoints that return JSON or CSV, not HTML — this package can't parse those and will throw `unsupported content type` if your proxy config is pointed at one. That's expected; those are a different kind of tool. Stick to each vendor's plain HTML-fetching endpoint (ex: ScraperAPI's https://api.scraperapi.com/, ScrapingAnt's https://api.scrapingant.com/v2/general).

**Note:** `redirects`, `responseHeaders` and `performance` timing reflect the proxy call, not the target server's actual response. Errors carry `responseHeaders` too, whenever a response was received (absent on network/transport errors). In proxy mode they're the vendor's headers — useful for telling a blocked or failed *target* apart from an error with the proxy vendor itself (bad api key, out of credits, rate limits).

#### 👉 ScraperAPI.com

[Grab your ScraperAPI key (affiliate link supports the author)](https://docs.scraperapi.com/getting-started/quick-start/grab-your-api-key?fp_ref=lauren37). It's a premium choice for hard-to-get pages.

ScraperAPI's [proxy params list](https://docs.scraperapi.com/control-and-optimization/supported-parameters?fp_ref=lauren37) offers headless javascript rendering via `render: true`, also supports `screenshot: true`. Activate residential and mobile IPs by setting `premium: true`. `ultra_premium: true` activates advanced bypassing mechanisms - can exceed the 60s default, so pass a higher `timeout` option explicitly in that case.

ScraperAPI [status codes and error list](https://docs.scraperapi.com/ai-parser/status-codes-and-errors?fp_ref=lauren37) is here.

```javascript
const metadata = await urlMetadata('https://hardto.get', {
  proxyUrl: 'https://api.scraperapi.com/',
  proxyParams: {
    api_key: '<YOUR_API_KEY>'
  }
});
// Find out how many credits the request cost you:
console.log(metadata.responseHeaders['sa-credit-cost']);
// Get screenshot URL when you set `proxyParams: { screenshot: true }`
console.log(metadata.responseHeaders['sa-screenshot']);
```

#### 👉 ScrapingAnt.com

[Grab your ScrapingAnt API Key (affiliate link supports the author)](https://scrapingant.com/?ref=ztg1mgz). Good choice if you are more price-sensitive.

It works the same way, with its own `proxyUrl` and `proxyParams`. Offers headless javascript rendering by default. Set proxy param `browser: false` to turn it off. You may pass your API key as proxy param `x-api-key` or send it as a header for extra security and privacy. Set `proxy_type` to `residential` for pages that are more difficult (default is `datacenter`).

ScrapingAnt [status codes and error list](https://docs.scrapingant.com/errors?ref=ztg1mgz) is here.

```javascript
const metadata = await urlMetadata('https://hardto.get', {
  proxyUrl: 'https://api.scrapingant.com/v2/general',
  proxyParams: {
    'x-api-key': '<YOUR_API_KEY>'
  }
});
```

### Performance Tuning

This package has powerful levers for optimizing the amount of processing that takes place for each target URL *and* for reducing response sizes. By default every metadata field is extracted and returned. Pass the `fields` option an array to narrow that down. The response only includes what you list, and the extractors for everything else never run.

`fields` accepts atomic field names ([any key from `lib/metadata-fields.js`](https://github.com/laurengarcia/url-metadata/blob/master/lib/metadata-fields.js)) plus these named groups:
- **`network`** — transport data only (status, headers, redirects, timing); see below, it's special
- **`meta`** — every meta tag, including page-specific ones not on the built-in list
- **`og`** — all [Open Graph](http://ogp.me/) (`og:`) tags
- **`twitter`** — all [Twitter Card](https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup) (`twitter:`) tags

Selecting a group keeps its empty fields so the shape stays predictable; add `omitEmpty: true` to drop empty fields entirely for further streamlining. An unknown token or an empty `fields: []` throws.

For a single page-specific meta tag, use the `meta:<name>` syntax — the part after `meta:` is the exact tag name, colons and all (`meta:og:url` is valid):

```js
const metadata = await urlMetadata(url, {
  fields: ['title', 'og', 'meta:dc.creator']
});
```

#### The `network` group: a fast, universal probe

`network` resolves as soon as the response **headers** arrive and **never downloads the response body**. A cheap way to check status, redirect chain, and timing without fetching the whole page.

Because it doesn't read the body, **`network` works on any URL regardless of content type** — HTML, PDFs, images, JSON, binaries where a normal fetch would otherwise throw `unsupported content type`:

```js
// Probe any URL — no body downloaded, any content type works:
const probe = await urlMetadata('https://pdfobject.com/pdf/sample.pdf', {
  fields: ['network']
});
probe.responseStatusCode; // 200
probe.responseHeaders;    // { 'content-type': 'application/pdf', ... }
probe.redirects;          // { count, chain }
probe.performance;        // { ttfbMs, redirectTimeMs, ... }
```

Since the body is never read, `performance.responseTimeMs` is `undefined` in this mode; there's no body-read to measure.

**Note:** `fields: ['network']` (only) and `proxyUrl` throw an error when used together, since no body-read happens in `network`-only mode, and the proxy executes the fetch on your target URL and returns the result in the body.

## Returns
Returns a promise resolved with a JSON object. Note that the returned `url` field will be the last hop in the request chain if there are redirects.

A basic template for the object can be found in `lib/metadata-fields.js`. Any additional meta tags found on the page are appended as new fields to the object. Extractor details live in `/lib/extract-*`.

Content is returned raw, by design, as found on the page without sanitizing it. Only you know its output context, so sanitization is your call (e.g. DOMPurify before rendering to a DOM; parameterized queries when storing to SQL). Treat all extracted values as untrusted.

The object consists of key/value pairs as strings, with exceptions:
- `redirects` is an object with `count` (number) and `chain` (array of `{ order, url, statusCode, location, ttfbMs }` objects)
- `canonical` is a string, and returns the first `<link rel="canonical">` found on the page (empty string if none). The raw href of every canonical tag found — including if there's only one, or none, or many — is also returned as `canonicalUrls`, an array of strings in the order encountered while parsing the page, so you can reason about this yourself (e.g. detect and fix multiples).
- `hreflang`, `favicons`, and `responseHeaders` is an array of objects containing key/value pairs of strings
- `jsonld` is an array of objects
- `meta` tags with multiple or duplicate tags on the page are returned as strings with comma-separated values since these tags don't typically contain commas, except:
- meta tags that begin with `citation_` (ex: `citation_author`) return with keys as strings and values that are an array of strings conforming to the [Google Scholar spec](https://scholar.google.com/intl/en/scholar/inclusion.html#indexing) which allows for multiple citation meta tags with different content values, including commas. So if the html contains:
```
<meta name="citation_author" content="Arlitsch, Kenning">
<meta name="citation_author" content="OBrien, Patrick">
```
... it will return as:
```
'citation_author': ["Arlitsch, Kenning", "OBrien, Patrick"],
```

### TypeScript

See `index.d.ts` for the full field catalog and the other exported interfaces: `Options`, `HreflangTag`, `FaviconTag`, `Heading`, `ImgTag`, `RedirectHop`, and `UrlMetadataError`.

The default return type `Result` is intentionally loose (collapses to `any`) so that existing codebases, mocks, and defensive scraping code compile untouched. If you want a documented, autocompleting shape, opt in with a cast. No runtime change needed:

```ts
import urlMetadata from 'url-metadata';

// Known fields fully typed; any extra page-specific
// meta tags fall through to `any`:
const metadata = await urlMetadata(url) as urlMetadata.KnownFields;
metadata.title;      // string
metadata.favicons;   // FaviconTag[]
metadata.redirects;  // { count: number, chain: RedirectHop[] }
metadata.foobar;     // any (arbitrary meta tag found on page; mostly returns as string)

// Strictest variant: known fields ONLY, closed shape.
// Catches typos at compile time, but arbitrary page-specific
// meta tags (incl. `citation_*`) are compile errors:
const strict = await urlMetadata(url) as urlMetadata.KnownFieldsStrict;
```

When you pass `fields` or `omitEmpty: true` options, the return type narrows to `Partial<urlMetadata.KnownFields>` automatically. Known fields become optional (any of them may be filtered out), and no cast is needed:

```ts
const partial = await urlMetadata(url, { fields: ['og', 'network'] });
partial['og:title'];        // string | undefined
partial.responseStatusCode; // number | undefined
```

### Troubleshooting

**⛔ Issue: Returns `404`, `403` errors or a CAPTCHA form.** Your fetch request may have been blocked by the target server because it suspects you are a bot or scraper. This package has a built-in proxy mode you can use for hard-to-get pages, see "Proxy Mode" section above.

**⛔ Issue: ERR_REQUIRE_ESM.** You're on a Node.js version below 20.19 (or 22.0–22.11). Upgrade Node, or use url-metadata@5.12.0.

**⛔ Issue: `No fetch implementation found`.** You're in either an older browser that doesn't have the native `fetch` API or a Node.js environment that doesn't support `node-fetch`. File a GitHub issue or try dowgrading to `url-metadata` version 5.12.0 for Node.js v18 support, or version 2.5.0 which uses the now-deprecated `request` module.

**⛔ Issue: Proxy mode throws `unsupported content type` errors.** This is expected behavior when you use a proxy url or proxy params that produce anything other than HTML. See "Proxy Mode" section above.

**⛔ Issue: Proxy mode with `render: true` param (headless browser rendering) throws `unsupported content type: text/x-component`.** Some JS-rendered sites (esp Next.js App Router sites) can return a React Server Component payload — a serialized component tree — instead of the rendered HTML page. This package can't parse that as HTML, so it throws instead of returning corrupted metadata. This is a quirk of the target site (and sometimes intermittent, tied to CDN/caching behavior), not something fixable from this package's side; retry or test without `render: true`. You may also try a different proxy.

**⛔ Issue: `DNS Lookup` errors.** The SSRF filtering agent defaults on this package prevent calls to private ip addresses, link-local addresses and reserved ip addresses. To change or disable this feature you need to pass custom `requestFilteringAgentOptions`. More info [here](https://www.npmjs.com/package/request-filtering-agent).

**⛔ Issue: `Response status code 0` or `CORS` errors.** The `fetch` request failed at either the network or protocol level. Possible causes:

- CORS errors. Try changing the mode option (ex: `cors`, `same-origin`, etc) or setting the `Access-Control-Allow-Origin` header on the server response from the url you are requesting if you have access to it.

- Trying to access an `https` resource that has invalid certificate, or trying to access an `http` resource from a page with an `https` origin.

- A browser plugin such as an ad-blocker or privacy protector.

---

👉 You may also want to try the hosted version of this package: [Minifetch](https://www.npmjs.com/package/minifetch-api).

---
📫 Get in touch: [Feedback Form](https://forms.gle/UyXmzp596ZYzWCEH8) | [Discord](https://discord.gg/BqVBeeGsc5) | [Github Issues](https://github.com/laurengarcia/url-metadata/issues)