---
name: scraping
description: Firecrawl asset collection and network merge
metadata:
  tags: firecrawl, assets, scraping
---

# Scraping with Firecrawl

## Goals

- Collect assets missed by Playwright network logs
- Capture background images, inline CSS URLs, and fonts
- Merge asset metadata into a single inventory

## Firecrawl Inputs

- URL
- Desired formats (`html`, `links`)
- Asset discovery for `img`, `link`, `style`, `svg`

## Example Client Wrapper

```ts
export type FirecrawlAsset = {
  url: string;
  contentType?: string;
};

export type FirecrawlScrapeResponse = {
  html?: string;
  links?: string[];
  assets?: FirecrawlAsset[];
};

export type FirecrawlClient = {
  scrapeUrl: (url: string, options: Record<string, unknown>) => Promise<FirecrawlScrapeResponse>;
};

export const scrapeWithFirecrawl = async (client: FirecrawlClient, url: string) => {
  const response = await client.scrapeUrl(url, {
    formats: ['html', 'links'],
    includeAssets: true,
  });

  return {
    html: response.html,
    links: response.links ?? [],
    assets: response.assets ?? [],
  };
};
```

## Asset Normalization

- Ignore data URLs unless they contain unique SVGs
- Convert asset URLs to deterministic filenames
- Group fonts by family and weight for easy mapping
- Write to `public/assets` and `public/fonts`

## Merge Strategy

1. Normalize all URLs (strip tracking params)
2. Deduplicate by canonical URL
3. Prefer Playwright metadata when both exist
4. Track `source` so debugging is easier