# Web Scrapify A simple web scraper that can scrape product details from various e-commerce platforms. ## Features - [x] Scrape product details from Amazon. - [x] Scrape product details from Flipkart. - [x] Supports more agent rotations. - [x] Supports timeout for each request. - [x] Supports logging for each request. - [x] Supports search capabilities. - [x] Supports filter capabilities. - [x] Supports sort capabilities. ## Amazon Scraper ``` import { AmazonScraper, HeaderOptions, Product } from "web-scrapify"; const headers: HeaderOptions = { enableAgentRotations: true, enableLogging: true, timeout: 30000, baseUrl: 'https://www.amazon.com', headers: { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.9', 'Connection': 'keep-alive', 'Referer': 'https://www.amazon.in/', 'Sec-Fetch-Dest': 'document', 'Sec-Fetch-Mode': 'navigate', 'Sec-Fetch-Site': 'same-origin', 'Sec-Fetch-User': '?1', 'Upgrade-Insecure-Requests': '1' } } const scraper = new AmazonScraper(headers); const productURL = "Solimo-Extra-Durable-Cleaning-Eraser/dp/B08M77515R/ref=sr_1_1_ffob_sspa?_encoding=UTF8&content-id=amzn1.sym.83009b1f-702c-4be7-814b-0240b8f687d2&dib=eyJ2IjoiMSJ9.WjeStSyS-UtpCOFcUJU0C0ckHCNJY1aZA9PWNVc3qmbRJZErSZQm63vX7XbtORshGpfXkgTFSZ1k-azY0kdxLjH8aFiQWBBaFVvAi2h7YsiHOkaHFG2ubcrLhzT48y-CysqYiDvBwnVmuFziFi5IPfl0FWW9N4Pgee3ihxQ1ZrY6YeVNmVH_RG4JEBfL_oRMMs7U2OBMmx1yugmR-jfBChLADGRUtVP6tksfjCSjtXK88En7M0lgIdav1SxM6d_DYk6545ltZwcBsUu13RamO5mcAAS3AjKPdaM69OJypEc.DhGRSGmv-vBrqxQkseb5ZO0RIAu2qS1tF7NEoZsMq3s&dib_tag=se&keywords=cleaning+tools&pd_rd_r=6e139182-a89d-4e25-997f-2955f4c11ad6&pd_rd_w=wFuiM&pd_rd_wg=zVfLd&pf_rd_p=83009b1f-702c-4be7-814b-0240b8f687d2&pf_rd_r=144BC1H67M2M2X0Y99VB&qid=1735278131&sr=8-1-spons&sp_csd=d2lkZ2V0TmFtZT1zcF9hdGY&psc=1" // example 1 scraper.scrape(productURL).then((data : Product | null) => { console.log(data); }); // example 2 scraper.search({ search: 'cleaning tools', category: 'tools', page: 1 }).filter({ price: { min: 100, max: 200 } }).sort({ sort: AMAZON_SORT_OPTIONS.PRICE_LOW_TO_HIGH }) .scrapListings().then((data: Partial[] ) => { console.log(data); }); ``` ## Flipkart Scraper ``` import { FlipkartScraper, HeaderOptions, Product, FLIPKART_SORT_OPTIONS } from "web-scrapify"; const headers: HeaderOptions = { enableAgentRotations: true, // Enable agent rotations enableLogging: true, // Enable logging timeout: 30000, // Set timeout to 30 seconds baseUrl: 'https://www.flipkart.com', // Set base URL to Flipkart headers: { // Set headers 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-US,en;q=0.9', 'Connection': 'keep-alive', 'Referer': 'https://www.flipkart.com/', 'Sec-Fetch-Dest': 'document', 'Sec-Fetch-Mode': 'navigate', 'Sec-Fetch-Site': 'same-origin', 'Sec-Fetch-User': '?1', 'Upgrade-Insecure-Requests': '1' } } const scraper = new FlipkartScraper(headers); const productURL = "emotorad-x1-red-27-5-inches-single-speed-lithium-ion-li-ion-electric-cycle/p/itm830863420d539?pid=ECYH28HFFWQTZWYW&lid=LSTECYH28HFFWQTZWYWXEFGWN&marketplace=FLIPKART&store=abc%2Fulv%2Ftwp&srno=b_1_1&otracker=browse&fm=organic&iid=en_ba7G51hZyaKh1mbny_aCQiMdq62YizF5evHJOdOFh1LQDMNdtP9N36fqiox692HbthuIoCjih2bGcP9BMLYb-PUFjCTyOHoHZs-Z5_PS_w0%3D&ppt=browse&ppn=browse&ssid=kqmfdqknk00000001735280804034"; // example 1 scraper.scrape(productURL).then((data: Product | null) => { console.log(data); }); // example 2 scraper.search({ search: 'cleaning tools', page: 2 }).filter({ price: { min: 100, max: 200 } }).sort({ sort: FLIPKART_SORT_OPTIONS.PRICE_ASC }) .scrapListings().then((data: Partial[] ) => { console.log(data); }); ``` ## Header Options Here’s a table describing each parameter of the HeaderOptions interface, including its type, description, example, and whether it is required or not: | Parameter | Type | Description | Example | Required | |--------------------------|-------------------------------|-----------------------------------------------------------------------------|---------------------------------------------------------------------------------------------|----------| | enableAgentRotations | boolean | Indicates if agent rotations should be enabled for requests. | true | No | | enableLogging | boolean | Indicates if logging should be enabled for requests. | true | No | | headers | RawAxiosRequestHeaders | Custom headers to be sent with the request. | { 'User-Agent': 'Mozilla/5.0', 'Accept': 'application/json' } | No | | baseUrl | string | The base URL for the scraper. | 'https://www.amazon.com' for Amazon Scraper and 'https://www.flipkart.com' for Flipkart Scraper | No | | timeout | number | The timeout duration for requests in milliseconds. | 30000 (30 seconds) | No | This table provides a clear overview of the HeaderOptions interface parameters, making it easier to understand their purpose and usage. ## Product Interface | Property | Type | Description | |---------------|-------------------------------------------|--------------------------------------------------| | id | `string \| null` | The id of the product. | | title | `string \| null` | The title of the product. | | price | `string \| null` | The price of the product. | | imageUrl | `string \| null` | The URL of the product image. | | rating | `string \| null` | The rating of the product. | | ratingCount | `string \| null` | The number of ratings the product has received. | | details | `Record \| null` | Additional details about the product. | | description | `string \| null` | A description of the product. | | features | `string[]` | An array of features of the product. | | reviews | `Review[]` | An array of reviews for the product. | ## Review Interface | Property | Type | Description | |---------------|-------------------------------------------|--------------------------------------------------| | rating | `string \| null` | The rating given in the review. | | title | `string \| null` | The title of the review. | | author | `string \| null` | The author of the review. | | description | `string \| null` | The content of the review. | | date | `string \| null` | The date the review was written. | ## Future Features - [ ] Add retries - [ ] Add more scrapers. - [ ] Add AI capable scraper. - [ ] Add Web Search Capability (google, duckduckgo, bing etc..) - [ ] Add more documentation - [ ] Add more error handling ## Contributing & Feature Requests - If you have any feature requests or suggestions, please feel free to open an issue or submit a PR. - Contributions are welcome! Please feel free to submit a PR.