> ## Documentation Index
> Fetch the complete documentation index at: https://docs.geekflare.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scraping Guide

> Extract full page content from any URL as Markdown, JSON, HTML, or LLM-ready text. Supports JavaScript rendering, stealth mode, proxy routing, and structured data extraction.

The Web Scraping endpoint scrapes any URL and returns clean page content in your preferred format. It handles JavaScript sites, blocks ads, rotates proxies, and can extract structured data using CSS or XPath selectors.

**Endpoint:** `POST https://api.geekflare.com/webscraping`

<Info>
  Install the official SDK: `npm install @geekflare/api-node` or `pip install
      geekflare-api`
</Info>

Now, you can connect Web Scraping with your AI Agents or LLMs to give context. Refer to our [MCP Server](/mcp) guide.

***

## Basic Scrape

Scrape a URL and get back LLM-ready content. Costs **1 credit**.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  import { GeekflareClient } from '@geekflare/api-node';

  const client = new GeekflareClient({ apiKey: 'YOUR_API_KEY' });
  const result = await client.webScrape({ url: 'https://example.com' });
  console.log(result);

  ```

  ```python Python SDK theme={null}
  from geekflare_api.client import GeekflareClient
  from geekflare_api.models import WebScrapeDto

  with GeekflareClient(api_key='YOUR_API_KEY') as client:
      result = client.web_scrape(WebScrapeDto(url='https://example.com'))
      print(result)
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com",
      "device": "desktop",
      "format": ["html-llm"],
      "fileOutput": false,
      "blockAds": true,
      "renderJS": true,
      "stealth": false,
      "waitTime": 0,
      "extractionMode": "default",
      "test": { "id": "abc123" }
    },
    "data": "# Example Domain\n\nThis domain is for use in illustrative examples..."
  }
  ```
</Accordion>

***

## JavaScript Rendering

By default, `renderJS` is automatic: Geekflare first fetches the page without a browser, and only falls back to full JavaScript rendering if the page actually needs it. You don't need to choose between "lite" and "standard" scraping, the API detects it for you at no extra cost.

Set `renderJS: true` to always force full JavaScript rendering (skip the auto-detect and go straight to a headless browser), or `renderJS: false` to always skip rendering, even if the page would otherwise need it.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    renderJS: true
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', render_js=True))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "renderJS": true}'
  ```
</CodeGroup>

<Info>
  Leave `renderJS` unset for the best default experience. Only set it
  explicitly when you already know a page requires (or doesn't require)
  JavaScript and want to skip the auto-detection round trip.
</Info>

***

## Output Formats

Choose one or more output formats. You can request up to 3 formats in a single call.

| Format         | Description                                 |
| -------------- | ------------------------------------------- |
| `html`         | Raw HTML                                    |
| `markdown`     | Clean Markdown                              |
| `json`         | Structured JSON                             |
| `html-llm`     | HTML stripped for LLM consumption (default) |
| `markdown-llm` | Markdown stripped for LLM consumption       |
| `text`         | Plain text                                  |
| `text-llm`     | Plain text stripped for LLM consumption     |

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    format: ['markdown', 'html', 'text']
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com',
      format=['markdown', 'html', 'text']
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "format": ["markdown", "html", "text"]}'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com",
      "format": ["markdown", "html", "text"],
      "test": { "id": "abc123" }
    },
    "data": {
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
      "html": "<!DOCTYPE html><html><head><title>Example Domain</title>...",
      "text": "Example Domain\n\nThis domain is for use in illustrative examples..."
    }
  }
  ```
</Accordion>

***

## File Output

Get a CDN URL instead of inline content. Useful for large pages or when you need to store the result.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    format: ['markdown'],
    fileOutput: true
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com',
      format=['markdown'],
      file_output=True
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "format": ["markdown"], "fileOutput": true}'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com",
      "format": ["markdown"],
      "fileOutput": true,
      "test": { "id": "abc123" }
    },
    "data": "https://cdn.geekflare.com/tests/webscraping/ZuyhINuAZPQQabbN.md"
  }
  ```
</Accordion>

***

## Stealth Mode

Bypass bot detection on protected pages. Slower but more reliable on heavily guarded sites.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    stealth: true
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', stealth=True))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "stealth": true}'
  ```
</CodeGroup>

***

## Wait Time

Add a delay after page load to capture lazy-loaded content or bypass bot checks.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    waitTime: 2.5
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', wait_time=2.5))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "waitTime": 2.5}'
  ```
</CodeGroup>

***

## Proxy Routing

Web Scraping supports three proxy modes via `proxyMode`:

| Mode    | Behavior                                                                                                           |
| ------- | ------------------------------------------------------------------------------------------------------------------ |
| `false` | Never uses a proxy, even if the request would otherwise be blocked. **(default)**                                  |
| `auto`  | Tries the request without a proxy first, and automatically retries through a proxy if the site blocks the request. |
| `true`  | Always routes the request through a proxy.                                                                         |

Combine `proxyMode: true` (or `auto`, when a proxy ends up being used) with `proxyCountry` to route through a specific country's IP address — useful for bypassing geo-blocks or scraping region-specific content.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    proxyMode: true,
    proxyCountry: 'gb'
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com',
      proxy_mode=True,
      proxy_country='gb'
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "proxyMode": true, "proxyCountry": "gb"}'
  ```
</CodeGroup>

<Info>
  Proxy modes apply to the Web Scraping endpoint only. Other endpoints that
  support proxy routing (Screenshot, Lighthouse, etc.) still use
  `proxyCountry` alone — see [Using Proxies](/using-proxies).
</Info>

***

## Device Emulation

Emulate a mobile device to scrape mobile-specific content.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    device: 'mobile'
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', device='mobile'))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "device": "mobile"}'
  ```
</CodeGroup>

***

## Structured Extraction — CSS Schema

Extract specific fields from a page using CSS selectors. Returns structured JSON.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com/products',
    format: ['json'],
    extractionMode: 'cssSchema',
    extractionSchema: {
      name: 'Product Schema',
      baseSelector: '.product',
      fields: [
        { name: 'title', selector: 'h1.product-title', type: 'text' },
        { name: 'price', selector: '.price', type: 'text' },
        { name: 'link', selector: 'a.product-link', type: 'attr', attribute: 'href' }
      ]
    }
  });
  ```

  ```python Python SDK theme={null}
  from geekflare_api.models import WebScrapeDto, ExtractionSchemaDto, SelectorExtractionFieldDto

  result = client.web_scrape(WebScrapeDto(
      url='https://example.com/products',
      format=['json'],
      extraction_mode='cssSchema',
      extraction_schema=ExtractionSchemaDto(
          name='Product Schema',
          base_selector='.product',
          fields=[
              SelectorExtractionFieldDto(name='title', selector='h1.product-title', type='text'),
              SelectorExtractionFieldDto(name='price', selector='.price', type='text'),
          ]
      )
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com/products",
      "format": ["json"],
      "extractionMode": "cssSchema",
      "extractionSchema": {
        "name": "Product Schema",
        "baseSelector": ".product",
        "fields": [
          {"name": "title", "selector": "h1.product-title", "type": "text"},
          {"name": "price", "selector": ".price", "type": "text"}
        ]
      }
    }'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com/products",
      "format": ["json"],
      "extractionMode": "cssSchema",
      "test": { "id": "abc123" }
    },
    "data": [
      { "title": "Running Shoe Pro", "price": "$129.99", "link": "/products/running-shoe-pro" },
      { "title": "Trail Runner X", "price": "$89.99", "link": "/products/trail-runner-x" }
    ]
  }
  ```
</Accordion>

***

## Structured Extraction — XPath Schema

Use XPath expressions for more precise extraction.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com/articles',
    format: ['json'],
    extractionMode: 'xpathSchema',
    extractionSchema: {
      name: 'Article Schema',
      baseSelector: "//div[@class='article']",
      fields: [
        { name: 'title', selector: './/h1/text()', type: 'text' },
        { name: 'author', selector: ".//span[@class='author']/text()", type: 'text' }
      ]
    }
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com/articles',
      format=['json'],
      extraction_mode='xpathSchema',
      extraction_schema=ExtractionSchemaDto(
          name='Article Schema',
          base_selector="//div[@class='article']",
          fields=[
              SelectorExtractionFieldDto(name='title', selector='.//h1/text()', type='text'),
              SelectorExtractionFieldDto(name='author', selector=".//span[@class='author']/text()", type='text'),
          ]
      )
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com/articles",
      "format": ["json"],
      "extractionMode": "xpathSchema",
      "extractionSchema": {
        "name": "Article Schema",
        "baseSelector": "//div[@class='\''article'\'']",
        "fields": [
          {"name": "title", "selector": ".//h1/text()", "type": "text"}
        ]
      }
    }'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com/articles",
      "format": ["json"],
      "extractionMode": "xpathSchema",
      "test": { "id": "abc123" }
    },
    "data": [
      { "title": "How to Run Faster", "author": "Jane Smith" },
      { "title": "Best Trails in 2025", "author": "John Doe" }
    ]
  }
  ```
</Accordion>

***

## Default Extraction — Static Fields

Inject static metadata fields alongside scraped content.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    format: ['json'],
    extractionMode: 'default',
    extractionSchema: {
      name: 'Quick Fields',
      fields: [
        { title: 'Category', value: 'Electronics' },
        { title: 'Country', value: 'India' }
      ]
    }
  });
  ```

  ```python Python SDK theme={null}
  from geekflare_api.models import DefaultExtractionFieldDto

  result = client.web_scrape(WebScrapeDto(
      url='https://example.com',
      format=['json'],
      extraction_mode='default',
      extraction_schema=ExtractionSchemaDto(
          name='Quick Fields',
          fields=[
              DefaultExtractionFieldDto(title='Category', value='Electronics'),
              DefaultExtractionFieldDto(title='Country', value='India'),
          ]
      )
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com",
      "format": ["json"],
      "extractionMode": "default",
      "extractionSchema": {
        "name": "Quick Fields",
        "fields": [
          {"title": "Category", "value": "Electronics"},
          {"title": "Country", "value": "India"}
        ]
      }
    }'
  ```
</CodeGroup>

***

## AI Extraction

Ask AI to answer questions, extract structured data, or analyze the scraped page — summaries, sentiment, keywords, contact info, and more. AI requests always run against Markdown content; any `format` you set is ignored when `aiPrompt` is present. Costs **+6 credits** on top of the base scraping cost (7 total).

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com/products/wireless-headphones',
    aiPrompt: {
      type: 'prompt',
      query: 'What is the return policy?'
    }
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com/products/wireless-headphones',
      ai_prompt={'type': 'prompt', 'query': 'What is the return policy?'}
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com/products/wireless-headphones",
      "aiPrompt": {"type": "prompt", "query": "What is the return policy?"}
    }'
  ```
</CodeGroup>

<Info>
  AI Extraction supports 8 modes — open-ended questions, custom JSON Schema
  extraction, product data, category listings, summaries, contact info,
  sentiment analysis, and keyword/entity extraction. See the full [AI
  Extraction](/ai-extraction-web-scraping) guide for
  request/response examples of every mode.
</Info>

***

## All Parameters

| Parameter          | Type                                      | Default        | Description                                                                                                                        |
| ------------------ | ----------------------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `url`              | string                                    | required       | Target URL                                                                                                                         |
| `device`           | `desktop` \| `mobile`                     | `desktop`      | Device to emulate                                                                                                                  |
| `format`           | array                                     | `["html-llm"]` | Output format(s). Up to 3. Ignored if `aiPrompt` is set.                                                                           |
| `renderJS`         | boolean                                   | auto           | Execute JavaScript. Left unset, Geekflare auto-detects whether the page needs it. Set `true`/`false` to force rendering on or off. |
| `blockAds`         | boolean                                   | `true`         | Block ads during scrape                                                                                                            |
| `stealth`          | boolean                                   | `false`        | Bypass bot detection                                                                                                               |
| `waitTime`         | number                                    | `0`            | Seconds to wait after page load                                                                                                    |
| `fileOutput`       | boolean                                   | `false`        | Return CDN URL instead of inline data                                                                                              |
| `proxyMode`        | `false` \| `auto` \| `true`               | `false`        | Whether to route the request through a proxy. See [Proxy Routing](#proxy-routing).                                                 |
| `proxyCountry`     | string                                    | —              | Route through country ISO code (e.g. `us`, `gb`). Used when a proxy is active.                                                     |
| `extractionMode`   | `default` \| `cssSchema` \| `xpathSchema` | `default`      | Extraction mode (used when `format` includes `json`)                                                                               |
| `extractionSchema` | object                                    | —              | Schema for structured extraction (used when `format` includes `json`)                                                              |
| `aiPrompt`         | object                                    | —              | Ask AI to extract/analyze the page. See [AI Extraction](/ai-extraction-web-scraping). Adds +6 credits.                             |

## Credits

| Mode                                      | Credits |
| ----------------------------------------- | ------- |
| Web Scraping (standard or lite)           | 1       |
| Web Scraping + AI Extraction (`aiPrompt`) | 7       |

<CardGroup cols={2}>
  <Card title="Node.js SDK" icon="npm" href="https://www.npmjs.com/package/@geekflare/api-node">
    `npm install @geekflare/api-node`
  </Card>

  <Card title="Python SDK" icon="python" href="https://pypi.org/project/geekflare-api/">
    `pip install geekflare-api`
  </Card>
</CardGroup>