> ## Documentation Index
> Fetch the complete documentation index at: https://docs.geekflare.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Web Scraping

> Extract full page content from any URL as Markdown, JSON, HTML, or LLM-ready text. Supports JavaScript rendering, stealth mode, proxy routing, and structured data extraction.

The Web Scraping endpoint scrapes any URL and returns clean page content in your preferred format. It handles JavaScript sites, blocks ads, rotates proxies, and can extract structured data using CSS or XPath selectors.

**Endpoint:** `POST https://api.geekflare.com/webscraping`

<Info>
  Install the official SDK: `npm install @geekflare/api-node` or `pip install
      geekflare-api`
</Info>

Now, you can connect Web Scraping with your AI Agents or LLMs to give context. Refer to our [MCP Server](/api/mcp) guide.

***

## Basic Scrape

Scrape a URL and get back LLM-ready content. Costs **2 credits**.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  import { GeekflareClient } from '@geekflare/api-node';

  const client = new GeekflareClient({ apiKey: 'YOUR_API_KEY' });
  const result = await client.webScrape({ url: 'https://example.com' });
  console.log(result);

  ```

  ```python Python SDK theme={null}
  from geekflare_api.client import GeekflareClient
  from geekflare_api.models import WebScrapeDto

  with GeekflareClient(api_key='YOUR_API_KEY') as client:
      result = client.web_scrape(WebScrapeDto(url='https://example.com'))
      print(result)
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com",
      "device": "desktop",
      "format": ["html-llm"],
      "fileOutput": false,
      "blockAds": true,
      "renderJS": true,
      "stealth": false,
      "waitTime": 0,
      "extractionMode": "default",
      "test": { "id": "abc123" }
    },
    "data": "# Example Domain\n\nThis domain is for use in illustrative examples..."
  }
  ```
</Accordion>

***

## Lite Scraping

Set `renderJS: false` to skip JavaScript rendering. Returns the raw response with no browser overhead. This is ideal for static pages, feeds, or sitemaps. Costs **1 credit**.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    renderJS: false
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', render_js=False))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "renderJS": false}'
  ```
</CodeGroup>

<Info>
  Use lite scraping whenever the target page does not rely on JavaScript to
  render its content. It is significantly faster and costs half the credits.
</Info>

***

## Output Formats

Choose one or more output formats. You can request up to 3 formats in a single call.

| Format         | Description                                 |
| -------------- | ------------------------------------------- |
| `html`         | Raw HTML                                    |
| `markdown`     | Clean Markdown                              |
| `json`         | Structured JSON                             |
| `html-llm`     | HTML stripped for LLM consumption (default) |
| `markdown-llm` | Markdown stripped for LLM consumption       |
| `text`         | Plain text                                  |
| `text-llm`     | Plain text stripped for LLM consumption     |

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    format: ['markdown', 'html', 'text']
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com',
      format=['markdown', 'html', 'text']
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "format": ["markdown", "html", "text"]}'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com",
      "format": ["markdown", "html", "text"],
      "test": { "id": "abc123" }
    },
    "data": {
      "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
      "html": "<!DOCTYPE html><html><head><title>Example Domain</title>...",
      "text": "Example Domain\n\nThis domain is for use in illustrative examples..."
    }
  }
  ```
</Accordion>

***

## File Output

Get a CDN URL instead of inline content. Useful for large pages or when you need to store the result.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    format: ['markdown'],
    fileOutput: true
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com',
      format=['markdown'],
      file_output=True
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "format": ["markdown"], "fileOutput": true}'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com",
      "format": ["markdown"],
      "fileOutput": true,
      "test": { "id": "abc123" }
    },
    "data": "https://cdn.geekflare.com/tests/webscraping/ZuyhINuAZPQQabbN.md"
  }
  ```
</Accordion>

***

## Stealth Mode

Bypass bot detection on protected pages. Slower but more reliable on heavily guarded sites.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    stealth: true
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', stealth=True))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "stealth": true}'
  ```
</CodeGroup>

***

## Wait Time

Add a delay after page load to capture lazy-loaded content or bypass bot checks.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    waitTime: 2.5
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', wait_time=2.5))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "waitTime": 2.5}'
  ```
</CodeGroup>

***

## Proxy Routing

Route the request through a specific country's IP address to bypass geo-blocks or scrape region-specific content.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    proxyCountry: 'gb'
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', proxy_country='gb'))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "proxyCountry": "gb"}'
  ```
</CodeGroup>

***

## Device Emulation

Emulate a mobile device to scrape mobile-specific content.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    device: 'mobile'
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(url='https://example.com', device='mobile'))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com", "device": "mobile"}'
  ```
</CodeGroup>

***

## Structured Extraction — CSS Schema

Extract specific fields from a page using CSS selectors. Returns structured JSON.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com/products',
    format: ['json'],
    extractionMode: 'cssSchema',
    extractionSchema: {
      name: 'Product Schema',
      baseSelector: '.product',
      fields: [
        { name: 'title', selector: 'h1.product-title', type: 'text' },
        { name: 'price', selector: '.price', type: 'text' },
        { name: 'link', selector: 'a.product-link', type: 'attr', attribute: 'href' }
      ]
    }
  });
  ```

  ```python Python SDK theme={null}
  from geekflare_api.models import WebScrapeDto, ExtractionSchemaDto, SelectorExtractionFieldDto

  result = client.web_scrape(WebScrapeDto(
      url='https://example.com/products',
      format=['json'],
      extraction_mode='cssSchema',
      extraction_schema=ExtractionSchemaDto(
          name='Product Schema',
          base_selector='.product',
          fields=[
              SelectorExtractionFieldDto(name='title', selector='h1.product-title', type='text'),
              SelectorExtractionFieldDto(name='price', selector='.price', type='text'),
          ]
      )
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com/products",
      "format": ["json"],
      "extractionMode": "cssSchema",
      "extractionSchema": {
        "name": "Product Schema",
        "baseSelector": ".product",
        "fields": [
          {"name": "title", "selector": "h1.product-title", "type": "text"},
          {"name": "price", "selector": ".price", "type": "text"}
        ]
      }
    }'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com/products",
      "format": ["json"],
      "extractionMode": "cssSchema",
      "test": { "id": "abc123" }
    },
    "data": [
      { "title": "Running Shoe Pro", "price": "$129.99", "link": "/products/running-shoe-pro" },
      { "title": "Trail Runner X", "price": "$89.99", "link": "/products/trail-runner-x" }
    ]
  }
  ```
</Accordion>

***

## Structured Extraction — XPath Schema

Use XPath expressions for more precise extraction.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com/articles',
    format: ['json'],
    extractionMode: 'xpathSchema',
    extractionSchema: {
      name: 'Article Schema',
      baseSelector: "//div[@class='article']",
      fields: [
        { name: 'title', selector: './/h1/text()', type: 'text' },
        { name: 'author', selector: ".//span[@class='author']/text()", type: 'text' }
      ]
    }
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com/articles',
      format=['json'],
      extraction_mode='xpathSchema',
      extraction_schema=ExtractionSchemaDto(
          name='Article Schema',
          base_selector="//div[@class='article']",
          fields=[
              SelectorExtractionFieldDto(name='title', selector='.//h1/text()', type='text'),
              SelectorExtractionFieldDto(name='author', selector=".//span[@class='author']/text()", type='text'),
          ]
      )
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com/articles",
      "format": ["json"],
      "extractionMode": "xpathSchema",
      "extractionSchema": {
        "name": "Article Schema",
        "baseSelector": "//div[@class='\''article'\'']",
        "fields": [
          {"name": "title", "selector": ".//h1/text()", "type": "text"}
        ]
      }
    }'
  ```
</CodeGroup>

<Accordion title="Response">
  ```json theme={null}
  {
    "timestamp": 1778737930991,
    "apiStatus": "success",
    "apiCode": 200,
    "meta": {
      "url": "https://example.com/articles",
      "format": ["json"],
      "extractionMode": "xpathSchema",
      "test": { "id": "abc123" }
    },
    "data": [
      { "title": "How to Run Faster", "author": "Jane Smith" },
      { "title": "Best Trails in 2025", "author": "John Doe" }
    ]
  }
  ```
</Accordion>

***

## Default Extraction — Static Fields

Inject static metadata fields alongside scraped content.

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com',
    format: ['json'],
    extractionMode: 'default',
    extractionSchema: {
      name: 'Quick Fields',
      fields: [
        { title: 'Category', value: 'Electronics' },
        { title: 'Country', value: 'India' }
      ]
    }
  });
  ```

  ```python Python SDK theme={null}
  from geekflare_api.models import DefaultExtractionFieldDto

  result = client.web_scrape(WebScrapeDto(
      url='https://example.com',
      format=['json'],
      extraction_mode='default',
      extraction_schema=ExtractionSchemaDto(
          name='Quick Fields',
          fields=[
              DefaultExtractionFieldDto(title='Category', value='Electronics'),
              DefaultExtractionFieldDto(title='Country', value='India'),
          ]
      )
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com",
      "format": ["json"],
      "extractionMode": "default",
      "extractionSchema": {
        "name": "Quick Fields",
        "fields": [
          {"title": "Category", "value": "Electronics"},
          {"title": "Country", "value": "India"}
        ]
      }
    }'
  ```
</CodeGroup>

***

## AI Extraction

Ask AI to answer questions, extract structured data, or analyze the scraped page — summaries, sentiment, keywords, contact info, and more. AI requests always run against Markdown content; any `format` you set is ignored when `aiPrompt` is present. Costs **+6 credits** on top of the base scraping cost (8 total for a standard scrape).

<CodeGroup>
  ```typescript Node.js SDK theme={null}
  const result = await client.webScrape({
    url: 'https://example.com/products/wireless-headphones',
    aiPrompt: {
      type: 'prompt',
      query: 'What is the return policy?'
    }
  });
  ```

  ```python Python SDK theme={null}
  result = client.web_scrape(WebScrapeDto(
      url='https://example.com/products/wireless-headphones',
      ai_prompt={'type': 'prompt', 'query': 'What is the return policy?'}
  ))
  ```

  ```bash cURL theme={null}
  curl -X POST https://api.geekflare.com/webscraping \
    -H "x-api-key: YOUR_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "url": "https://example.com/products/wireless-headphones",
      "aiPrompt": {"type": "prompt", "query": "What is the return policy?"}
    }'
  ```
</CodeGroup>

<Info>
  AI Extraction supports 8 modes — open-ended questions, custom JSON Schema
  extraction, product data, category listings, summaries, contact info,
  sentiment analysis, and keyword/entity extraction. See the full [AI
  Extraction](/api/ai-extraction-web-scraping) guide for
  request/response examples of every mode.
</Info>

***

## All Parameters

## All Parameters

| Parameter          | Type                                      | Default        | Description                                                                                                |
| ------------------ | ----------------------------------------- | -------------- | ---------------------------------------------------------------------------------------------------------- |
| `url`              | string                                    | required       | Target URL                                                                                                 |
| `device`           | `desktop` \| `mobile`                     | `desktop`      | Device to emulate                                                                                          |
| `format`           | array                                     | `["html-llm"]` | Output format(s). Up to 3. Ignored if `aiPrompt` is set.                                                   |
| `renderJS`         | boolean                                   | `true`         | Execute JavaScript. Set `false` for lite scraping (1 credit)                                               |
| `blockAds`         | boolean                                   | `true`         | Block ads during scrape                                                                                    |
| `stealth`          | boolean                                   | `false`        | Bypass bot detection                                                                                       |
| `waitTime`         | number                                    | `0`            | Seconds to wait after page load                                                                            |
| `fileOutput`       | boolean                                   | `false`        | Return CDN URL instead of inline data                                                                      |
| `proxyCountry`     | string                                    | —              | Route through country ISO code (e.g. `us`, `gb`)                                                           |
| `extractionMode`   | `default` \| `cssSchema` \| `xpathSchema` | `default`      | Extraction mode (used when `format` includes `json`)                                                       |
| `extractionSchema` | object                                    | —              | Schema for structured extraction (used when `format` includes `json`)                                      |
| `aiPrompt`         | object                                    | —              | Ask AI to extract/analyze the page. See [AI Extraction](/api/ai-extraction-web-scraping). Adds +6 credits. |

## Credits

## Credits

| Mode                                         | Credits |
| -------------------------------------------- | ------- |
| Standard scrape (`renderJS: true`)           | 2       |
| Lite scrape (`renderJS: false`)              | 1       |
| Standard scrape + AI Extraction (`aiPrompt`) | 8       |

<CardGroup cols={2}>
  <Card title="Node.js SDK" icon="npm" href="https://www.npmjs.com/package/@geekflare/api-node">
    `npm install @geekflare/api-node`
  </Card>

  <Card title="Python SDK" icon="python" href="https://pypi.org/project/geekflare-api/">
    `pip install geekflare-api`
  </Card>
</CardGroup>