Web Scraping Guide

The Web Scraping endpoint scrapes any URL and returns clean page content in your preferred format. It handles JavaScript sites, blocks ads, rotates proxies, and can extract structured data using CSS or XPath selectors. Endpoint: POST https://api.geekflare.com/webscraping

Install the official SDK: npm install @geekflare/api-node or pip install geekflare-api

Now, you can connect Web Scraping with your AI Agents or LLMs to give context. Refer to our MCP Server guide.

Basic Scrape

Scrape a URL and get back LLM-ready content. Costs 1 credit.

import { GeekflareClient } from '@geekflare/api-node';

const client = new GeekflareClient({ apiKey: 'YOUR_API_KEY' });
const result = await client.webScrape({ url: 'https://example.com' });
console.log(result);

from geekflare_api.client import GeekflareClient
from geekflare_api.models import WebScrapeDto

with GeekflareClient(api_key='YOUR_API_KEY') as client:
    result = client.web_scrape(WebScrapeDto(url='https://example.com'))
    print(result)

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com",
    "device": "desktop",
    "format": ["html-llm"],
    "fileOutput": false,
    "blockAds": true,
    "renderJS": true,
    "stealth": false,
    "waitTime": 0,
    "extractionMode": "default",
    "test": { "id": "abc123" }
  },
  "data": "# Example Domain\n\nThis domain is for use in illustrative examples..."
}

JavaScript Rendering

By default, renderJS is automatic: Geekflare first fetches the page without a browser, and only falls back to full JavaScript rendering if the page actually needs it. You don’t need to choose between “lite” and “standard” scraping, the API detects it for you at no extra cost. Set renderJS: true to always force full JavaScript rendering (skip the auto-detect and go straight to a headless browser), or renderJS: false to always skip rendering, even if the page would otherwise need it.

const result = await client.webScrape({
  url: 'https://example.com',
  renderJS: true
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', render_js=True))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "renderJS": true}'

Leave renderJS unset for the best default experience. Only set it explicitly when you already know a page requires (or doesn’t require) JavaScript and want to skip the auto-detection round trip.

Output Formats

Choose one or more output formats. You can request up to 3 formats in a single call.

Format	Description
`html`	Raw HTML
`markdown`	Clean Markdown
`json`	Structured JSON
`html-llm`	HTML stripped for LLM consumption (default)
`markdown-llm`	Markdown stripped for LLM consumption
`text`	Plain text
`text-llm`	Plain text stripped for LLM consumption

const result = await client.webScrape({
  url: 'https://example.com',
  format: ['markdown', 'html', 'text']
});

result = client.web_scrape(WebScrapeDto(
    url='https://example.com',
    format=['markdown', 'html', 'text']
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": ["markdown", "html", "text"]}'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com",
    "format": ["markdown", "html", "text"],
    "test": { "id": "abc123" }
  },
  "data": {
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "html": "<!DOCTYPE html><html><head><title>Example Domain</title>...",
    "text": "Example Domain\n\nThis domain is for use in illustrative examples..."
  }
}

File Output

Get a CDN URL instead of inline content. Useful for large pages or when you need to store the result.

const result = await client.webScrape({
  url: 'https://example.com',
  format: ['markdown'],
  fileOutput: true
});

result = client.web_scrape(WebScrapeDto(
    url='https://example.com',
    format=['markdown'],
    file_output=True
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": ["markdown"], "fileOutput": true}'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com",
    "format": ["markdown"],
    "fileOutput": true,
    "test": { "id": "abc123" }
  },
  "data": "https://cdn.geekflare.com/tests/webscraping/ZuyhINuAZPQQabbN.md"
}

Stealth Mode

Bypass bot detection on protected pages. Slower but more reliable on heavily guarded sites.

const result = await client.webScrape({
  url: 'https://example.com',
  stealth: true
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', stealth=True))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "stealth": true}'

Wait Time

Add a delay after page load to capture lazy-loaded content or bypass bot checks.

const result = await client.webScrape({
  url: 'https://example.com',
  waitTime: 2.5
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', wait_time=2.5))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "waitTime": 2.5}'

Proxy Routing

Web Scraping supports three proxy modes via proxyMode:

Mode	Behavior
`false`	Never uses a proxy, even if the request would otherwise be blocked. (default)
`auto`	Tries the request without a proxy first, and automatically retries through a proxy if the site blocks the request.
`true`	Always routes the request through a proxy.

Combine proxyMode: true (or auto, when a proxy ends up being used) with proxyCountry to route through a specific country’s IP address — useful for bypassing geo-blocks or scraping region-specific content.

const result = await client.webScrape({
  url: 'https://example.com',
  proxyMode: true,
  proxyCountry: 'gb'
});

result = client.web_scrape(WebScrapeDto(
    url='https://example.com',
    proxy_mode=True,
    proxy_country='gb'
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "proxyMode": true, "proxyCountry": "gb"}'

Proxy modes apply to the Web Scraping endpoint only. Other endpoints that support proxy routing (Screenshot, Lighthouse, etc.) still use proxyCountry alone — see Using Proxies.

Device Emulation

Emulate a mobile device to scrape mobile-specific content.

const result = await client.webScrape({
  url: 'https://example.com',
  device: 'mobile'
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', device='mobile'))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "device": "mobile"}'

Structured Extraction — CSS Schema

Extract specific fields from a page using CSS selectors. Returns structured JSON.

const result = await client.webScrape({
  url: 'https://example.com/products',
  format: ['json'],
  extractionMode: 'cssSchema',
  extractionSchema: {
    name: 'Product Schema',
    baseSelector: '.product',
    fields: [
      { name: 'title', selector: 'h1.product-title', type: 'text' },
      { name: 'price', selector: '.price', type: 'text' },
      { name: 'link', selector: 'a.product-link', type: 'attr', attribute: 'href' }
    ]
  }
});

from geekflare_api.models import WebScrapeDto, ExtractionSchemaDto, SelectorExtractionFieldDto

result = client.web_scrape(WebScrapeDto(
    url='https://example.com/products',
    format=['json'],
    extraction_mode='cssSchema',
    extraction_schema=ExtractionSchemaDto(
        name='Product Schema',
        base_selector='.product',
        fields=[
            SelectorExtractionFieldDto(name='title', selector='h1.product-title', type='text'),
            SelectorExtractionFieldDto(name='price', selector='.price', type='text'),
        ]
    )
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "format": ["json"],
    "extractionMode": "cssSchema",
    "extractionSchema": {
      "name": "Product Schema",
      "baseSelector": ".product",
      "fields": [
        {"name": "title", "selector": "h1.product-title", "type": "text"},
        {"name": "price", "selector": ".price", "type": "text"}
      ]
    }
  }'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com/products",
    "format": ["json"],
    "extractionMode": "cssSchema",
    "test": { "id": "abc123" }
  },
  "data": [
    { "title": "Running Shoe Pro", "price": "$129.99", "link": "/products/running-shoe-pro" },
    { "title": "Trail Runner X", "price": "$89.99", "link": "/products/trail-runner-x" }
  ]
}

Structured Extraction — XPath Schema

Use XPath expressions for more precise extraction.

const result = await client.webScrape({
  url: 'https://example.com/articles',
  format: ['json'],
  extractionMode: 'xpathSchema',
  extractionSchema: {
    name: 'Article Schema',
    baseSelector: "//div[@class='article']",
    fields: [
      { name: 'title', selector: './/h1/text()', type: 'text' },
      { name: 'author', selector: ".//span[@class='author']/text()", type: 'text' }
    ]
  }
});

result = client.web_scrape(WebScrapeDto(
    url='https://example.com/articles',
    format=['json'],
    extraction_mode='xpathSchema',
    extraction_schema=ExtractionSchemaDto(
        name='Article Schema',
        base_selector="//div[@class='article']",
        fields=[
            SelectorExtractionFieldDto(name='title', selector='.//h1/text()', type='text'),
            SelectorExtractionFieldDto(name='author', selector=".//span[@class='author']/text()", type='text'),
        ]
    )
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/articles",
    "format": ["json"],
    "extractionMode": "xpathSchema",
    "extractionSchema": {
      "name": "Article Schema",
      "baseSelector": "//div[@class='\''article'\'']",
      "fields": [
        {"name": "title", "selector": ".//h1/text()", "type": "text"}
      ]
    }
  }'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com/articles",
    "format": ["json"],
    "extractionMode": "xpathSchema",
    "test": { "id": "abc123" }
  },
  "data": [
    { "title": "How to Run Faster", "author": "Jane Smith" },
    { "title": "Best Trails in 2025", "author": "John Doe" }
  ]
}

Default Extraction — Static Fields

Inject static metadata fields alongside scraped content.

const result = await client.webScrape({
  url: 'https://example.com',
  format: ['json'],
  extractionMode: 'default',
  extractionSchema: {
    name: 'Quick Fields',
    fields: [
      { title: 'Category', value: 'Electronics' },
      { title: 'Country', value: 'India' }
    ]
  }
});

from geekflare_api.models import DefaultExtractionFieldDto

result = client.web_scrape(WebScrapeDto(
    url='https://example.com',
    format=['json'],
    extraction_mode='default',
    extraction_schema=ExtractionSchemaDto(
        name='Quick Fields',
        fields=[
            DefaultExtractionFieldDto(title='Category', value='Electronics'),
            DefaultExtractionFieldDto(title='Country', value='India'),
        ]
    )
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": ["json"],
    "extractionMode": "default",
    "extractionSchema": {
      "name": "Quick Fields",
      "fields": [
        {"title": "Category", "value": "Electronics"},
        {"title": "Country", "value": "India"}
      ]
    }
  }'

AI Extraction

Ask AI to answer questions, extract structured data, or analyze the scraped page — summaries, sentiment, keywords, contact info, and more. AI requests always run against Markdown content; any format you set is ignored when aiPrompt is present. Costs +6 credits on top of the base scraping cost (7 total).

const result = await client.webScrape({
  url: 'https://example.com/products/wireless-headphones',
  aiPrompt: {
    type: 'prompt',
    query: 'What is the return policy?'
  }
});

result = client.web_scrape(WebScrapeDto(
    url='https://example.com/products/wireless-headphones',
    ai_prompt={'type': 'prompt', 'query': 'What is the return policy?'}
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/wireless-headphones",
    "aiPrompt": {"type": "prompt", "query": "What is the return policy?"}
  }'

AI Extraction supports 8 modes — open-ended questions, custom JSON Schema extraction, product data, category listings, summaries, contact info, sentiment analysis, and keyword/entity extraction. See the full AI Extraction guide for request/response examples of every mode.

All Parameters

Parameter	Type	Default	Description
`url`	string	required	Target URL
`device`	`desktop` \| `mobile`	`desktop`	Device to emulate
`format`	array	`["html-llm"]`	Output format(s). Up to 3. Ignored if `aiPrompt` is set.
`renderJS`	boolean	auto	Execute JavaScript. Left unset, Geekflare auto-detects whether the page needs it. Set `true`/`false` to force rendering on or off.
`blockAds`	boolean	`true`	Block ads during scrape
`stealth`	boolean	`false`	Bypass bot detection
`waitTime`	number	`0`	Seconds to wait after page load
`fileOutput`	boolean	`false`	Return CDN URL instead of inline data
`proxyMode`	`false` \| `auto` \| `true`	`false`	Whether to route the request through a proxy. See Proxy Routing.
`proxyCountry`	string	—	Route through country ISO code (e.g. `us`, `gb`). Used when a proxy is active.
`extractionMode`	`default` \| `cssSchema` \| `xpathSchema`	`default`	Extraction mode (used when `format` includes `json`)
`extractionSchema`	object	—	Schema for structured extraction (used when `format` includes `json`)
`aiPrompt`	object	—	Ask AI to extract/analyze the page. See AI Extraction. Adds +6 credits.

Credits

Mode	Credits
Web Scraping (standard or lite)	1
Web Scraping + AI Extraction (`aiPrompt`)	7

Node.js SDK

npm install @geekflare/api-node

Python SDK

pip install geekflare-api

Getting started

Quickstarts

Guides

MCP Server

Capabilities

Best Practices

Troubleshooting

Changelog

Web Scraping Guide

Basic Scrape

JavaScript Rendering

Output Formats

File Output

Stealth Mode

Wait Time

Proxy Routing

Device Emulation

Structured Extraction — CSS Schema

Structured Extraction — XPath Schema

Default Extraction — Static Fields

AI Extraction

All Parameters

Credits

Node.js SDK

Python SDK

​Basic Scrape

​JavaScript Rendering

​Output Formats

​File Output

​Stealth Mode

​Wait Time

​Proxy Routing

​Device Emulation

​Structured Extraction — CSS Schema

​Structured Extraction — XPath Schema

​Default Extraction — Static Fields

​AI Extraction

​All Parameters

​Credits

Node.js SDK

Python SDK

Basic Scrape

JavaScript Rendering

Output Formats

File Output

Stealth Mode

Wait Time

Proxy Routing

Device Emulation

Structured Extraction — CSS Schema

Structured Extraction — XPath Schema

Default Extraction — Static Fields

AI Extraction

All Parameters

Credits