Web Scraping - Geekflare

The Web Scraping endpoint scrapes any URL and returns clean page content in your preferred format. It handles JavaScript sites, blocks ads, rotates proxies, and can extract structured data using CSS or XPath selectors. Endpoint: POST https://api.geekflare.com/webscraping

Install the official SDK: npm install @geekflare/api-node or pip install geekflare-api

Now, you can connect Web Scraping with your AI Agents or LLMs to give context. Refer to our MCP Server guide.

Basic Scrape

Scrape a URL and get back LLM-ready content. Costs 2 credits.

import { GeekflareClient } from '@geekflare/api-node';

const client = new GeekflareClient({ apiKey: 'YOUR_API_KEY' });
const result = await client.webScrape({ url: 'https://example.com' });
console.log(result);

from geekflare_api.client import GeekflareClient
from geekflare_api.models import WebScrapeDto

with GeekflareClient(api_key='YOUR_API_KEY') as client:
    result = client.web_scrape(WebScrapeDto(url='https://example.com'))
    print(result)

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com",
    "device": "desktop",
    "format": ["html-llm"],
    "fileOutput": false,
    "blockAds": true,
    "renderJS": true,
    "stealth": false,
    "waitTime": 0,
    "extractionMode": "default",
    "test": { "id": "abc123" }
  },
  "data": "# Example Domain\n\nThis domain is for use in illustrative examples..."
}

Lite Scraping

Set renderJS: false to skip JavaScript rendering. Returns the raw response with no browser overhead. This is ideal for static pages, feeds, or sitemaps. Costs 1 credit.

const result = await client.webScrape({
  url: 'https://example.com',
  renderJS: false
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', render_js=False))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "renderJS": false}'

Use lite scraping whenever the target page does not rely on JavaScript to render its content. It is significantly faster and costs half the credits.

Output Formats

Choose one or more output formats. You can request up to 3 formats in a single call.

Format	Description
`html`	Raw HTML
`markdown`	Clean Markdown
`json`	Structured JSON
`html-llm`	HTML stripped for LLM consumption (default)
`markdown-llm`	Markdown stripped for LLM consumption
`text`	Plain text
`text-llm`	Plain text stripped for LLM consumption

const result = await client.webScrape({
  url: 'https://example.com',
  format: ['markdown', 'html', 'text']
});

result = client.web_scrape(WebScrapeDto(
    url='https://example.com',
    format=['markdown', 'html', 'text']
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": ["markdown", "html", "text"]}'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com",
    "format": ["markdown", "html", "text"],
    "test": { "id": "abc123" }
  },
  "data": {
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "html": "<!DOCTYPE html><html><head><title>Example Domain</title>...",
    "text": "Example Domain\n\nThis domain is for use in illustrative examples..."
  }
}

File Output

Get a CDN URL instead of inline content. Useful for large pages or when you need to store the result.

const result = await client.webScrape({
  url: 'https://example.com',
  format: ['markdown'],
  fileOutput: true
});

result = client.web_scrape(WebScrapeDto(
    url='https://example.com',
    format=['markdown'],
    file_output=True
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "format": ["markdown"], "fileOutput": true}'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com",
    "format": ["markdown"],
    "fileOutput": true,
    "test": { "id": "abc123" }
  },
  "data": "https://cdn.geekflare.com/tests/webscraping/ZuyhINuAZPQQabbN.md"
}

Stealth Mode

Bypass bot detection on protected pages. Slower but more reliable on heavily guarded sites.

const result = await client.webScrape({
  url: 'https://example.com',
  stealth: true
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', stealth=True))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "stealth": true}'

Wait Time

Add a delay after page load to capture lazy-loaded content or bypass bot checks.

const result = await client.webScrape({
  url: 'https://example.com',
  waitTime: 2.5
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', wait_time=2.5))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "waitTime": 2.5}'

Proxy Routing

Route the request through a specific country’s IP address to bypass geo-blocks or scrape region-specific content.

const result = await client.webScrape({
  url: 'https://example.com',
  proxyCountry: 'gb'
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', proxy_country='gb'))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "proxyCountry": "gb"}'

Device Emulation

Emulate a mobile device to scrape mobile-specific content.

const result = await client.webScrape({
  url: 'https://example.com',
  device: 'mobile'
});

result = client.web_scrape(WebScrapeDto(url='https://example.com', device='mobile'))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "device": "mobile"}'

Structured Extraction — CSS Schema

Extract specific fields from a page using CSS selectors. Returns structured JSON.

const result = await client.webScrape({
  url: 'https://example.com/products',
  format: ['json'],
  extractionMode: 'cssSchema',
  extractionSchema: {
    name: 'Product Schema',
    baseSelector: '.product',
    fields: [
      { name: 'title', selector: 'h1.product-title', type: 'text' },
      { name: 'price', selector: '.price', type: 'text' },
      { name: 'link', selector: 'a.product-link', type: 'attr', attribute: 'href' }
    ]
  }
});

from geekflare_api.models import WebScrapeDto, ExtractionSchemaDto, SelectorExtractionFieldDto

result = client.web_scrape(WebScrapeDto(
    url='https://example.com/products',
    format=['json'],
    extraction_mode='cssSchema',
    extraction_schema=ExtractionSchemaDto(
        name='Product Schema',
        base_selector='.product',
        fields=[
            SelectorExtractionFieldDto(name='title', selector='h1.product-title', type='text'),
            SelectorExtractionFieldDto(name='price', selector='.price', type='text'),
        ]
    )
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products",
    "format": ["json"],
    "extractionMode": "cssSchema",
    "extractionSchema": {
      "name": "Product Schema",
      "baseSelector": ".product",
      "fields": [
        {"name": "title", "selector": "h1.product-title", "type": "text"},
        {"name": "price", "selector": ".price", "type": "text"}
      ]
    }
  }'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com/products",
    "format": ["json"],
    "extractionMode": "cssSchema",
    "test": { "id": "abc123" }
  },
  "data": [
    { "title": "Running Shoe Pro", "price": "$129.99", "link": "/products/running-shoe-pro" },
    { "title": "Trail Runner X", "price": "$89.99", "link": "/products/trail-runner-x" }
  ]
}

Structured Extraction — XPath Schema

Use XPath expressions for more precise extraction.

const result = await client.webScrape({
  url: 'https://example.com/articles',
  format: ['json'],
  extractionMode: 'xpathSchema',
  extractionSchema: {
    name: 'Article Schema',
    baseSelector: "//div[@class='article']",
    fields: [
      { name: 'title', selector: './/h1/text()', type: 'text' },
      { name: 'author', selector: ".//span[@class='author']/text()", type: 'text' }
    ]
  }
});

result = client.web_scrape(WebScrapeDto(
    url='https://example.com/articles',
    format=['json'],
    extraction_mode='xpathSchema',
    extraction_schema=ExtractionSchemaDto(
        name='Article Schema',
        base_selector="//div[@class='article']",
        fields=[
            SelectorExtractionFieldDto(name='title', selector='.//h1/text()', type='text'),
            SelectorExtractionFieldDto(name='author', selector=".//span[@class='author']/text()", type='text'),
        ]
    )
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/articles",
    "format": ["json"],
    "extractionMode": "xpathSchema",
    "extractionSchema": {
      "name": "Article Schema",
      "baseSelector": "//div[@class='\''article'\'']",
      "fields": [
        {"name": "title", "selector": ".//h1/text()", "type": "text"}
      ]
    }
  }'

Response

{
  "timestamp": 1778737930991,
  "apiStatus": "success",
  "apiCode": 200,
  "meta": {
    "url": "https://example.com/articles",
    "format": ["json"],
    "extractionMode": "xpathSchema",
    "test": { "id": "abc123" }
  },
  "data": [
    { "title": "How to Run Faster", "author": "Jane Smith" },
    { "title": "Best Trails in 2025", "author": "John Doe" }
  ]
}

Default Extraction — Static Fields

Inject static metadata fields alongside scraped content.

const result = await client.webScrape({
  url: 'https://example.com',
  format: ['json'],
  extractionMode: 'default',
  extractionSchema: {
    name: 'Quick Fields',
    fields: [
      { title: 'Category', value: 'Electronics' },
      { title: 'Country', value: 'India' }
    ]
  }
});

from geekflare_api.models import DefaultExtractionFieldDto

result = client.web_scrape(WebScrapeDto(
    url='https://example.com',
    format=['json'],
    extraction_mode='default',
    extraction_schema=ExtractionSchemaDto(
        name='Quick Fields',
        fields=[
            DefaultExtractionFieldDto(title='Category', value='Electronics'),
            DefaultExtractionFieldDto(title='Country', value='India'),
        ]
    )
))

curl -X POST https://api.geekflare.com/webscraping \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": ["json"],
    "extractionMode": "default",
    "extractionSchema": {
      "name": "Quick Fields",
      "fields": [
        {"title": "Category", "value": "Electronics"},
        {"title": "Country", "value": "India"}
      ]
    }
  }'

All Parameters

Parameter	Type	Default	Description
`url`	string	required	Target URL
`device`	`desktop` \| `mobile`	`desktop`	Device to emulate
`format`	array	`["html-llm"]`	Output format(s). Up to 3.
`renderJS`	boolean	`true`	Execute JavaScript. Set `false` for lite scraping (1 credit)
`blockAds`	boolean	`true`	Block ads during scrape
`stealth`	boolean	`false`	Bypass bot detection
`waitTime`	number	`0`	Seconds to wait after page load
`fileOutput`	boolean	`false`	Return CDN URL instead of inline data
`proxyCountry`	string	—	Route through country ISO code (e.g. `us`, `gb`)
`extractionMode`	`default` \| `cssSchema` \| `xpathSchema`	`default`	Extraction mode (used when `format` includes `json`)
`extractionSchema`	object	—	Schema for structured extraction

Credits

Mode	Credits
Standard scrape (`renderJS: true`)	2
Lite scrape (`renderJS: false`)	1

Node.js SDK

npm install @geekflare/api-node

Python SDK

pip install geekflare-api

​Basic Scrape

​Lite Scraping

​Output Formats

​File Output

​Stealth Mode

​Wait Time

​Proxy Routing

​Device Emulation

​Structured Extraction — CSS Schema

​Structured Extraction — XPath Schema

​Default Extraction — Static Fields

​All Parameters

​Credits

Node.js SDK

Python SDK

Basic Scrape

Lite Scraping

Output Formats

File Output

Stealth Mode

Wait Time

Proxy Routing

Device Emulation

Structured Extraction — CSS Schema

Structured Extraction — XPath Schema

Default Extraction — Static Fields

All Parameters

Credits