> ## Documentation Index > Fetch the complete documentation index at: https://docs.geekflare.com/llms.txt > Use this file to discover all available pages before exploring further. # Web Scraping ## OpenAPI ````yaml POST /webscraping openapi: 3.1.0 info: title: Geekflare description: Official OpenAPI specification for all Geekflare endpoints. version: 1.0.0 license: name: MIT servers: - url: https://api.geekflare.com security: - x-api-key: [] paths: /webscraping: post: tags: - api-tool summary: Scrape a webpage with custom options operationId: webScrape parameters: [] requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/WebScrapeDto' responses: '200': description: Successfully scraped webpage content: application/json: schema: $ref: '#/components/schemas/WebScrapeResponseDto' '400': description: Invalid URL. content: application/json: schema: $ref: '#/components/schemas/BaseErrorResponseDto' example: timestamp: 1700000000000 apiStatus: failure apiCode: 400 message: INVALID_URL details: The URL must be a valid HTTP or HTTPS URL. '422': description: Unable to connect to the target website. content: application/json: schema: $ref: '#/components/schemas/BaseErrorResponseDto' example: timestamp: 1700000000000 apiStatus: failure apiCode: 422 message: UNABLE_TO_CONNECT details: >- The destination server could not be resolved, refused the connection, timed out, or is redirecting indefinitely. '500': description: Crawling failed. content: application/json: schema: $ref: '#/components/schemas/BaseErrorResponseDto' example: timestamp: 1700000000000 apiStatus: failure apiCode: 500 message: CRAWL_FAILED details: >- Our crawling service encountered an error while attempting to fetch data from the specified URL. components: schemas: WebScrapeDto: type: object properties: url: type: string description: Target URL example: https://example.com device: type: string description: Device type to emulate. Defaults to desktop. example: desktop enum: - desktop - mobile default: desktop blockAds: type: boolean description: Whether to block ads example: true default: true renderJS: type: boolean description: Whether to render JavaScript example: true default: true proxyCountry: type: string description: Proxy country code to route the request example: us format: type: array description: >- Format(s) of the scraped result. Comma-separated or array. Defaults to html. example: html,markdown default: - html-llm items: type: string enum: - html - markdown - json - markdown-llm - html-llm - text - text-llm fileOutput: type: boolean description: Whether to get response in file format example: false default: false stealth: type: boolean description: >- Enable stealth mode to bypass basic bot detection (removes webdriver signals, patches navigator properties) example: false default: false waitTime: type: number description: >- Seconds to wait after page load before capturing content. Helps bypass lazy-loaded content and bot checks. example: 2.5 default: 0 extractionMode: type: string description: Extraction mode (only used if format=json) example: default enum: - default - cssSchema - xpathSchema default: default extractionSchema: description: Extraction schema (optional in default mode, required in css/xpath) examples: default: summary: Default Mode Schema value: name: Quick Fields fields: - title: Category value: Electronics - title: Country value: India cssSchema: summary: CSS Schema value: name: Product Schema baseSelector: .product fields: - name: title selector: h1.product-title type: text - name: price selector: .price type: text xpathSchema: summary: XPath Schema value: name: Article Schema baseSelector: //div[@class='article'] fields: - name: title selector: //h1/text() type: text allOf: - $ref: '#/components/schemas/ExtractionSchemaDto' aiPrompt: description: >- Ask AI to extract or analyze the scraped page. Always runs against the Markdown of the page regardless of the format field. Adds +6 credits on top of the base scraping cost. oneOf: - $ref: '#/components/schemas/PromptAiPromptDto' - $ref: '#/components/schemas/SchemaAiPromptDto' - $ref: '#/components/schemas/ProductAiPromptDto' - $ref: '#/components/schemas/ListingAiPromptDto' - $ref: '#/components/schemas/SummaryAiPromptDto' - $ref: '#/components/schemas/ContactAiPromptDto' - $ref: '#/components/schemas/SentimentAiPromptDto' - $ref: '#/components/schemas/KeywordsAiPromptDto' required: - url WebScrapeResponseDto: type: object properties: timestamp: type: number description: Timestamp of the request in milliseconds example: 1783943217189 apiStatus: type: string description: API status message example: success enum: - success - failure apiCode: type: number description: API status code example: 200 meta: description: Metadata about the request allOf: - $ref: '#/components/schemas/WebScrapeMetaDto' data: description: Scraped data (URL or inline content depending on output) example: https://example.com/9bulgk075ed9m3vhua5vcrp0.html oneOf: - type: string - type: object aiResult: type: object description: >- AI extraction/analysis result. Shape depends on aiPrompt.type. Omitted when aiPrompt was not provided. required: - timestamp - apiStatus - apiCode - meta - data BaseErrorResponseDto: type: object properties: timestamp: type: number description: Timestamp of the request in milliseconds example: 1778737930991 apiStatus: type: string description: API status message example: success enum: - success - failure apiCode: type: number description: API status code example: 200 message: type: string description: Error message example: Invalid URL provided details: type: string description: Detailed error information example: The URL must be a valid HTTP or HTTPS URL required: - timestamp - apiStatus - apiCode - message ExtractionSchemaDto: type: object properties: name: type: string description: Name/Label for this extraction schema example: Product Schema baseSelector: type: string description: Base selector for scoping extraction (css/xpath only) example: .product fields: type: array description: List of fields to extract items: oneOf: - $ref: '#/components/schemas/DefaultExtractionFieldDto' - $ref: '#/components/schemas/SelectorExtractionFieldDto' required: - name - fields PromptAiPromptDto: type: object properties: type: type: string description: AI extraction mode enum: - prompt - schema - product - listing - summary - contact - sentiment - keywords example: prompt query: type: string description: Open-ended question about the page example: What is the return policy? required: - type - query SchemaAiPromptDto: type: object properties: type: type: string description: AI extraction mode enum: - prompt - schema - product - listing - summary - contact - sentiment - keywords example: prompt schema: type: object description: JSON Schema-like object describing fields to extract example: type: object properties: title: type: string price: type: number required: - type - schema ProductAiPromptDto: type: object properties: type: type: string description: AI extraction mode enum: - prompt - schema - product - listing - summary - contact - sentiment - keywords example: prompt required: - type ListingAiPromptDto: type: object properties: type: type: string description: AI extraction mode enum: - prompt - schema - product - listing - summary - contact - sentiment - keywords example: prompt itemSchema: type: object description: >- JSON Schema-like object describing each item to extract from a listing/category page example: type: object properties: name: type: string price: type: number maxItems: type: number description: Maximum number of items to extract default: 20 example: 20 required: - type - itemSchema SummaryAiPromptDto: type: object properties: type: type: string description: AI extraction mode enum: - prompt - schema - product - listing - summary - contact - sentiment - keywords example: prompt style: type: string description: Summary style enum: - paragraph - bullets - tldr default: paragraph focus: type: string description: Only summarize the parts of the content relevant to this focus area example: pricing maxLength: type: number description: Sentence count (paragraph/tldr) or bullet count (bullets) default: 5 example: 5 required: - type ContactAiPromptDto: type: object properties: type: type: string description: AI extraction mode enum: - prompt - schema - product - listing - summary - contact - sentiment - keywords example: prompt required: - type SentimentAiPromptDto: type: object properties: type: type: string description: AI extraction mode enum: - prompt - schema - product - listing - summary - contact - sentiment - keywords example: prompt aspects: description: >- Aspects to score individually (aspect-based sentiment). If omitted, only overall sentiment is returned. example: - sound quality - battery life - comfort - price type: array items: type: string required: - type KeywordsAiPromptDto: type: object properties: type: type: string description: AI extraction mode enum: - prompt - schema - product - listing - summary - contact - sentiment - keywords example: prompt maxKeywords: type: number description: Maximum number of keywords to return default: 10 example: 10 includeEntities: type: boolean description: >- Also extract named entities (organizations, people, dates, laws, locations) default: false required: - type WebScrapeMetaDto: type: object properties: url: type: string description: The target URL that was scraped example: https://example.com device: type: string description: Device type used example: desktop enum: - desktop - mobile format: type: array description: Output format(s) of the result example: - html-llm items: type: string enum: - html - markdown - json - markdown-llm - html-llm - text - text-llm fileOutput: type: boolean description: Whether to get response in file format example: false blockAds: type: boolean description: Whether ads were blocked example: true renderJS: type: boolean description: Whether JavaScript was rendered example: true stealth: type: boolean description: Whether stealth mode was enabled example: false waitTime: type: number description: >- Seconds to wait after page load before capturing content. Helps bypass lazy-loaded content and bot checks. example: 2.5 default: 0 proxyCountry: type: string description: Proxy country used, if any extractionMode: type: string description: Extraction mode (only used if format=json) example: default extractionSchema: description: Extraction schema (optional in default mode, required in css/xpath) examples: default: summary: Default Mode Schema value: name: Quick Fields fields: - title: Category value: Electronics - title: Country value: India allOf: - $ref: '#/components/schemas/ExtractionSchemaDto' test: description: Test details object allOf: - $ref: '#/components/schemas/TestMetaDto' aiPromptType: type: string description: The aiPrompt.type used for this request, if any example: prompt required: - url - device - format - fileOutput - blockAds - renderJS - stealth - waitTime - extractionMode - extractionSchema - test DefaultExtractionFieldDto: type: object properties: title: type: string description: Title/key of the extracted field example: Product Name value: type: object description: Static value to assign to this field example: Some hardcoded string required: - title - value SelectorExtractionFieldDto: type: object properties: name: type: string description: Field name in the extracted JSON example: title selector: type: string description: Selector or XPath to extract value example: h1.product-title type: type: string description: Type of data to extract example: text attribute: type: string description: If type=attr, specify attribute name example: href fields: description: Nested fields type: array items: $ref: '#/components/schemas/SelectorExtractionFieldDto' required: - name - selector - type TestMetaDto: type: object properties: id: type: string description: Unique test identifier example: mxqx9v9y0742lap6altwdteqd28t23nq required: - id securitySchemes: x-api-key: type: apiKey in: header name: x-api-key description: API Key required for all endpoints ````