POST /api/scrape
The main endpoint for scraping URLs. Send a URL, get back the page content with intelligent strategy escalation.
Endpoint
Request Body
Request body must be valid JSON and must not exceed 10 KB in size. The body should include the url field and optional parameters.
| Parameter | Type | Required | Description |
|---|---|---|---|
| url | string (URL) | Yes | The URL to scrape. Must be a valid HTTP or HTTPS URL. Maximum length: 2048 characters. |
Request Example
Here's a simple example of how to make a request to the /api/scrape endpoint using the x402 SDK ( For more information, see the x402 documentation).
Response Formats
Success Response (200)
Partial Response (200)
Returned when content is incomplete (login wall, geoblocked, etc.) but some content was retrieved.
Error Response (400/502)
Metadata Fields
The metadata object contains detailed information about the scraping process.
| Field | Type | Description |
|---|---|---|
| domain | string | The normalized domain (without www) that was scraped. |
| country | "us" | "eu" | null | The country code used for the request. May be flipped automatically if geoblocked. |
| attempts | number | Total number of scraping attempts made (including escalations). |
| escalations | number | Number of times the scraping strategy was escalated (static → js → premium). |
| byte_size | number | Size of the returned HTML content in bytes. |
| is_partial | boolean | Whether the returned content is partial/incomplete. true if login wall detected or content incomplete. |
| domain_confidence | number | Confidence score (0-1) for the domain strategy. Higher means more reliable strategy. |
| classification_version | string | Version of the classification algorithm used (e.g., 'v1.3'). |
| decision_reason | string | Reason for the final decision. Common values: 'ok', 'hydration_skeleton', 'antibot_challenge', 'geoblocked', 'missing_semantics', 'too_small', etc. |
| report_url | string (optional) | URL to report issues with this domain. Only present in partial responses. |
Attempts and Escalations
The attempts array shows the sequence of scraping attempts made. Each entry follows the format: mode:reason
Example attempts array:
This indicates: first attempt used static mode and detected hydration_skeleton, then escalated to js mode which succeeded (ok).
Escalation Strategy:
- static → Basic HTTP request, no JavaScript rendering
- js → JavaScript rendering enabled for SPAs and dynamic content
- premium → Premium proxies and anti-bot bypass for protected sites
The system automatically escalates based on detected issues (hydration skeletons, anti-bot challenges, etc.). The escalations count shows how many times the strategy was upgraded.
Decision Reasons
The decision_reason field explains why the scraping succeeded, failed, or required escalation.
| Reason | Description |
|---|---|
| ok | Content is complete and valid |
| hydration_skeleton | Detected React/Next.js hydration skeleton - needs JS rendering |
| antibot_challenge | Detected anti-bot protection (Cloudflare, etc.) - needs premium mode |
| geoblocked | Content blocked by geographic restrictions |
| missing_semantics | Missing title or semantic HTML elements - may need JS rendering |
| under_baseline | Content size significantly below expected baseline |
| too_small | Content too small (<700 bytes) - likely incomplete |
| empty_redirect | Empty response with redirect header |
| empty_no_content | Empty or very small response (<100 bytes) |
| empty_response_max_escalation | Empty response even after maximum escalation |
| http_403 | HTTP 403 Forbidden response |
| http_429 | HTTP 429 Too Many Requests |
| http_503 | HTTP 503 Service Unavailable |
Response Headers
| Header | Description |
|---|---|
| X-Strategy-Attempts | Number of attempts made during the scraping process. |
| Content-Type | application/json |
Pricing
Per-Request Pricing
PayPerScrape uses x402 (Coinbase's payment protocol) for per-request payments. No accounts, no API keys—just pay with your crypto wallet.
HTTP Status Codes
| Code | Status | Description |
|---|---|---|
| 400 | Bad Request | Invalid URL, malformed request body, or validation error |
| 401 | Unauthorized | Invalid x402 payment signature (if signature verification is enabled) |
| 402 | Payment Required | x402 payment needed to complete request |
| 403 | Forbidden | Domain is blocked (localhost, private networks) or restricted |
| 413 | Payload Too Large | Request body exceeds maximum size of 10 KB |
| 429 | Too Many Requests | Rate limit exceeded. Check Retry-After header for when to retry |
| 500 | Internal Server Error | Server configuration error or internal processing failure |
| 502 | Bad Gateway | Target site unreachable, incomplete content (in strict_mode), or scraping failure |
| 206 | Partial Content | Content retrieved but marked as partial/incomplete (only in non-strict mode) |
How It Works
PayPerScrape uses intelligent strategy escalation to get you the best results at the lowest cost:
- Domain Strategy Cache: Each domain's optimal scraping strategy is cached and reused.
- Automatic Escalation: If static scraping fails or detects issues (hydration skeleton, anti-bot, etc.), the system automatically escalates to JS rendering or premium mode.
- Content Evaluation: The returned HTML is evaluated for completeness. If incomplete, the system tries a more advanced strategy.
- Efficiency Optimization: The system always tries the fastest strategy first (static), only escalating when necessary to ensure complete content.
No Platform, Just an Endpoint: Unlike traditional scraping platforms, PayPerScrape is a single API endpoint. No dashboard, no account management, no complex setup. Just send a request with x402 payment and get your content.