Skip to main content
ScrayleScrayle
universal scraping apisingle endpoint scrapingjavascript rendering apianti-bot scraping apicaptcha bypass api

Universal Scraping API: One Endpoint for Any Website

One API call to scrape any website with automatic JS rendering, proxy rotation, CAPTCHA solving, and structured data extraction.

Scrayle Team28-Jul-202510 min read

Not every scraping job needs a full browser session. If you just need the HTML, rendered JavaScript output, or structured data from a URL, spinning up a headless browser is overkill. It's slower, more expensive, and more complex than necessary.

Scrayle's Universal Scraping API gives you a single endpoint that handles everything: proxy rotation, CAPTCHA solving, JavaScript rendering, and anti-bot bypass. You send a URL, you get back the page content. No browser management, no WebSocket connections, no session lifecycle.

When to Use the Universal API vs. the Scraping Browser

The two products solve different problems:

  • Universal Scraping API: Best for fetching page content at scale. You don't control a browser - you send a request and get back HTML, markdown, or JSON. Ideal for data pipelines, monitoring, and high-volume extraction where you don't need interactive browser control.
  • Scraping Browser: Best when you need to interact with a page - click buttons, fill forms, scroll, handle multi-step flows. You get a real browser you control via Puppeteer, Playwright, or Selenium.

If you can describe what you need as "give me the content of this URL," use the Universal API. If you need "click this, then type that, then extract this," use the Scraping Browser.

Basic Usage

Install the SDK and make your first request:

npm install @scrayle/sdk
import { Scrayle } from '@scrayle/sdk';
const client = new Scrayle();
const result = await client.universal.scrape({
actor: 'unlocker.webUnlocker',
input: {
url: 'https://example.com/products',
js: true, // Enable JavaScript rendering
},
});
console.log(result);

The actor field tells Scrayle which rendering pipeline to use. The webUnlocker actor handles most sites automatically - it detects and bypasses anti-bot protections, solves CAPTCHAs, and renders JavaScript if you enable the js flag.

JavaScript Rendering with Instructions

Some pages need interaction before the data you want is visible - maybe a "load more" button, or content behind a tab. The jsRender actor supports browser instructions that execute before the page content is captured:

const result = await client.universal.scrape({
actor: 'unlocker.jsRender',
input: {
url: 'https://example.com/search?q=laptops',
js: true,
instructions: [
// Wait for results to load
{ type: 'waitFor', selector: '.product-grid', timeout: 10000 },
// Click "Show all" to expand results
{ type: 'click', selector: '.show-all-btn' },
// Wait for expanded content
{ type: 'wait', value: 2000 },
],
},
config: {
proxy: { country: 'US', type: 'residential' },
timeout: 30000,
},
});

Available instruction types:

  • wait - Pause for a specified number of milliseconds
  • waitFor - Wait for a CSS selector to appear on the page
  • click - Click an element matching a CSS selector
  • fill - Type text into an input field
  • evaluate - Run arbitrary JavaScript on the page

Proxy Configuration

Control where your requests originate from with the config.proxy option:

const result = await client.universal.scrape({
actor: 'unlocker.webUnlocker',
input: { url: 'https://example.co.uk/prices' },
config: {
proxy: {
country: 'GB', // Exit through UK IPs
type: 'residential', // residential, datacenter, or ipv6
},
retries: 3, // Retry failed requests up to 3 times
},
});

Scraping and Crawling with Structured Output

For more structured extraction, the SDK also provides dedicated scrape and crawl services that return content in multiple formats - markdown, HTML, JSON, links, and even screenshots:

// Single page scrape with structured output
const result = await client.scrapingCrawl.scrape.scrapeUrl(
'https://example.com/article',
{
formats: ['markdown', 'json', 'links'],
onlyMainContent: true,
timeout: 15000,
}
);
console.log(result.data?.markdown); // Clean markdown text
console.log(result.data?.json); // Structured JSON data
console.log(result.data?.links); // All links found on the page

Batch Scraping

Need to scrape multiple URLs at once? The batch endpoint processes them in parallel and returns all results together:

const urls = [
'https://example.com/product/1',
'https://example.com/product/2',
'https://example.com/product/3',
'https://example.com/product/4',
'https://example.com/product/5',
];
const result = await client.scrapingCrawl.scrape.batchScrapeUrls(
urls,
{
formats: ['markdown', 'json'],
onlyMainContent: true,
}
);
console.log(`Completed: ${result.completed} / ${result.total}`);
for (const doc of result.data || []) {
console.log(doc.metadata?.title, doc.metadata?.statusCode);
}

Batch scraping is significantly faster than making individual requests because Scrayle parallelizes the work across multiple workers. It also handles retries per-URL, so one failing page doesn't block the rest.

LLM-Powered Extraction

You can pass an extraction schema to have Scrayle use an LLM to pull structured data from the page content:

const result = await client.scrapingCrawl.scrape.scrapeUrl(
'https://example.com/product/laptop-pro',
{
formats: ['json'],
extract: {
prompt: 'Extract the product details from this page',
schema: {
type: 'object',
properties: {
name: { type: 'string' },
price: { type: 'number' },
currency: { type: 'string' },
specs: {
type: 'object',
properties: {
cpu: { type: 'string' },
ram: { type: 'string' },
storage: { type: 'string' },
},
},
inStock: { type: 'boolean' },
},
},
},
}
);
console.log(result.data?.json);
// { name: "Laptop Pro 16", price: 1299.99, currency: "USD", specs: { cpu: "M3 Pro", ... }, inStock: true }

Async Requests

For long-running scrapes, you can submit jobs asynchronously and poll for results:

// Submit the job
const job = await client.scrapingCrawl.scrape.asyncScrapeUrl(
'https://example.com/large-page',
{ formats: ['markdown', 'html'], timeout: 60000 }
);
console.log('Job submitted:', job.id);
// Poll for results
let status;
do {
await new Promise(r => setTimeout(r, 2000));
status = await client.scrapingCrawl.scrape.checkScrapeStatus(job.id);
console.log('Status:', status.status);
} while (status.status === 'scraping');
if (status.success) {
console.log(status.data);
}

Error Handling

The API returns clear error states. Always check the success field and handle failures appropriately:

try {
const result = await client.scrapingCrawl.scrape.scrapeUrl(url, {
formats: ['markdown'],
timeout: 15000,
});
if (!result.success) {
console.error('Scrape failed:', result.error);
return;
}
// Process result.data
} catch (err) {
console.error('Request error:', err.message);
}

What's Next


API reference: See the full endpoint documentation at docs.scrayle.com.

Related reading: