Universal Scraping API: One Endpoint for Any Website
One API call to scrape any website with automatic JS rendering, proxy rotation, CAPTCHA solving, and structured data extraction.
Not every scraping job needs a full browser session. If you just need the HTML, rendered JavaScript output, or structured data from a URL, spinning up a headless browser is overkill. It's slower, more expensive, and more complex than necessary.
Scrayle's Universal Scraping API gives you a single endpoint that handles everything: proxy rotation, CAPTCHA solving, JavaScript rendering, and anti-bot bypass. You send a URL, you get back the page content. No browser management, no WebSocket connections, no session lifecycle.
When to Use the Universal API vs. the Scraping Browser
The two products solve different problems:
- Universal Scraping API: Best for fetching page content at scale. You don't control a browser - you send a request and get back HTML, markdown, or JSON. Ideal for data pipelines, monitoring, and high-volume extraction where you don't need interactive browser control.
- Scraping Browser: Best when you need to interact with a page - click buttons, fill forms, scroll, handle multi-step flows. You get a real browser you control via Puppeteer, Playwright, or Selenium.
If you can describe what you need as "give me the content of this URL," use the Universal API. If you need "click this, then type that, then extract this," use the Scraping Browser.
Basic Usage
Install the SDK and make your first request:
npm install @scrayle/sdk
import { Scrayle } from '@scrayle/sdk';const client = new Scrayle();const result = await client.universal.scrape({actor: 'unlocker.webUnlocker',input: {url: 'https://example.com/products',js: true, // Enable JavaScript rendering},});console.log(result);
The actor field tells Scrayle which rendering pipeline to use. The webUnlocker actor handles most sites automatically - it detects and bypasses anti-bot protections, solves CAPTCHAs, and renders JavaScript if you enable the js flag.
JavaScript Rendering with Instructions
Some pages need interaction before the data you want is visible - maybe a "load more" button, or content behind a tab. The jsRender actor supports browser instructions that execute before the page content is captured:
const result = await client.universal.scrape({actor: 'unlocker.jsRender',input: {url: 'https://example.com/search?q=laptops',js: true,instructions: [// Wait for results to load{ type: 'waitFor', selector: '.product-grid', timeout: 10000 },// Click "Show all" to expand results{ type: 'click', selector: '.show-all-btn' },// Wait for expanded content{ type: 'wait', value: 2000 },],},config: {proxy: { country: 'US', type: 'residential' },timeout: 30000,},});
Available instruction types:
wait- Pause for a specified number of millisecondswaitFor- Wait for a CSS selector to appear on the pageclick- Click an element matching a CSS selectorfill- Type text into an input fieldevaluate- Run arbitrary JavaScript on the page
Proxy Configuration
Control where your requests originate from with the config.proxy option:
const result = await client.universal.scrape({actor: 'unlocker.webUnlocker',input: { url: 'https://example.co.uk/prices' },config: {proxy: {country: 'GB', // Exit through UK IPstype: 'residential', // residential, datacenter, or ipv6},retries: 3, // Retry failed requests up to 3 times},});
Scraping and Crawling with Structured Output
For more structured extraction, the SDK also provides dedicated scrape and crawl services that return content in multiple formats - markdown, HTML, JSON, links, and even screenshots:
// Single page scrape with structured outputconst result = await client.scrapingCrawl.scrape.scrapeUrl('https://example.com/article',{formats: ['markdown', 'json', 'links'],onlyMainContent: true,timeout: 15000,});console.log(result.data?.markdown); // Clean markdown textconsole.log(result.data?.json); // Structured JSON dataconsole.log(result.data?.links); // All links found on the page
Batch Scraping
Need to scrape multiple URLs at once? The batch endpoint processes them in parallel and returns all results together:
const urls = ['https://example.com/product/1','https://example.com/product/2','https://example.com/product/3','https://example.com/product/4','https://example.com/product/5',];const result = await client.scrapingCrawl.scrape.batchScrapeUrls(urls,{formats: ['markdown', 'json'],onlyMainContent: true,});console.log(`Completed: ${result.completed} / ${result.total}`);for (const doc of result.data || []) {console.log(doc.metadata?.title, doc.metadata?.statusCode);}
Batch scraping is significantly faster than making individual requests because Scrayle parallelizes the work across multiple workers. It also handles retries per-URL, so one failing page doesn't block the rest.
LLM-Powered Extraction
You can pass an extraction schema to have Scrayle use an LLM to pull structured data from the page content:
const result = await client.scrapingCrawl.scrape.scrapeUrl('https://example.com/product/laptop-pro',{formats: ['json'],extract: {prompt: 'Extract the product details from this page',schema: {type: 'object',properties: {name: { type: 'string' },price: { type: 'number' },currency: { type: 'string' },specs: {type: 'object',properties: {cpu: { type: 'string' },ram: { type: 'string' },storage: { type: 'string' },},},inStock: { type: 'boolean' },},},},});console.log(result.data?.json);// { name: "Laptop Pro 16", price: 1299.99, currency: "USD", specs: { cpu: "M3 Pro", ... }, inStock: true }
Async Requests
For long-running scrapes, you can submit jobs asynchronously and poll for results:
// Submit the jobconst job = await client.scrapingCrawl.scrape.asyncScrapeUrl('https://example.com/large-page',{ formats: ['markdown', 'html'], timeout: 60000 });console.log('Job submitted:', job.id);// Poll for resultslet status;do {await new Promise(r => setTimeout(r, 2000));status = await client.scrapingCrawl.scrape.checkScrapeStatus(job.id);console.log('Status:', status.status);} while (status.status === 'scraping');if (status.success) {console.log(status.data);}
Error Handling
The API returns clear error states. Always check the success field and handle failures appropriately:
try {const result = await client.scrapingCrawl.scrape.scrapeUrl(url, {formats: ['markdown'],timeout: 15000,});if (!result.success) {console.error('Scrape failed:', result.error);return;}// Process result.data} catch (err) {console.error('Request error:', err.message);}
What's Next
- Need to crawl entire sites? Read the Crawl guide for recursive site indexing
- Need interactive browser control? See the Scraping Browser guide
- Want AI to figure out what to extract? Try the AI Browser Agent
API reference: See the full endpoint documentation at docs.scrayle.com.
Related reading: