Skip to main content
ScrayleScrayle
cloud browser tutorialpuppeteer cloudscraping browserheadless browser apiweb scraping getting started

Get Started: Your First Scraping Browser Session

A step-by-step guide to launching your first cloud browser session, connecting with Puppeteer or Selenium, and extracting data from a dynamic page.

Scrayle Team10-Jul-20258 min read

Web scraping is harder than it used to be. JavaScript-heavy single-page applications, bot detection systems, CAPTCHAs, and fingerprinting techniques mean that simple HTTP requests no longer cut it for most targets. You need a real browser - but running headless Chrome at scale brings its own headaches: memory management, proxy rotation, crash recovery, and infrastructure costs.

Scrayle's Scraping Browser solves this by giving you on-demand cloud browser sessions that you control with the same Puppeteer or Playwright code you already know. Each session runs in an isolated container with built-in fingerprint rotation, automatic CAPTCHA solving, and residential proxy egress. You write normal browser automation code - Scrayle handles everything else.

What You'll Build

By the end of this tutorial, you will have a working Node.js script that:

  1. Connects to a Scrayle cloud browser session via WebSocket
  2. Navigates to a JavaScript-rendered page
  3. Waits for dynamic content to load
  4. Extracts structured data using CSS selectors
  5. Outputs clean JSON you can pipe to a file or database

Prerequisites

  • A Scrayle account (the free tier includes 1,000 credits)
  • Node.js 18 or later installed locally
  • Basic familiarity with JavaScript and CSS selectors

Step 1: Install the Scrayle SDK

Start by creating a new project directory and installing the SDK:

mkdir my-scraper && cd my-scraper
npm init -y
npm install @scrayle/sdk puppeteer-core

The SDK provides the Scrayle client that manages authentication and session lifecycle. We use puppeteer-core (not the full puppeteer package) because Scrayle provides the browser - you only need the client protocol library.

Step 2: Get Your API Key

Log into the Scrayle dashboard and navigate to Settings > API Keys. Create a new key and copy it. Store it as an environment variable:

# .env
SCRAYLE_API_KEY=sk_live_your_key_here

Never commit API keys to version control. The SDK automatically reads from theSCRAYLE_API_KEY environment variable if you don't pass one explicitly.

Step 3: Launch a Browser Session

Create a file called scrape.js with the following code:

import { Scrayle } from '@scrayle/sdk';
import puppeteer from 'puppeteer-core';
const client = new Scrayle();
async function main() {
// Launch a cloud browser session
const session = await client.browser.create({
// Optional: specify a proxy region
proxyCountry: 'US',
});
console.log('WebSocket endpoint:', session.browserWSEndpoint);
// Connect Puppeteer to the remote browser
const browser = await puppeteer.connect({
browserWSEndpoint: session.browserWSEndpoint,
});
const page = await browser.newPage();
// Navigate to a target page
await page.goto('https://news.ycombinator.com', {
waitUntil: 'networkidle2',
timeout: 30000,
});
// Extract the top 10 stories
const stories = await page.evaluate(() => {
const rows = document.querySelectorAll('.athing');
return Array.from(rows).slice(0, 10).map(row => {
const titleEl = row.querySelector('.titleline > a');
const subtextRow = row.nextElementSibling;
const scoreEl = subtextRow?.querySelector('.score');
const ageEl = subtextRow?.querySelector('.age');
return {
rank: row.querySelector('.rank')?.textContent?.trim(),
title: titleEl?.textContent?.trim(),
url: titleEl?.getAttribute('href'),
points: scoreEl?.textContent?.trim() || '0 points',
age: ageEl?.textContent?.trim(),
};
});
});
console.log(JSON.stringify(stories, null, 2));
// Clean up
await browser.close();
}
main().catch(console.error);

Step 4: Run It

node scrape.js

You should see output like this:

WebSocket endpoint: wss://browser.scrayle.com/sess_a1b2c3d4
[
{
"rank": "1.",
"title": "Show HN: I built a self-hosted alternative to Notion",
"url": "https://example.com/project",
"points": "342 points",
"age": "3 hours ago"
},
...
]

The entire session - browser launch, navigation, rendering, and extraction - runs in an isolated cloud container. Your local machine only sends commands over WebSocket.

What's Happening Under the Hood

When you call client.browser.create(), Scrayle provisions a browser instance in approximately 2-4 seconds. Here's what it sets up for you automatically:

  • Fingerprint rotation: Each session gets a unique browser fingerprint (user-agent, WebGL hash, canvas noise, timezone, language) that matches the proxy region. This prevents sites from recognizing you based on browser characteristics.
  • Residential proxy egress: Requests leave through residential IP addresses in your chosen country. This matters because datacenter IPs are the first thing bot detection systems block.
  • Automatic stealth patches: The browser instance runs with stealth modifications that remove detectable automation signals - navigator.webdriver, Chrome DevTools protocol artifacts, and other giveaways that headless browsers normally expose.
  • CAPTCHA handling: If the target site presents a Cloudflare Turnstile or reCAPTCHA challenge, Scrayle solves it automatically. You don't need to add any special logic.

Working with Dynamic Content

Many modern sites load content asynchronously after the initial page render. Here are strategies for handling common patterns:

Wait for a Specific Element

// Wait for a data table to appear
await page.waitForSelector('.results-table tbody tr', {
timeout: 15000,
});

Wait for Network Activity to Settle

// Wait until no more than 2 network connections
// are active for 500ms
await page.goto(url, { waitUntil: 'networkidle2' });

Intercept API Responses Directly

Sometimes it's more efficient to intercept the underlying API call instead of parsing the rendered DOM:

// Listen for the API response before navigating
const dataPromise = page.waitForResponse(
res => res.url().includes('/api/products') && res.status() === 200
);
await page.goto('https://example.com/products');
const response = await dataPromise;
const products = await response.json();
console.log(products);

Handling Pagination

For multi-page scraping, loop through pages while reusing the same browser session. This is more efficient than creating a new session for each page:

const allItems = [];
for (let pageNum = 1; pageNum <= 5; pageNum++) {
await page.goto(
`https://example.com/listings?page=${pageNum}`,
{ waitUntil: 'networkidle2' }
);
const items = await page.evaluate(() => {
return Array.from(
document.querySelectorAll('.listing-card')
).map(card => ({
title: card.querySelector('h3')?.textContent?.trim(),
price: card.querySelector('.price')?.textContent?.trim(),
link: card.querySelector('a')?.href,
}));
});
allItems.push(...items);
console.log(`Page ${pageNum}: ${items.length} items`);
// Respectful delay between requests
await new Promise(r => setTimeout(r, 1000 + Math.random() * 2000));
}
console.log(`Total: ${allItems.length} items`);

Using Selenium Instead of Puppeteer

If your team prefers Selenium, Scrayle supports that too. The SDK provides a Selenium-compatible WebDriver URL:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
# Point Selenium at the Scrayle remote browser
driver = webdriver.Remote(
command_executor='https://browser.scrayle.com/wd/hub',
options=options,
# Pass your API key as a capability
desired_capabilities={
'scrayle:apiKey': 'sk_live_your_key_here',
'scrayle:proxy': {'country': 'US'},
}
)
driver.get('https://news.ycombinator.com')
titles = driver.find_elements('css selector', '.titleline > a')
for t in titles[:10]:
print(t.text, t.get_attribute('href'))
driver.quit()

Tips for Production Scraping

  • Set timeouts generously. Cloud browsers are fast, but network conditions and target site response times vary. Use 30-second timeouts for navigation and 15 seconds for element waits as a baseline.
  • Add random delays. Even with residential proxies, hitting pages faster than a human would is suspicious. Add 1-3 seconds of jitter between actions.
  • Handle errors gracefully. Pages may fail to load, elements may not appear, or sessions may time out. Wrap your extraction logic in try/catch blocks and implement retry logic for transient failures.
  • Close sessions when done. Each open session consumes credits. Always call browser.close() in a finally block to avoid leaked sessions.
  • Use the Live View. Scrayle's dashboard lets you watch browser sessions in real-time. This is invaluable for debugging selectors and understanding why a scraper isn't finding the expected elements.

What's Next

You now have a working cloud browser scraper. From here, you can:

  • Try the AI Browser Agent to describe what you want to scrape in plain English and let AI write the selectors for you
  • Set up scheduled workflows to run your scraper on a cron schedule with automatic data delivery
  • Explore the Universal Scraping API for lightweight HTML extraction without a full browser (faster and cheaper for static pages)

Need help? Check the documentation or reach out to the team through the dashboard. We typically respond within a few hours.

Related reading: