Get Started: Your First Scraping Browser Session
A step-by-step guide to launching your first cloud browser session, connecting with Puppeteer or Selenium, and extracting data from a dynamic page.
Web scraping is harder than it used to be. JavaScript-heavy single-page applications, bot detection systems, CAPTCHAs, and fingerprinting techniques mean that simple HTTP requests no longer cut it for most targets. You need a real browser - but running headless Chrome at scale brings its own headaches: memory management, proxy rotation, crash recovery, and infrastructure costs.
Scrayle's Scraping Browser solves this by giving you on-demand cloud browser sessions that you control with the same Puppeteer or Playwright code you already know. Each session runs in an isolated container with built-in fingerprint rotation, automatic CAPTCHA solving, and residential proxy egress. You write normal browser automation code - Scrayle handles everything else.
What You'll Build
By the end of this tutorial, you will have a working Node.js script that:
- Connects to a Scrayle cloud browser session via WebSocket
- Navigates to a JavaScript-rendered page
- Waits for dynamic content to load
- Extracts structured data using CSS selectors
- Outputs clean JSON you can pipe to a file or database
Prerequisites
- A Scrayle account (the free tier includes 1,000 credits)
- Node.js 18 or later installed locally
- Basic familiarity with JavaScript and CSS selectors
Step 1: Install the Scrayle SDK
Start by creating a new project directory and installing the SDK:
mkdir my-scraper && cd my-scrapernpm init -ynpm install @scrayle/sdk puppeteer-core
The SDK provides the Scrayle client that manages authentication and session lifecycle. We use puppeteer-core (not the full puppeteer package) because Scrayle provides the browser - you only need the client protocol library.
Step 2: Get Your API Key
Log into the Scrayle dashboard and navigate to Settings > API Keys. Create a new key and copy it. Store it as an environment variable:
# .envSCRAYLE_API_KEY=sk_live_your_key_here
Never commit API keys to version control. The SDK automatically reads from theSCRAYLE_API_KEY environment variable if you don't pass one explicitly.
Step 3: Launch a Browser Session
Create a file called scrape.js with the following code:
import { Scrayle } from '@scrayle/sdk';import puppeteer from 'puppeteer-core';const client = new Scrayle();async function main() {// Launch a cloud browser sessionconst session = await client.browser.create({// Optional: specify a proxy regionproxyCountry: 'US',});console.log('WebSocket endpoint:', session.browserWSEndpoint);// Connect Puppeteer to the remote browserconst browser = await puppeteer.connect({browserWSEndpoint: session.browserWSEndpoint,});const page = await browser.newPage();// Navigate to a target pageawait page.goto('https://news.ycombinator.com', {waitUntil: 'networkidle2',timeout: 30000,});// Extract the top 10 storiesconst stories = await page.evaluate(() => {const rows = document.querySelectorAll('.athing');return Array.from(rows).slice(0, 10).map(row => {const titleEl = row.querySelector('.titleline > a');const subtextRow = row.nextElementSibling;const scoreEl = subtextRow?.querySelector('.score');const ageEl = subtextRow?.querySelector('.age');return {rank: row.querySelector('.rank')?.textContent?.trim(),title: titleEl?.textContent?.trim(),url: titleEl?.getAttribute('href'),points: scoreEl?.textContent?.trim() || '0 points',age: ageEl?.textContent?.trim(),};});});console.log(JSON.stringify(stories, null, 2));// Clean upawait browser.close();}main().catch(console.error);
Step 4: Run It
node scrape.js
You should see output like this:
WebSocket endpoint: wss://browser.scrayle.com/sess_a1b2c3d4[{"rank": "1.","title": "Show HN: I built a self-hosted alternative to Notion","url": "https://example.com/project","points": "342 points","age": "3 hours ago"},...]
The entire session - browser launch, navigation, rendering, and extraction - runs in an isolated cloud container. Your local machine only sends commands over WebSocket.
What's Happening Under the Hood
When you call client.browser.create(), Scrayle provisions a browser instance in approximately 2-4 seconds. Here's what it sets up for you automatically:
- Fingerprint rotation: Each session gets a unique browser fingerprint (user-agent, WebGL hash, canvas noise, timezone, language) that matches the proxy region. This prevents sites from recognizing you based on browser characteristics.
- Residential proxy egress: Requests leave through residential IP addresses in your chosen country. This matters because datacenter IPs are the first thing bot detection systems block.
- Automatic stealth patches: The browser instance runs with stealth modifications that remove detectable automation signals -
navigator.webdriver, Chrome DevTools protocol artifacts, and other giveaways that headless browsers normally expose. - CAPTCHA handling: If the target site presents a Cloudflare Turnstile or reCAPTCHA challenge, Scrayle solves it automatically. You don't need to add any special logic.
Working with Dynamic Content
Many modern sites load content asynchronously after the initial page render. Here are strategies for handling common patterns:
Wait for a Specific Element
// Wait for a data table to appearawait page.waitForSelector('.results-table tbody tr', {timeout: 15000,});
Wait for Network Activity to Settle
// Wait until no more than 2 network connections// are active for 500msawait page.goto(url, { waitUntil: 'networkidle2' });
Intercept API Responses Directly
Sometimes it's more efficient to intercept the underlying API call instead of parsing the rendered DOM:
// Listen for the API response before navigatingconst dataPromise = page.waitForResponse(res => res.url().includes('/api/products') && res.status() === 200);await page.goto('https://example.com/products');const response = await dataPromise;const products = await response.json();console.log(products);
Handling Pagination
For multi-page scraping, loop through pages while reusing the same browser session. This is more efficient than creating a new session for each page:
const allItems = [];for (let pageNum = 1; pageNum <= 5; pageNum++) {await page.goto(`https://example.com/listings?page=${pageNum}`,{ waitUntil: 'networkidle2' });const items = await page.evaluate(() => {return Array.from(document.querySelectorAll('.listing-card')).map(card => ({title: card.querySelector('h3')?.textContent?.trim(),price: card.querySelector('.price')?.textContent?.trim(),link: card.querySelector('a')?.href,}));});allItems.push(...items);console.log(`Page ${pageNum}: ${items.length} items`);// Respectful delay between requestsawait new Promise(r => setTimeout(r, 1000 + Math.random() * 2000));}console.log(`Total: ${allItems.length} items`);
Using Selenium Instead of Puppeteer
If your team prefers Selenium, Scrayle supports that too. The SDK provides a Selenium-compatible WebDriver URL:
from selenium import webdriverfrom selenium.webdriver.chrome.options import Optionsoptions = Options()# Point Selenium at the Scrayle remote browserdriver = webdriver.Remote(command_executor='https://browser.scrayle.com/wd/hub',options=options,# Pass your API key as a capabilitydesired_capabilities={'scrayle:apiKey': 'sk_live_your_key_here','scrayle:proxy': {'country': 'US'},})driver.get('https://news.ycombinator.com')titles = driver.find_elements('css selector', '.titleline > a')for t in titles[:10]:print(t.text, t.get_attribute('href'))driver.quit()
Tips for Production Scraping
- Set timeouts generously. Cloud browsers are fast, but network conditions and target site response times vary. Use 30-second timeouts for navigation and 15 seconds for element waits as a baseline.
- Add random delays. Even with residential proxies, hitting pages faster than a human would is suspicious. Add 1-3 seconds of jitter between actions.
- Handle errors gracefully. Pages may fail to load, elements may not appear, or sessions may time out. Wrap your extraction logic in try/catch blocks and implement retry logic for transient failures.
- Close sessions when done. Each open session consumes credits. Always call
browser.close()in a finally block to avoid leaked sessions. - Use the Live View. Scrayle's dashboard lets you watch browser sessions in real-time. This is invaluable for debugging selectors and understanding why a scraper isn't finding the expected elements.
What's Next
You now have a working cloud browser scraper. From here, you can:
- Try the AI Browser Agent to describe what you want to scrape in plain English and let AI write the selectors for you
- Set up scheduled workflows to run your scraper on a cron schedule with automatic data delivery
- Explore the Universal Scraping API for lightweight HTML extraction without a full browser (faster and cheaper for static pages)
Need help? Check the documentation or reach out to the team through the dashboard. We typically respond within a few hours.
Related reading: