Browser Profiles: Authenticated Scraping at Scale
Persist browser sessions across runs with profiles, store login credentials securely, and automate authenticated scraping at scale.
Most valuable web data sits behind a login. Price sheets for existing customers, internal dashboards, account-specific product catalogs, social media feeds - you need to be authenticated to access any of it. But managing login sessions across dozens of scraping jobs running on disposable cloud browser instances is a real problem.
Scrayle solves this with two features that work together: Browser Profiles(persistent browser state that survives across sessions) and the Credential Center (a secure vault for storing login credentials that your workflows can use automatically).
Browser Profiles
A browser profile is a saved browser state - cookies, local storage, session tokens, IndexedDB data - that persists between Scrayle browser sessions. Without profiles, every new browser session starts fresh. With profiles, you can log in once and reuse that authenticated state for days or weeks.
Creating a Profile
import { Scrayle } from '@scrayle/sdk';const client = new Scrayle();// Create a named profileconst profile = await client.profiles.create('amazon-us-account');console.log('Profile ID:', profile.profileId);console.log('Name:', profile.name);console.log('Created:', profile.createdAt);
Listing Profiles
// List all profiles with paginationconst profiles = await client.profiles.list({page: 1,limit: 20,});console.log(`Total profiles: ${profiles.totalDocs}`);for (const p of profiles.docs) {console.log(` ${p.name} (ID: ${p.profileId}) - updated ${p.updatedAt}`);}
Using a Profile in a Browser Session
When you create a browser session, pass the profile ID to load saved state and persist any new state back when the session ends:
import puppeteer from 'puppeteer-core';const client = new Scrayle();// Create a session with a profile attachedconst session = await client.browser.create({profileId: 'prof_abc123',proxyCountry: 'US',});const browser = await puppeteer.connect({browserWSEndpoint: session.browserWSEndpoint,});const page = await browser.newPage();// First run: log in manuallyawait page.goto('https://example.com/login');await page.type('#password', 'your-password');await page.click('#login-button');await page.waitForNavigation();// The session cookies and state are now saved to the profile.// Next time you use this profile, you'll already be logged in.await browser.close();
On subsequent sessions using the same profile ID, the browser starts with the saved cookies and storage. The login page will be skipped entirely because the session tokens are already present.
Profile Management
// Get a specific profileconst profile = await client.profiles.get('prof_abc123');console.log(profile.name, profile.updatedAt);// Search profiles by nameconst results = await client.profiles.list({page: 1,limit: 10,name: 'amazon',});// Delete a profile when no longer neededawait client.profiles.delete('prof_abc123');
Credential Center
The Credential Center is a dashboard feature for securely storing login credentials that your scraping workflows can use. Instead of hardcoding usernames and passwords in your workflow definitions or code, you store them once in the Credential Center and reference them by name.
What You Can Store
Each credential entry supports:
- Name: A descriptive label like "Amazon US Business Account" or "LinkedIn Recruiter"
- Identifier: The login username, email, or account ID
- Password: Encrypted at rest, never exposed in logs or API responses
- 2FA Secret: TOTP secret key for accounts with two-factor authentication. Scrayle generates the 2FA code automatically at login time.
- Auth Mode: The authentication method - standard form login, OAuth flow, or custom
- Usage Rules: Which workflows and team members can access this credential
Setting Up Credentials in the Dashboard
- Navigate to Credentials in the dashboard sidebar
- Click Add Credential
- Fill in the credential details - name, identifier, password, and optionally the 2FA secret
- Set access rules to control which workflows can use this credential
- Save - the password and 2FA secret are encrypted immediately
How Credentials Work with Workflows
When you build a workflow in the visual editor or via the AI agent, you can reference stored credentials for login steps. The workflow engine:
- Detects the login form on the target page
- Retrieves the credential from the encrypted vault
- Fills in the username and password fields
- If 2FA is configured, generates the current TOTP code and enters it
- Completes the login and continues with the rest of the workflow
The credential values are never written to workflow definitions, logs, or execution history. They're decrypted at runtime only for the specific browser session that needs them.
Combining Profiles and Credentials
The real power comes from using both features together. Here's the typical workflow for authenticated scraping at scale:
- Store credentials in the Credential Center for each target site account
- Create a browser profile for each account
- Build a workflow that uses the credential for initial login and the profile for session persistence
- Schedule the workflow to run on a regular cadence. The first run logs in using the credential and saves the session to the profile. Subsequent runs use the cached session from the profile, only falling back to a fresh login if the session expires.
This approach minimizes login frequency (which reduces the risk of account lockouts) while keeping your credentials secure.
Security
- Encryption: All credential passwords and 2FA secrets are encrypted at rest using AES-256. They are decrypted only in memory during workflow execution.
- Access control: Credentials are scoped to your account. Team members can only access credentials they've been explicitly granted access to.
- Audit logging: Every credential access is logged with the workflow ID, timestamp, and action performed.
- No exposure in API responses: The API never returns password or 2FA secret values. Only the credential name, identifier, and metadata are visible.
Tips for Authenticated Scraping
- Use dedicated accounts for scraping. Don't use your personal login. Create a separate account on each target site specifically for data extraction.
- Rotate profiles periodically. Session tokens expire. If a profile's session becomes stale, the workflow will need to re-authenticate. Set up health checks to detect this early.
- Match proxy regions to account regions. If your target site account is registered in the US, use a US proxy to avoid triggering geo-based security checks.
- Keep login frequency low. Profiles are specifically designed to reduce how often you need to log in. Logging in too frequently from different IPs can trigger account security measures.
- Use 2FA when available. Counter-intuitively, enabling 2FA on your scraping accounts is more secure. Scrayle handles the TOTP codes automatically, and it prevents anyone else from accessing the account if the password is compromised.
What's Next
- New to Scrayle? Start with the getting started guide
- Want to control which IPs you scrape from? Read the proxy solutions guide
- Ready to automate? Set up scheduled workflows with self-healing selectors
Dashboard access: Manage profiles and credentials at app.scrayle.com.