Storage API: KV, Datasets, Objects & Queues
Store, retrieve, and process scraped data with four built-in storage primitives - key-value store, datasets, object storage, and message queues.
Extracting data is only half the problem. You also need somewhere to put it. Most scraping setups cobble together S3 buckets, Redis instances, Postgres databases, and message queues from different providers. Scrayle's built-in Storage API gives you all four data primitives in one place, fully integrated with the SDK, workflows, and dashboard.
The four storage systems:
- Key-Value (KV): Fast key-value store for configuration, caching, and session state
- Datasets: Structured collections of items with automatic stats tracking
- Object Storage: File and binary storage with bucket organization
- Queues: Message queues for async task processing and pipeline orchestration
Key-Value Store
The KV store is organized into namespaces. Each namespace contains key-value pairs. Use it for storing configuration, caching scraped data between runs, tracking scraper state, or any scenario where you need fast key-based lookups.
Namespaces
import { Scrayle } from '@scrayle/sdk';const client = new Scrayle();const kv = client.storage.kv;// Create a namespaceconst ns = await kv.createNamespace('price-cache');console.log('Namespace ID:', ns.id);// List all namespacesconst namespaces = await kv.listNamespaces({ page: 1, pageSize: 20 });for (const n of namespaces.items) {console.log(` ${n.name} - ${n.stats.count} keys, ${n.stats.size} bytes`);}// Rename a namespaceawait kv.renameNamespace(ns.id, 'product-price-cache');// Delete a namespace (and all its keys)await kv.delNamespace(ns.id);
Key-Value Operations
const ns = await kv.createNamespace('scraper-state');// Set a single valueawait kv.setValue(ns.id, {key: 'last-run-timestamp',value: new Date().toISOString(),});// Set a value with expiration (TTL in seconds)await kv.setValue(ns.id, {key: 'session-token',value: 'abc123xyz',expiration: 3600, // Expires in 1 hour});// Get a valueconst timestamp = await kv.getValue(ns.id, 'last-run-timestamp');console.log('Last run:', timestamp);// Bulk set multiple values at onceconst result = await kv.bulkSetValue(ns.id, [{ key: 'page-1-cursor', value: 'cursor_abc' },{ key: 'page-2-cursor', value: 'cursor_def' },{ key: 'page-3-cursor', value: 'cursor_ghi' },]);console.log(`Set ${result.successfulKeyCount} keys`);// List all keys in a namespaceconst keys = await kv.listKeys(ns.id, { page: 1, pageSize: 100 });for (const k of keys.items) {console.log(` ${k.key} (${k.size} bytes)`);}// Delete a single keyawait kv.delValue(ns.id, 'session-token');// Bulk deleteawait kv.bulkDelValue(ns.id, ['page-1-cursor', 'page-2-cursor']);
Datasets
Datasets are structured collections designed for scraped data. Each dataset tracks item count and byte size automatically. Use datasets when you want to store tabular or structured data that you'll query or export later.
const ds = client.storage.dataset;// Create a datasetconst dataset = await ds.createDataset('laptop-prices');console.log('Dataset ID:', dataset.id);// Add items (any JSON objects)await ds.addItems(dataset.id, [{name: 'MacBook Pro 16"',price: 2499,currency: 'USD',url: 'https://apple.com/macbook-pro',scrapedAt: new Date().toISOString(),},{name: 'ThinkPad X1 Carbon',price: 1649,currency: 'USD',url: 'https://lenovo.com/thinkpad-x1',scrapedAt: new Date().toISOString(),},]);// Retrieve items with paginationconst items = await ds.getItems(dataset.id, {page: 1,pageSize: 50,desc: true, // Newest first});console.log(`Total items: ${items.total}`);for (const item of items.items) {console.log(item);}// Check dataset statsconst info = await ds.getDataset(dataset.id);console.log(`Items: ${info.stats.count}, Size: ${info.stats.size} bytes`);// List all datasetsconst datasets = await ds.listDatasets({ page: 1, pageSize: 20 });for (const d of datasets.items) {console.log(` ${d.name} - ${d.stats.count} items`);}// Rename a datasetawait ds.updateDataset(dataset.id, 'laptop-prices-2025');// Delete a dataset and all its itemsawait ds.delDataset(dataset.id);
Object Storage
Object storage is for files - screenshots, PDFs, downloaded documents, exported CSVs, or any binary data. Objects are organized into buckets, similar to S3.
const obj = client.storage.object;// Create a bucketconst bucket = await obj.createBucket({name: 'screenshots',description: 'Page screenshots from crawl jobs',});// Upload a fileconst uploadResult = await obj.put(bucket.id, {file: '/path/to/screenshot.png',});console.log('Object ID:', uploadResult.object_id);console.log('Size:', uploadResult.size, 'bytes');// List objects in a bucketconst objects = await obj.list(bucket.id, {page: 1,pageSize: 50,search: '.png',});for (const o of objects.objects) {console.log(` ${o.filename} (${o.fileType}, ${o.size} bytes)`);}// Download an objectconst blob = await obj.get(bucket.id, 'obj_abc123');// Delete an objectawait obj.delete(bucket.id, 'obj_abc123');// List all bucketsconst buckets = await obj.listBuckets({ page: 1, pageSize: 20 });for (const b of buckets.buckets) {console.log(` ${b.name} - ${b.size} bytes total`);}// Delete a bucketawait obj.deleteBucket(bucket.id);
Queues
Queues enable async task processing. Push URLs or job descriptions onto a queue, then have workers pull and process them. This is the foundation for building scalable scraping pipelines where one stage discovers URLs and the next stage extracts data.
const q = client.storage.queue;// Create a queueconst queue = await q.create({name: 'urls-to-scrape',description: 'Discovered URLs waiting for extraction',runId: 'run_001',actorId: 'crawler',});// Push messages onto the queueawait q.push(queue.id, {name: 'product-page',payload: JSON.stringify({url: 'https://example.com/product/123',category: 'electronics',}),retry: 3, // Retry up to 3 times on failuretimeout: 30000, // 30 second processing timeoutdeadline: 0,});// Pull messages for processing (up to 5 at a time)const messages = await q.pull(queue.id, 5);for (const msg of messages) {const data = JSON.parse(msg.payload);console.log(`Processing: ${data.url}`);try {// Scrape the URL...// On success, acknowledge the messageawait q.ack(queue.id, msg.id);} catch (err) {// Message will be retried automaticallyconsole.error(`Failed: ${data.url} - ${err.message}`);}}// Check queue statsconst info = await q.get('urls-to-scrape');console.log('Pending:', info.stats.pending);console.log('Running:', info.stats.running);console.log('Succeeded:', info.stats.success);console.log('Failed:', info.stats.failed);// Delete a queueawait q.delete(queue.id);
Building a Pipeline with Storage
Here's a complete example that uses queues to coordinate a multi-stage scraping pipeline:
import { Scrayle } from '@scrayle/sdk';const client = new Scrayle();async function discoveryStage() {// Stage 1: Crawl a site to discover product URLsconst result = await client.scrapingCrawl.crawl.crawlUrl('https://example.com/products',{maxDepth: 2,limit: 100,scrapeOptions: { formats: ['links'] },});// Push discovered URLs onto the queueconst queue = await client.storage.queue.create({name: 'product-urls',description: 'URLs discovered by crawler',runId: 'pipeline-001',actorId: 'discovery',});for (const page of result.data || []) {const productLinks = (page.links || []).filter(link => link.includes('/product/'));for (const url of productLinks) {await client.storage.queue.push(queue.id, {name: 'product',payload: url,retry: 2,timeout: 30000,deadline: 0,});}}return queue.id;}async function extractionStage(queueId) {// Stage 2: Process URLs from the queueconst dataset = await client.storage.dataset.createDataset('products');let batch;do {batch = await client.storage.queue.pull(queueId, 5);for (const msg of batch) {const result = await client.scrapingCrawl.scrape.scrapeUrl(msg.payload,{ formats: ['json'] });if (result.success && result.data?.json) {await client.storage.dataset.addItems(dataset.id, [result.data.json]);}await client.storage.queue.ack(queueId, msg.id);}} while (batch.length > 0);return dataset.id;}// Run the pipelineconst queueId = await discoveryStage();const datasetId = await extractionStage(queueId);const stats = await client.storage.dataset.getDataset(datasetId);console.log(`Pipeline complete: ${stats.stats.count} products extracted`);
What's Next
- Learn about recursive crawling to feed your storage pipelines with data
- Set up scheduled workflows to automate data collection into datasets
- Get started with the Scraping Browser for interactive data extraction
Full API reference: See the storage documentation at docs.scrayle.com.