Skip to main content
ScrayleScrayle
scraping storage apidataset apikey-value store scrapingweb scraping data pipelinemessage queue scraping

Storage API: KV, Datasets, Objects & Queues

Store, retrieve, and process scraped data with four built-in storage primitives - key-value store, datasets, object storage, and message queues.

Scrayle Team30-Jul-202511 min read

Extracting data is only half the problem. You also need somewhere to put it. Most scraping setups cobble together S3 buckets, Redis instances, Postgres databases, and message queues from different providers. Scrayle's built-in Storage API gives you all four data primitives in one place, fully integrated with the SDK, workflows, and dashboard.

The four storage systems:

  • Key-Value (KV): Fast key-value store for configuration, caching, and session state
  • Datasets: Structured collections of items with automatic stats tracking
  • Object Storage: File and binary storage with bucket organization
  • Queues: Message queues for async task processing and pipeline orchestration

Key-Value Store

The KV store is organized into namespaces. Each namespace contains key-value pairs. Use it for storing configuration, caching scraped data between runs, tracking scraper state, or any scenario where you need fast key-based lookups.

Namespaces

import { Scrayle } from '@scrayle/sdk';
const client = new Scrayle();
const kv = client.storage.kv;
// Create a namespace
const ns = await kv.createNamespace('price-cache');
console.log('Namespace ID:', ns.id);
// List all namespaces
const namespaces = await kv.listNamespaces({ page: 1, pageSize: 20 });
for (const n of namespaces.items) {
console.log(` ${n.name} - ${n.stats.count} keys, ${n.stats.size} bytes`);
}
// Rename a namespace
await kv.renameNamespace(ns.id, 'product-price-cache');
// Delete a namespace (and all its keys)
await kv.delNamespace(ns.id);

Key-Value Operations

const ns = await kv.createNamespace('scraper-state');
// Set a single value
await kv.setValue(ns.id, {
key: 'last-run-timestamp',
value: new Date().toISOString(),
});
// Set a value with expiration (TTL in seconds)
await kv.setValue(ns.id, {
key: 'session-token',
value: 'abc123xyz',
expiration: 3600, // Expires in 1 hour
});
// Get a value
const timestamp = await kv.getValue(ns.id, 'last-run-timestamp');
console.log('Last run:', timestamp);
// Bulk set multiple values at once
const result = await kv.bulkSetValue(ns.id, [
{ key: 'page-1-cursor', value: 'cursor_abc' },
{ key: 'page-2-cursor', value: 'cursor_def' },
{ key: 'page-3-cursor', value: 'cursor_ghi' },
]);
console.log(`Set ${result.successfulKeyCount} keys`);
// List all keys in a namespace
const keys = await kv.listKeys(ns.id, { page: 1, pageSize: 100 });
for (const k of keys.items) {
console.log(` ${k.key} (${k.size} bytes)`);
}
// Delete a single key
await kv.delValue(ns.id, 'session-token');
// Bulk delete
await kv.bulkDelValue(ns.id, ['page-1-cursor', 'page-2-cursor']);

Datasets

Datasets are structured collections designed for scraped data. Each dataset tracks item count and byte size automatically. Use datasets when you want to store tabular or structured data that you'll query or export later.

const ds = client.storage.dataset;
// Create a dataset
const dataset = await ds.createDataset('laptop-prices');
console.log('Dataset ID:', dataset.id);
// Add items (any JSON objects)
await ds.addItems(dataset.id, [
{
name: 'MacBook Pro 16"',
price: 2499,
currency: 'USD',
url: 'https://apple.com/macbook-pro',
scrapedAt: new Date().toISOString(),
},
{
name: 'ThinkPad X1 Carbon',
price: 1649,
currency: 'USD',
url: 'https://lenovo.com/thinkpad-x1',
scrapedAt: new Date().toISOString(),
},
]);
// Retrieve items with pagination
const items = await ds.getItems(dataset.id, {
page: 1,
pageSize: 50,
desc: true, // Newest first
});
console.log(`Total items: ${items.total}`);
for (const item of items.items) {
console.log(item);
}
// Check dataset stats
const info = await ds.getDataset(dataset.id);
console.log(`Items: ${info.stats.count}, Size: ${info.stats.size} bytes`);
// List all datasets
const datasets = await ds.listDatasets({ page: 1, pageSize: 20 });
for (const d of datasets.items) {
console.log(` ${d.name} - ${d.stats.count} items`);
}
// Rename a dataset
await ds.updateDataset(dataset.id, 'laptop-prices-2025');
// Delete a dataset and all its items
await ds.delDataset(dataset.id);

Object Storage

Object storage is for files - screenshots, PDFs, downloaded documents, exported CSVs, or any binary data. Objects are organized into buckets, similar to S3.

const obj = client.storage.object;
// Create a bucket
const bucket = await obj.createBucket({
name: 'screenshots',
description: 'Page screenshots from crawl jobs',
});
// Upload a file
const uploadResult = await obj.put(bucket.id, {
file: '/path/to/screenshot.png',
});
console.log('Object ID:', uploadResult.object_id);
console.log('Size:', uploadResult.size, 'bytes');
// List objects in a bucket
const objects = await obj.list(bucket.id, {
page: 1,
pageSize: 50,
search: '.png',
});
for (const o of objects.objects) {
console.log(` ${o.filename} (${o.fileType}, ${o.size} bytes)`);
}
// Download an object
const blob = await obj.get(bucket.id, 'obj_abc123');
// Delete an object
await obj.delete(bucket.id, 'obj_abc123');
// List all buckets
const buckets = await obj.listBuckets({ page: 1, pageSize: 20 });
for (const b of buckets.buckets) {
console.log(` ${b.name} - ${b.size} bytes total`);
}
// Delete a bucket
await obj.deleteBucket(bucket.id);

Queues

Queues enable async task processing. Push URLs or job descriptions onto a queue, then have workers pull and process them. This is the foundation for building scalable scraping pipelines where one stage discovers URLs and the next stage extracts data.

const q = client.storage.queue;
// Create a queue
const queue = await q.create({
name: 'urls-to-scrape',
description: 'Discovered URLs waiting for extraction',
runId: 'run_001',
actorId: 'crawler',
});
// Push messages onto the queue
await q.push(queue.id, {
name: 'product-page',
payload: JSON.stringify({
url: 'https://example.com/product/123',
category: 'electronics',
}),
retry: 3, // Retry up to 3 times on failure
timeout: 30000, // 30 second processing timeout
deadline: 0,
});
// Pull messages for processing (up to 5 at a time)
const messages = await q.pull(queue.id, 5);
for (const msg of messages) {
const data = JSON.parse(msg.payload);
console.log(`Processing: ${data.url}`);
try {
// Scrape the URL...
// On success, acknowledge the message
await q.ack(queue.id, msg.id);
} catch (err) {
// Message will be retried automatically
console.error(`Failed: ${data.url} - ${err.message}`);
}
}
// Check queue stats
const info = await q.get('urls-to-scrape');
console.log('Pending:', info.stats.pending);
console.log('Running:', info.stats.running);
console.log('Succeeded:', info.stats.success);
console.log('Failed:', info.stats.failed);
// Delete a queue
await q.delete(queue.id);

Building a Pipeline with Storage

Here's a complete example that uses queues to coordinate a multi-stage scraping pipeline:

import { Scrayle } from '@scrayle/sdk';
const client = new Scrayle();
async function discoveryStage() {
// Stage 1: Crawl a site to discover product URLs
const result = await client.scrapingCrawl.crawl.crawlUrl(
'https://example.com/products',
{
maxDepth: 2,
limit: 100,
scrapeOptions: { formats: ['links'] },
}
);
// Push discovered URLs onto the queue
const queue = await client.storage.queue.create({
name: 'product-urls',
description: 'URLs discovered by crawler',
runId: 'pipeline-001',
actorId: 'discovery',
});
for (const page of result.data || []) {
const productLinks = (page.links || [])
.filter(link => link.includes('/product/'));
for (const url of productLinks) {
await client.storage.queue.push(queue.id, {
name: 'product',
payload: url,
retry: 2,
timeout: 30000,
deadline: 0,
});
}
}
return queue.id;
}
async function extractionStage(queueId) {
// Stage 2: Process URLs from the queue
const dataset = await client.storage.dataset.createDataset('products');
let batch;
do {
batch = await client.storage.queue.pull(queueId, 5);
for (const msg of batch) {
const result = await client.scrapingCrawl.scrape.scrapeUrl(
msg.payload,
{ formats: ['json'] }
);
if (result.success && result.data?.json) {
await client.storage.dataset.addItems(dataset.id, [result.data.json]);
}
await client.storage.queue.ack(queueId, msg.id);
}
} while (batch.length > 0);
return dataset.id;
}
// Run the pipeline
const queueId = await discoveryStage();
const datasetId = await extractionStage(queueId);
const stats = await client.storage.dataset.getDataset(datasetId);
console.log(`Pipeline complete: ${stats.stats.count} products extracted`);

What's Next


Full API reference: See the storage documentation at docs.scrayle.com.

See also: Scrayle vs Apify: Full Platform Comparison