Skip to main content
ScrayleScrayle
web crawling apirecursive site crawlingsite indexing apiasync crawlingbulk url crawling

Crawl: Recursive Site Indexing at Scale

Recursively crawl entire websites with configurable depth, path filtering, output format control, and async job management through a single SDK call.

Scrayle Team28-Jul-20259 min read

Scraping a single page is straightforward. Scraping an entire website - hundreds or thousands of pages, following links, respecting rate limits, handling errors, and producing clean structured output - is a different challenge entirely. That's what Scrayle's Crawl service is built for.

You give it a starting URL and a set of rules (depth limits, path filters, output formats), and it recursively discovers and scrapes every reachable page. The output comes back as clean markdown, HTML, JSON, or even screenshots - ready to feed into an LLM, a database, or a search index.

Basic Crawl

Start a crawl with a single function call:

import { Scrayle } from '@scrayle/sdk';
const client = new Scrayle();
// Crawl and wait for completion (synchronous)
const result = await client.scrapingCrawl.crawl.crawlUrl(
'https://docs.example.com',
{
maxDepth: 3,
limit: 100,
scrapeOptions: {
formats: ['markdown', 'links'],
onlyMainContent: true,
},
}
);
console.log(`Status: ${result.status}`);
console.log(`Pages crawled: ${result.completed} / ${result.total}`);
for (const page of result.data || []) {
console.log(page.metadata?.title, page.metadata?.sourceURL);
console.log(page.markdown?.substring(0, 200));
console.log('---');
}

The crawlUrl method starts the crawl and polls until it completes. For large sites, this can take minutes. Use the async variant if you need to do other work while waiting.

Async Crawling

For large sites, start the crawl asynchronously and check back later:

// Start the crawl without waiting
const job = await client.scrapingCrawl.crawl.asyncCrawlUrl(
'https://example.com',
{
maxDepth: 5,
limit: 500,
scrapeOptions: {
formats: ['markdown', 'html'],
onlyMainContent: true,
},
}
);
console.log('Crawl started:', job.id);
// Check progress periodically
const interval = setInterval(async () => {
const status = await client.scrapingCrawl.crawl.checkCrawlStatus(job.id);
console.log(`Progress: ${status.completed}/${status.total} (${status.status})`);
if (status.status === 'completed' || status.status === 'failed') {
clearInterval(interval);
if (status.data) {
console.log(`Crawl complete. ${status.data.length} pages collected.`);
}
}
}, 5000);

Path Filtering

Control which pages get crawled by specifying include and exclude paths. This is essential for large sites where you only need specific sections:

const result = await client.scrapingCrawl.crawl.crawlUrl(
'https://example.com',
{
maxDepth: 4,
limit: 200,
// Only crawl pages under /docs/ and /blog/
includePaths: ['/docs/*', '/blog/*'],
// Skip media galleries and admin pages
excludePaths: ['/gallery/*', '/admin/*', '/wp-json/*'],
// Follow links that go back up the path hierarchy
allowBackwardLinks: true,
// Don't follow links to other domains
allowExternalLinks: false,
scrapeOptions: {
formats: ['markdown'],
onlyMainContent: true,
},
}
);

Output Formats

Each crawled page can be returned in multiple formats simultaneously. Choose what works best for your use case:

  • markdown - Clean text with structure preserved. Best for LLM ingestion, RAG pipelines, and content analysis.
  • html - Cleaned HTML with boilerplate removed. Good for content processing pipelines that need markup.
  • rawHtml - The full, unprocessed HTML. Use this when you need to inspect the original page structure.
  • links - All URLs found on each page. Useful for building site maps or finding broken links.
  • screenshot - A screenshot of each page. Useful for visual monitoring or archival.
  • json - Structured data extracted from the page. Works with the extraction schema feature.
const result = await client.scrapingCrawl.crawl.crawlUrl(
'https://docs.example.com',
{
maxDepth: 2,
limit: 50,
scrapeOptions: {
formats: ['markdown', 'html', 'links', 'screenshot'],
// Only extract the main content, skip nav/footer/sidebar
onlyMainContent: true,
// Filter which HTML elements to process
includeTags: ['article', 'main', '.content'],
excludeTags: ['nav', 'footer', '.sidebar', '.ads'],
},
}
);
for (const page of result.data || []) {
console.log('URL:', page.metadata?.sourceURL);
console.log('Title:', page.metadata?.title);
console.log('Status:', page.metadata?.statusCode);
console.log('Links found:', page.links?.length);
console.log('Markdown length:', page.markdown?.length);
}

Webhook Notifications

For large crawls, you can set a webhook URL to be notified when the crawl finishes instead of polling:

const job = await client.scrapingCrawl.crawl.asyncCrawlUrl(
'https://example.com',
{
maxDepth: 5,
limit: 1000,
webhook: 'https://your-api.com/crawl-complete',
scrapeOptions: {
formats: ['markdown'],
onlyMainContent: true,
},
}
);
// Scrayle will POST the results to your webhook URL when done

Error Handling

Not every page in a crawl will succeed. Some URLs may be broken, require authentication, or time out. Check per-page errors after the crawl completes:

const job = await client.scrapingCrawl.crawl.asyncCrawlUrl(url, params);
// Wait for completion...
const status = await client.scrapingCrawl.crawl.checkCrawlStatus(job.id);
// Check for individual page errors
const errors = await client.scrapingCrawl.crawl.checkCrawlErrors(job.id);
for (const err of errors.errors) {
console.error(`Failed: ${err.url} - ${err.error}`);
}
// Cancel a running crawl if needed
await client.scrapingCrawl.crawl.cancelCrawl(job.id);

Building an LLM Knowledge Base

One of the most common use cases for crawling is building a knowledge base from documentation sites. Here's a complete example that crawls a docs site and prepares the content for embedding:

import { Scrayle } from '@scrayle/sdk';
const client = new Scrayle();
async function buildKnowledgeBase(siteUrl) {
console.log('Starting crawl of', siteUrl);
const result = await client.scrapingCrawl.crawl.crawlUrl(siteUrl, {
maxDepth: 5,
limit: 500,
includePaths: ['/docs/*', '/guides/*', '/api/*'],
excludePaths: ['/changelog/*'],
ignoreSitemap: false,
scrapeOptions: {
formats: ['markdown'],
onlyMainContent: true,
excludeTags: ['nav', 'footer', '.sidebar', '.breadcrumb'],
},
});
console.log(`Crawled ${result.completed} pages`);
// Transform into documents ready for embedding
const documents = (result.data || [])
.filter(page => page.markdown && page.markdown.length > 100)
.map(page => ({
url: page.metadata?.sourceURL,
title: page.metadata?.title || 'Untitled',
content: page.markdown,
wordCount: page.markdown?.split(/\s+/).length,
}));
console.log(`${documents.length} documents ready for embedding`);
return documents;
}
buildKnowledgeBase('https://docs.example.com')
.then(docs => {
// Send to your embedding pipeline
// e.g., OpenAI embeddings, Pinecone, Weaviate, etc.
});

What's Next


Full API reference: See the crawl endpoint documentation at docs.scrayle.com.

See also: Scrayle vs Apify: Full Platform Comparison