Crawl: Recursive Site Indexing at Scale
Recursively crawl entire websites with configurable depth, path filtering, output format control, and async job management through a single SDK call.
Scraping a single page is straightforward. Scraping an entire website - hundreds or thousands of pages, following links, respecting rate limits, handling errors, and producing clean structured output - is a different challenge entirely. That's what Scrayle's Crawl service is built for.
You give it a starting URL and a set of rules (depth limits, path filters, output formats), and it recursively discovers and scrapes every reachable page. The output comes back as clean markdown, HTML, JSON, or even screenshots - ready to feed into an LLM, a database, or a search index.
Basic Crawl
Start a crawl with a single function call:
import { Scrayle } from '@scrayle/sdk';const client = new Scrayle();// Crawl and wait for completion (synchronous)const result = await client.scrapingCrawl.crawl.crawlUrl('https://docs.example.com',{maxDepth: 3,limit: 100,scrapeOptions: {formats: ['markdown', 'links'],onlyMainContent: true,},});console.log(`Status: ${result.status}`);console.log(`Pages crawled: ${result.completed} / ${result.total}`);for (const page of result.data || []) {console.log(page.metadata?.title, page.metadata?.sourceURL);console.log(page.markdown?.substring(0, 200));console.log('---');}
The crawlUrl method starts the crawl and polls until it completes. For large sites, this can take minutes. Use the async variant if you need to do other work while waiting.
Async Crawling
For large sites, start the crawl asynchronously and check back later:
// Start the crawl without waitingconst job = await client.scrapingCrawl.crawl.asyncCrawlUrl('https://example.com',{maxDepth: 5,limit: 500,scrapeOptions: {formats: ['markdown', 'html'],onlyMainContent: true,},});console.log('Crawl started:', job.id);// Check progress periodicallyconst interval = setInterval(async () => {const status = await client.scrapingCrawl.crawl.checkCrawlStatus(job.id);console.log(`Progress: ${status.completed}/${status.total} (${status.status})`);if (status.status === 'completed' || status.status === 'failed') {clearInterval(interval);if (status.data) {console.log(`Crawl complete. ${status.data.length} pages collected.`);}}}, 5000);
Path Filtering
Control which pages get crawled by specifying include and exclude paths. This is essential for large sites where you only need specific sections:
const result = await client.scrapingCrawl.crawl.crawlUrl('https://example.com',{maxDepth: 4,limit: 200,// Only crawl pages under /docs/ and /blog/includePaths: ['/docs/*', '/blog/*'],// Skip media galleries and admin pagesexcludePaths: ['/gallery/*', '/admin/*', '/wp-json/*'],// Follow links that go back up the path hierarchyallowBackwardLinks: true,// Don't follow links to other domainsallowExternalLinks: false,scrapeOptions: {formats: ['markdown'],onlyMainContent: true,},});
Output Formats
Each crawled page can be returned in multiple formats simultaneously. Choose what works best for your use case:
markdown- Clean text with structure preserved. Best for LLM ingestion, RAG pipelines, and content analysis.html- Cleaned HTML with boilerplate removed. Good for content processing pipelines that need markup.rawHtml- The full, unprocessed HTML. Use this when you need to inspect the original page structure.links- All URLs found on each page. Useful for building site maps or finding broken links.screenshot- A screenshot of each page. Useful for visual monitoring or archival.json- Structured data extracted from the page. Works with the extraction schema feature.
const result = await client.scrapingCrawl.crawl.crawlUrl('https://docs.example.com',{maxDepth: 2,limit: 50,scrapeOptions: {formats: ['markdown', 'html', 'links', 'screenshot'],// Only extract the main content, skip nav/footer/sidebaronlyMainContent: true,// Filter which HTML elements to processincludeTags: ['article', 'main', '.content'],excludeTags: ['nav', 'footer', '.sidebar', '.ads'],},});for (const page of result.data || []) {console.log('URL:', page.metadata?.sourceURL);console.log('Title:', page.metadata?.title);console.log('Status:', page.metadata?.statusCode);console.log('Links found:', page.links?.length);console.log('Markdown length:', page.markdown?.length);}
Webhook Notifications
For large crawls, you can set a webhook URL to be notified when the crawl finishes instead of polling:
const job = await client.scrapingCrawl.crawl.asyncCrawlUrl('https://example.com',{maxDepth: 5,limit: 1000,webhook: 'https://your-api.com/crawl-complete',scrapeOptions: {formats: ['markdown'],onlyMainContent: true,},});// Scrayle will POST the results to your webhook URL when done
Error Handling
Not every page in a crawl will succeed. Some URLs may be broken, require authentication, or time out. Check per-page errors after the crawl completes:
const job = await client.scrapingCrawl.crawl.asyncCrawlUrl(url, params);// Wait for completion...const status = await client.scrapingCrawl.crawl.checkCrawlStatus(job.id);// Check for individual page errorsconst errors = await client.scrapingCrawl.crawl.checkCrawlErrors(job.id);for (const err of errors.errors) {console.error(`Failed: ${err.url} - ${err.error}`);}// Cancel a running crawl if neededawait client.scrapingCrawl.crawl.cancelCrawl(job.id);
Building an LLM Knowledge Base
One of the most common use cases for crawling is building a knowledge base from documentation sites. Here's a complete example that crawls a docs site and prepares the content for embedding:
import { Scrayle } from '@scrayle/sdk';const client = new Scrayle();async function buildKnowledgeBase(siteUrl) {console.log('Starting crawl of', siteUrl);const result = await client.scrapingCrawl.crawl.crawlUrl(siteUrl, {maxDepth: 5,limit: 500,includePaths: ['/docs/*', '/guides/*', '/api/*'],excludePaths: ['/changelog/*'],ignoreSitemap: false,scrapeOptions: {formats: ['markdown'],onlyMainContent: true,excludeTags: ['nav', 'footer', '.sidebar', '.breadcrumb'],},});console.log(`Crawled ${result.completed} pages`);// Transform into documents ready for embeddingconst documents = (result.data || []).filter(page => page.markdown && page.markdown.length > 100).map(page => ({url: page.metadata?.sourceURL,title: page.metadata?.title || 'Untitled',content: page.markdown,wordCount: page.markdown?.split(/\s+/).length,}));console.log(`${documents.length} documents ready for embedding`);return documents;}buildKnowledgeBase('https://docs.example.com').then(docs => {// Send to your embedding pipeline// e.g., OpenAI embeddings, Pinecone, Weaviate, etc.});
What's Next
- Need to scrape individual pages? See the Universal Scraping API guide
- Want AI to handle the extraction? Try the AI Browser Agent
- Need to store crawl results? Read about the Storage API
Full API reference: See the crawl endpoint documentation at docs.scrayle.com.