Sitemap Extractor {API}

Extract every page URL from any website sitemap with a single API call.

Pass a domain name and get back up to 500 deduplicated page URLs. Sitemap index files are crawled recursively. Non-page resources are filtered out automatically.

Perfect for building content indexes, seeding crawlers, or auditing a competitor's full site structure in seconds.

What You Get

Each request crawls a domain's sitemaps and returns all discoverable page URLs.

Up to 500 page URLs

Deduplicated, page-only results with non-page resources like images and PDFs automatically filtered out.

Sitemap index support

Recursively crawls nested sitemap index files with parallel fetching and concurrency control.

Crawl metadata

See how many sitemaps were discovered, fetched, skipped, and errored in a single response.

Normalized domain input

Pass just the domain name, no protocol needed. The API validates and normalizes domains automatically.

How It Works

We discover every relevant sitemap, follow sitemap indexes, and return a clean, deduplicated list of page URLs.

— step 01 —

Send a domain

Pass the domain name (e.g., "example.com" or "blog.example.com"). No protocol required.

— step 02 —

Sitemaps discovered

The API checks robots.txt and common sitemap paths, then recursively follows sitemap index files.

— step 03 —

URLs extracted and deduplicated

All page URLs are collected from every sitemap, deduplicated, and filtered to exclude non-page resources.

— step 04 —

Clean URL list returned

You get up to 500 normalized page URLs plus crawl metadata about the sitemap discovery process.

API Response

Discovered URLs for context.dev

GET /v1/web/scrape/sitemap?domain=context.dev

{
  "success": true,
  "domain": "context.dev",
  "urls": [\
    "https://context.dev/",\\n    "https://context.dev/pricing",\\n    "https://context.dev/blog",\\n    "https://context.dev/data/logo-api",\\n    "https://context.dev/use-cases/logo-link",\\n    "... up to 500 URLs"\\n  ],\
  "meta": {\
    "sitemapsDiscovered": 3,\
    "sitemapsFetched": 3,\
    "sitemapsSkipped": 0,\
    "errors": 0\
  }\
}

Ship an agent that actually knows things.

Free tier, 10-minute integration, and the same API powering agents at Mintlify, daily.dev, and Propane. No credit card to start.