# Sitemap Extractor {API}

Extract every page URL from any website sitemap with a single API call.

Pass a domain name and get back up to 500 deduplicated page URLs. Sitemap index files are crawled recursively. Non-page resources are filtered out automatically.

Perfect for building content indexes, seeding crawlers, or auditing a competitor's full site structure in seconds.

## What You Get

Each request crawls a domain's sitemaps and returns all discoverable page URLs.

#### Up to 500 page URLs

Deduplicated, page-only results with non-page resources like images and PDFs automatically filtered out.

#### Sitemap index support

Recursively crawls nested sitemap index files with parallel fetching and concurrency control.

#### Crawl metadata

See how many sitemaps were discovered, fetched, skipped, and errored in a single response.

#### Normalized domain input

Pass just the domain name, no protocol needed. The API validates and normalizes domains automatically.

## How It Works

We discover every relevant sitemap, follow sitemap indexes, and return a clean, deduplicated list of page URLs.

— step 01 —

#### Send a domain

Pass the domain name (e.g., "example.com" or "blog.example.com"). No protocol required.

— step 02 —

#### Sitemaps discovered

The API checks robots.txt and common sitemap paths, then recursively follows sitemap index files.

— step 03 —

#### URLs extracted and deduplicated

All page URLs are collected from every sitemap, deduplicated, and filtered to exclude non-page resources.

— step 04 —

#### Clean URL list returned

You get up to 500 normalized page URLs plus crawl metadata about the sitemap discovery process.

## API Response

Discovered URLs for context.dev

`GET /v1/web/scrape/sitemap?domain=context.dev`

```
{
  "success": true,
  "domain": "context.dev",
  "urls": [\
    "https://context.dev/",\\n    "https://context.dev/pricing",\\n    "https://context.dev/blog",\\n    "https://context.dev/data/logo-api",\\n    "https://context.dev/use-cases/logo-link",\\n    "... up to 500 URLs"\\n  ],\
  "meta": {\
    "sitemapsDiscovered": 3,\
    "sitemapsFetched": 3,\
    "sitemapsSkipped": 0,\
    "errors": 0\
  }\
}
```

## Ship an agent that actually knows things.

Free tier, 10-minute integration, and the same API powering agents at Mintlify, daily.dev, and Propane. No credit card to start.