How the crawler works.

Crawl modes

The mode decides how pages are found, not how they are extracted — extraction is the same either way. If you are not sure which to use, probe the site first and take the recommendation.

All three fetch concurrently and crawl breadth-first, one depth level at a time.

How a page becomes text

Every fetched page goes through the same pipeline, whatever mode found it.

1. 1 Trafilatura extracts the main content as markdown, keeping links, tables and formatting, and dropping navigation, headers and boilerplate.
2. 2 If that comes back empty or suspiciously short next to the page's visible text, a DOM fallback runs instead and the larger result wins. The page records which one was used as extraction_method.
3. 3 Metadata, JSON-LD and microdata are pulled out separately, so structured data survives even when the prose does not.
4. 4 Links are collected and split into internal and external, which is what feeds the next depth level in normal mode.

URL handling

Two URLs that render the same page are normalised to the same key so the same content is never crawled twice: the scheme and host are lowercased, fragments and tracking parameters are dropped, remaining parameters are sorted, and index files and trailing slashes are stripped.

That normalised form is used for deduplication only. The URL actually requested is the one the site advertises — sitemap entries and links are fetched exactly as written. This matters on sites that canonicalise to a trailing slash, where requesting the stripped form would earn a redirect that misses the CDN and doubles the number of requests.

HTTP API

Four endpoints, no authentication. POST bodies are JSON. Every response carries a success boolean, and failures carry an error string.

A crawl, end to end

Probe, start, poll, export. The crawl runs detached — a dropped poll does not stop it, and you can pick the job back up with the same job_id until it is delivered.

curl -X POST https://crawller.dev/api/probe.php \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com"}'

# 1. start the job
curl -X POST https://crawller.dev/api/crawl.php \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","crawl_mode":"sitemap","max_pages":200}'

# → {"success":true,"job_id":"crawl_6a94...","mode":"sitemap","crawls_remaining":11}

# 2. poll it
curl "https://crawller.dev/api/status.php?job_id=crawl_6a94..."

# 3. re-render the finished crawl in any format
curl "https://crawller.dev/api/export.php?file=example.com-2026-08-30_120000-a1b2c3.json&format=llms-full&inline=1"

While it runs

{
  "success": true,
  "state": "running",
  "progress": {
    "state":      "running",
    "crawled":    84,
    "discovered": 397,
    "max_pages":  200,
    "current":    "https://example.com/pricing/"
  }
}

When it finishes

{
  "success": true,
  "state": "complete",
  "saved_as": "example.com-2026-08-30_120000-a1b2c3.json",
  "data": {
    "summary": {
      "startUrl":       "https://example.com/",
      "baseDomain":     "example.com",
      "totalPages":     120,
      "successPages":   120,
      "errorPages":     0,
      "emptyPages":     0,
      "discoveredUrls": 397,
      "maxDepth":       1,
      "crawlMode":      "sitemap",
      "workers":        5,
      "crawlTime":      "18.6s",
      "timestamp":      "2026-08-30T12:00:00.000000",
      "usedSitemap":    true,
      "truncated":      true
    },
    "pages": [
      {
        "url":         "https://example.com/pricing/",
        "title":       "Pricing",
        "description": "Simple per-seat pricing.",
        "content":     "## Pricing\n\nSimple per-seat pricing...",
        "wordCount":   612,
        "extraction_method": "trafilatura",
        "metadata":    { "canonical": "...", "openGraph": {} },
        "schema":      { "jsonLd": [], "microdata": [] },
        "links":       { "internal": [], "external": [] },
        "error":       null
      }
    ]
  }
}

Output formats

Set on the crawler with --format, or applied after the fact by export.php. Re-rendering a saved crawl never re-fetches the site.

Limits

The public API is unauthenticated, so it is rate limited per IP. Values a request exceeds are clamped rather than rejected.

crawl.php returns a crawls_remaining count with every job so you can see where you stand. Private and reserved IP addresses are refused.

Command line

The crawler runs standalone with no PHP and no web server — pyScripts/site_crawler.py. It needs trafilatura, beautifulsoup4, requests and lxml_html_clean.

# what will reach the most pages?
python pyScripts/site_crawler.py https://example.com --probe -p

# whole site from the sitemap, unlimited pages, 8 workers
python pyScripts/site_crawler.py https://example.com --sitemap --max-pages 0 --workers 8

# WordPress, every public post type
python pyScripts/site_crawler.py https://example.com --wordpress --wp-types all

# straight to llms-full on disk
python pyScripts/site_crawler.py https://example.com --sitemap --format llms-full -o site.txt

# one page, markdown only
python pyScripts/site_crawler.py https://example.com/about --single --markdown

# re-render an existing crawl without touching the network
python pyScripts/site_crawler.py --render-file site.json --format markdown