Model Context Protocol

Crawl websites from
inside Claude.

Connect Crawllr once and Claude can pull any site into the conversation — following links, reading sitemaps, or going straight through the WordPress REST API — and hand it back as Markdown, llms.txt or JSON.

Server URL

https://crawller.dev/api/mcp.php

No account, no API key. Rate limited to 150 requests per hour per IP.

Install

Pick whichever matches how you use Claude. All three end up with the same three tools.

Easiest

Claude Code

One command in your terminal. Add --scope user to make it available in every project rather than just the current one.

Terminal

claude mcp add --transport http crawllr https://crawller.dev/api/mcp.php

Confirm it registered:

claude mcp list

Claude.ai and the Claude desktop app

Added as a custom connector — no config file to edit.

1 Open Settings, then Connectors.
2 Click Add custom connector.
3 Paste the server URL above.
4 Save, then start a new conversation — the Crawllr tools appear in the tools menu.

On older desktop builds without a Connectors screen, edit claude_desktop_config.json instead — ~/Library/Application Support/Claude/ on macOS, %APPDATA%\Claude\ on Windows — and bridge the HTTP server through mcp-remote:

claude_desktop_config.json

{
  "mcpServers": {
    "crawllr": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://crawller.dev/api/mcp.php"]
    }
  }
}

Restart the app fully after editing — reopening the window is not enough.

Run it yourself

If you would rather not send URLs through a hosted service, run the STDIO server yourself. Needs PHP 8.1+ and Python 3.9+. The source is not public yet — get in touch if you want a copy.

Terminal

# from your Crawllr checkout

# PHP side (the MCP server itself)
composer install --ignore-platform-req=ext-fileinfo

# Python side (the crawler engine)
pip install trafilatura beautifulsoup4 requests lxml_html_clean

# register it
claude mcp add crawllr -- php "$(pwd)/mcp/server.php"

The --ignore-platform-req flag is there because the MCP SDK lists ext-fileinfo as a requirement it never actually exercises. Drop the flag if your PHP has the extension.

Tools

Three tools. Claude picks between them on its own, but knowing what they do helps you ask for the right thing.

inspect_site Call this first.

Checks what a site exposes before you crawl it: whether it publishes a sitemap, whether the WordPress REST API is open, and how many pages each strategy would reach. Returns a recommended crawl_mode to pass straight to crawl_website.

Parameter	Type	Default	Description
`url` req	string	—	The site to inspect. Must be http or https.

crawl_website The main one.

Crawls a site and returns its content. Three strategies, and six output formats — set output_format to markdown or llms-full when the result is going straight into a prompt.

Parameter	Type	Default	Description
`url` req	string	—	The site to crawl.
`crawl_mode`	string	"normal"	One of "normal", "sitemap", "wordpress". Use what inspect_site recommends.
`output_format`	string	"json"	"json", "clean-json", "markdown", "llms", "llms-full", or "text".
`max_pages`	integer	100	1–1000.
`depth`	integer	2	Link levels to follow, 1–5. Ignored in wordpress mode.
`wp_types`	string	"posts,pages"	WordPress mode only. Comma-separated REST bases, or "all" for every public post type.
`include_paths`	string	""	Restrict to these path prefixes, e.g. "/blog,/docs".
`exclude_paths`	string	""	Skip these paths, e.g. "/tag,/author".
`include_subdomains`	boolean	false	Follow links onto subdomains of the same site.

crawl_page One URL, nothing else.

Extracts a single page without following any links. Much faster than crawl_website when you only need one URL.

Parameter	Type	Default	Description
`url` req	string	—	The page to fetch.
`markdown_only`	boolean	false	Return just the markdown text instead of the full JSON object.

Crawl modes

The mode decides how pages are found, and it is the single biggest factor in how much of a site you actually get. inspect_site works this out for you.

normal

Any site. The fallback when there is nothing better.

Walks internal links breadth-first from the start page. Only reaches pages that something links to.

sitemap

Static sites, and anywhere with orphan pages.

Reads robots.txt and every common sitemap location, follows sitemap indexes, and handles gzipped sitemaps. Finds pages no menu links to.

wordpress

Any WordPress site with an open REST API.

Enumerates published content directly through /wp-json/wp/v2/. The most complete option, and several times faster — there is no HTML to parse. Falls back to fetching the live page when a page builder leaves the REST content empty.

Things to ask for

Once it is connected, plain English is enough — but naming the tool or mode makes Claude go straight there.

Pull the whole docs site into context

“Use crawllr to inspect docs.example.com, then crawl it with whatever mode it recommends and give me the result as llms-full.”

Audit a client site for broken links

“Crawl example.com from its sitemap with crawllr and list every page that returned an error.”

Grab one page, cleanly

“Use crawl_page on https://example.com/blog/post and give me just the markdown.”

Everything a WordPress site has published

“example.com is WordPress — crawl it with crawl_mode wordpress and wp_types all.”

What comes back

With the default output_format: "json" you get a summary plus one object per page. A page that fails to load is reported in pages[].error rather than aborting the crawl — which makes a crawl double as a broken-link report. Ask for markdown or llms-full instead when you just want the text.

output_format: json

{
  "summary": {
    "startUrl":     "https://example.com/",
    "baseDomain":   "example.com",
    "totalPages":   42,
    "successPages": 40,
    "errorPages":   2,
    "crawlMode":    "sitemap",
    "crawlTime":    "18.6s",
    "truncated":    false
  },
  "pages": [
    {
      "url":         "https://example.com/pricing",
      "title":       "Pricing",
      "description": "Simple per-seat pricing.",
      "content":     "## Pricing\n\nSimple per-seat pricing...",
      "wordCount":   612,
      "extraction_method": "trafilatura",
      "metadata":    { "canonical": "...", "openGraph": { } },
      "schema":      { "jsonLd": [ ], "microdata": [ ] },
      "links":       { "internal": [ ], "external": [ ] },
      "error":       null
    }
  ]
}

If something is not working

The tools do not show up

Start a new conversation — existing ones do not pick up newly added servers. In Claude Code, run claude mcp list to confirm it registered and shows as connected.

A crawl returns far fewer pages than the site has

It probably ran in normal mode on a site whose pages are not all linked. Ask for inspect_site first, then crawl with the mode it recommends — sitemap and wordpress both reach pages that link-following cannot.

Pages come back with no content

The page is likely rendered client-side in JavaScript. Crawllr reads the HTML the server sends, so a site that ships an empty shell and fills it in with JS will extract as blank.

The crawl timed out

Lower max_pages, or switch to wordpress or sitemap mode — both are much faster than link-following because they know every URL up front.

Rate limited

150 requests per hour per IP on the hosted endpoint. Run the STDIO server locally if you need more than that.

Prefer a browser? The same engine runs the web crawler.

Open the web crawler →

Crawl websites frominside Claude.