About Our Crawler

What this is

Dead Link Crawler is a broken-link checker at deadlinkcrawler.com. When a user submits your URL to scan, our server visits pages on your site to find dead links, then reports the results back to that user. We only crawl a site when a person explicitly requests a scan of it; we do not run unattended, scheduled crawls of the open web.

How to identify our requests

Our requests send this User-Agent:

Mozilla/5.0 (compatible; DeadLinkChecker/1.0; +https://deadlinkcrawler.com/bot)

Requests are also signed per RFC 9421 / Web Bot Auth. Our public signing key is published at /.well-known/http-message-signatures-directory, so services that support it can verify a request genuinely came from us rather than from an unrelated visitor sharing our IP range.

How we crawl

• We respect robots.txt, including crawl-delay and disallowed paths.
• We throttle ourselves per domain: at most two concurrent requests, with a minimum delay between requests, and we back off on 429 responses.
• We do not attempt to bypass logins, paywalls, or CAPTCHAs.
• A crawl only follows links within the same site as the URL the user submitted.

Blocking our crawler

If you would rather we not crawl your site, add the following to your robots.txt and we will honor it:

User-agent: DeadLinkChecker
Disallow: /

Contact

Questions about our crawler? Email us at contact@deadlinkcrawler.com.