Search, and what AI is doing to it. Consulting, writing and tools.

SEO & LLM Diagnostic & Reference Tools

New

Can AI crawlers read your site?

A robots.txt file is only one of the places a crawler can be turned away. This checks what robots.txt says and what the server actually does when each crawler asks for the page, using the user-agent string each company documents.

The check sends up to 19 small requests to the page from this server, each carrying one crawler’s documented user-agent string. Use it on sites you own or are allowed to test. Results are kept for 15 minutes; a few checks per hour per visitor.

New

AI crawler user-agent list

Every documented AI and search crawler in one spreadsheet: the token to use in robots.txt, what the company says it is for, whether it obeys robots.txt, and how to confirm a request really came from it. Compiled from each vendor’s own documentation, not from third-party bot directories.

Open the spreadsheet (Google Sheets; vendor documentation last checked October 20, 2026)