Search, and what AI is doing to it. Consulting, writing and tools.

SEO & LLM Diagnostic & Reference Tools

New

Can AI crawlers read your site?

A robots.txt file is only one of the places a crawler can be turned away. This checks what robots.txt says and what the server actually does when each crawler asks for the page, using the user-agent string each company documents.

The check sends up to 19 small requests to the page from this server, each carrying one crawler’s documented user-agent string. Use it on sites you own or are allowed to test. Results are kept for 15 minutes; a few checks per hour per visitor.

Known limit: the requests leave from this site’s host, and a host can exempt its own servers from the blocks it applies to everyone else (mine does). If the page you are checking is on the same host as this site, the check can report “Allowed” for a crawler the host actually blocks. Confirm with a request from somewhere else, such as a command line on your own computer.

New

AI crawler user-agent list

Every documented AI and search crawler in one spreadsheet: the token to use in robots.txt, what the company says it is for, whether it obeys robots.txt, and how to confirm a request really came from it. Compiled from each vendor’s own documentation, not from third-party bot directories.

Open the spreadsheet (Google Sheets; vendor documentation last checked October 20, 2026)