Blog
Technology, SEO, and Gen AI · Entrepreneurs and Startups · SEO Archive
Articles I have written for other publications are on the Writing page.
Recent writing on search and AI
-
AI Crawler Settings by Host and CDN: Cloudflare, Fastly, Akamai, WP Engine, Shopify, and More
Your host or CDN may block AI crawlers before they reach your site. Learn about defaults and available configurations for common ones.
-
Complete Reference On Allowing & Blocking AI Crawlers
Learn how LLM bots crawl your site and how to control the crawl. This comprehensive guide includes user agent lists for each LLM and what they do, a tool to check what your site is blocking and allowing, and the implications for search and LLM…