Robots.txt Checker

What robots.txt does

When Google, Bing or another search engine visits your site, robots.txt is the first file it asks for. It sits in the root of your domain, for example https://yourdomain.co.ke/robots.txt, and spells out what crawlers may and may not fetch.

The main directives

• User-agent: names the crawler a block is meant for. Use * for every crawler, or a name such as Googlebot or Bingbot for one bot.
• Allow: opens up a path explicitly.
• Disallow: keeps crawlers away from a path.
• Sitemap: points crawlers to your XML sitemap, and can list more than one URL.
• Crawl-delay: asks for a pause between requests. Bing and Yandex respect it, while Google ignores it.

Mistakes we see often

• Disallow: / left on a live site hides all of it from search engines. We have seen production traffic collapse when a staging robots.txt was pushed live by mistake.
• Leaving out the Sitemap line. Google can learn about your sitemap through Search Console, but listing it in robots.txt helps Bing and smaller crawlers find it too.
• Using robots.txt for security. Crawlers are asked, not forced, to obey it. Listing private paths there only tells everyone where they are.