robots.txt Generator
Create a properly formatted robots.txt file to guide search engine crawlers. Control which pages get indexed and add your sitemap for better SEO.
Builder
Allow
Disallow
robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /cart Disallow: /checkout
What Is robots.txt?
A robots.txt file is a text file placed in the root directory of your website that tells search engine crawlers which pages or files they can or cannot request from your site. It's a voluntary standard — well-behaved crawlers like Googlebot respect it, though malicious bots may ignore it.
Proper use of robots.txt helps you prevent thin or duplicate content from being indexed, protect admin areas, manage crawl budget, and direct search engines to your XML sitemap. However, never use robots.txt to hide sensitive data — use password protection or noindex tags instead.
Common robots.txt Rules for Shopify
User-agent: *
Allow: /
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Disallow: /password
Sitemap: https://yoursite.com/sitemap.xml