Free Audit Get Started

Home / Free SEO Tools / Robots.txt Generator

FREE · NO SIGNUP

Robots.txt generator for WordPress and Shopify

Choose your platform, add your sitemap and private paths, and watch the file build live. Download it and upload it to your domain root.

Settings

robots.txt

Your file will appear here after you generate it.

Upload the file to the root of your domain so it opens at yourdomain.com/robots.txt.

ABOUT THIS TOOL

About the Robots.txt Generator

The Robots.txt Generator builds a clean robots.txt file for WordPress or Shopify. It adds the common private paths for each platform, lets you add your own, includes your sitemap, and can block AI training crawlers if you choose. It works for both a WordPress robots.txt and a Shopify robots.txt setup.

Why use this tool

Start from a safe preset

WordPress and Shopify each have common private paths. The preset gives you a tested starting point.

Control AI crawler access

Choose whether GPTBot, ClaudeBot, PerplexityBot and similar crawlers can access your content.

Copy or download in seconds

Get the file as text or as a robots.txt download, ready to upload to your domain root.

How to use it

  1. 01

    Pick your platform

    Select WordPress, Shopify or a custom setup. The preset fills in the common rules for you.

  2. 02

    Add your sitemap and extra paths

    Enter your sitemap URL and any private paths, one per line.

  3. 03

    Download and upload

    Upload the file to the root of your domain and check it at yourdomain.com/robots.txt.

What robots.txt does and does not do

Robots.txt tells crawlers which paths they may request. It does not remove a page from search results. To keep a page out of Google, use a noindex tag and allow crawling, so Google can read the instruction. Blocking a page in robots.txt alone can leave its URL visible in results without a description.

WordPress robots.txt: what to block and what to allow

Block /wp-admin/ to keep the admin area out of crawls, but allow /wp-admin/admin-ajax.php because themes and plugins use it. Do not block your CSS or JavaScript folders, since Google needs them to render the page. Add your sitemap line so search engines can find all your important URLs.

Shopify robots.txt and why it is different

Shopify generates a default robots.txt, which you can customise through the robots.txt template in your theme code. Common private paths include checkout, cart, account and internal search pages. Keep product and collection pages open to crawlers, since they are the pages you want to rank.

Frequently asked questions

Where do I upload robots.txt on WordPress?

Upload the file to the root of your domain, so it opens at yourdomain.com/robots.txt. On WordPress you can also manage it with an SEO plugin such as Rank Math.

Should I block /wp-admin/ in robots.txt?

Yes, blocking /wp-admin/ is common practice, but keep /wp-admin/admin-ajax.php allowed because some themes and plugins rely on it.

Does robots.txt remove pages from Google?

No. Robots.txt controls crawling, not indexing. To remove a page from search results, use a noindex meta tag and allow crawling so Google can see it.

More free tools

FREE TOOL

WordPress Speed Checker

Open tool →

FREE TOOL

WordPress Cost Calculator

Open tool →

FREE TOOL

Shopify Store Mini Audit

Open tool →

FREE TOOL

XML Sitemap Checker

Open tool →

FREE TOOL

JSON-LD Schema Generator

Open tool →

Not sure your crawl settings are right?

Our free website audit checks crawling, indexing and sitemap issues on your live site.

Get a free audit

robots.txt generator

Create a clean robots.txt generator file

This robots.txt generator builds a valid file for WordPress, Shopify or a custom site. Choose your platform, add your sitemap URL and any private paths, then decide how AI crawlers should reach your content. Download the file and upload it to the root of your domain, so it opens at yourdomain.com/robots.txt. The file updates live as you change the settings, so you can see exactly what will be published.

What robots.txt does and does not do

A robots.txt file tells crawlers which parts of your site they may request. It helps manage crawl activity and keeps admin areas out of the way. It does not hide a page from search results. The Google robots.txt documentation explains this clearly. To keep a page out of Google, use a noindex tag, and check your XML sitemap so important pages stay listed.

Platform presets explained

WordPress sites usually block the admin area while allowing the AJAX endpoint, which some themes and plugins need to work. Shopify stores often block checkout, cart, account and search paths, since these do not need to appear in search results. Custom sites need a manual list based on their structure. Whichever preset you choose, test the live file after uploading it, and check it again after theme or plugin changes.

Setting up AI crawlers

AI crawlers from several companies can read public content. You can allow them to keep your site visible in AI search answers, or block training crawlers if you prefer. The generator includes the common AI crawler names, so you can make the choice in one step. Decide based on your goals, and review the setting every few months as the list of crawlers changes.

Connect the file to your sitemap

Adding your sitemap line helps crawlers find your important pages quickly. Check the sitemap with our XML sitemap checker to confirm it loads and lists real URLs on your domain. Then use the JSON-LD schema generator if you also want structured data on your key pages.

When crawl settings need a professional review

Mistakes in robots.txt can block important pages by accident, so it is worth a second look if your traffic or indexing changes suddenly. Our free website audit checks crawling, indexing and sitemap issues on your live site. Our SEO services include a full technical review and a monthly check. You can also contact our team to discuss your crawl settings.

Testing your robots.txt after upload

After uploading the file, open yourdomain.com/robots.txt in a browser and check that the text matches what you generated. Confirm that the sitemap line points to the correct address. Then use the robots.txt report and the URL inspection tool in Google Search Console to check that important pages are not blocked by accident.

Check the file again after you install a new plugin, change your theme, or move the site. Plugins sometimes add their own rules, and a single extra line can block a whole folder. A quick review every few months keeps the file accurate.

Common robots.txt mistakes to avoid

The most serious mistake is blocking the whole site with a single slash after Disallow. Another is blocking CSS or JavaScript folders that search engines need to render pages. Some owners also use robots.txt to hide sensitive pages, but the file is public, so it should never be the only protection for private information.

If you are unsure, our team can review the file with you. Our WordPress development builds include a crawl check before launch.

Keeping crawl rules simple

A short, readable robots.txt file is easier to maintain than a long list of exceptions. Use only the rules you need, add a comment for each block so the next person understands the purpose, and review the file whenever the site structure changes.

Remember that robots.txt controls crawling, not indexing. A page can still appear in search if other sites link to it, so use noindex for pages you want removed from results.