Knowledgebase

Why the robots.txt File Matters

The robots.txt file is a small but critical part of any website’s SEO foundation. It tells search engine bots (also known as crawlers or spiders) which pages they can or should not crawl and index on your site.

While modern search engines like Google and Bing are incredibly powerful, they still follow the guidelines you set in your robots.txt. This means:

• You can prevent indexing of sensitive pages (like checkout, login, or admin areas).

• You can prioritise which content gets crawled, ensuring your most important pages are seen first.

• You maintain better control over bandwidth and crawl budgets, especially useful for larger WooCommerce stores.

In short, having a well-configured robots.txt file can improve your site's searchability and protect it from accidentally exposing pages you’d rather keep private.

More of this is covered in our Sitemap and IndexNow Sections.

How to Use the Blue Raven robots.txt Tool

Once Blue Raven is installed on your WordPress site:

Navigate to the Blue Raven Menu:

• In your WordPress dashboard, look at the left-hand nav bar and click on Blue Raven. 

• At the top of the Blue Raven dashboard, click on the “Tools” tab.

• Locate the “Searchability” Section:

  • Under Tools, you’ll find a category labeled Searchability. This section includes powerful features to help new and existing WooCommerce stores optimise their visibility online.

• Using the robots.txt Tool:

  • If no robots.txt file is found:

    • Blue Raven will show you a default template.

    • You can edit this template to suit your site needs or simply click Save to publish it as-is.

   

• If a robots.txt file already exists:

  • Blue Raven will detect it and display the current file contents.

  • You can edit this file directly within the plugin interface.

  • Once you're happy with your changes, click Save to update the file.

• ✅ All of this is done without needing to access your server via a Hosting Control Panel or FTP.

If you're using tools like Google Search Console or Bing Webmaster Tools, it's a good idea to submit your robots.txtfile for verification. This ensures the latest version is used by search engines.

We also recommend submitting your sitemap file, which Blue Raven can generate for you under the Site Map tool (found in the same “Tools” section).

Want faster indexing? Blue Raven also supports IndexNow – a protocol backed by Bing for instant URL indexing. You’ll need to generate an API key from Bing Webmaster Tools. (We cover this in the Sitemap and IndexNow KB article.)

Feature Requests

If you have any feature requests that you would like us to consider please log Feature Request We would love to get your input to help make Blue Raven even better!