Custom Robot.txt Generator

Custom Robot.txt Generator for Blogger and WordPress

Generate a robots.txt for your Blogger or WordPress site in seconds.

Very important! Enter the complete URL of your website in the field.

For example: https://www.yoursitename.com


  

 

Copy Your Custom Robot.txt



What is robots.txt?

A custom robots.txt contains instructions for bots. Good bots, such as web crawlers, are controlled by robots.txt files.

A bot is computer software that interacts with websites and applications in an automated way. There are good bots and bad bots, and a web crawler bot is one of the good ones. These bots "crawl" webpages and index content for it to appear in search engine results. A robots.txt file helps web crawlers control their actions so that they don't overload the web server hosting the website or index pages that aren't intended for public viewing.

How robots.txt file work?

While a robots.txt file can provide bots instructions, it cannot enforce those instructions. Before visiting any other pages on a domain, a good bot, such as a web crawler or a news feed bot, will try to visit the robots.txt file and follow the instructions.

The most specific set of instructions in the robots.txt file will be followed by a web crawler bot. If the file contains contradictory commands, the bot will use the more granular command.

One thing to keep in mind is that each subdomain needs its robots.txt file. While www.mysite.com has its file, all of the mysite subdomains: blog.mysite.com, forum.mysite.com, and so on require their own.