What Is robots.txt, and Could It Be Hiding Your Site From Google?

Sometimes a website looks perfect, yet it never shows up in Google. One cause is a very short text file called robots.txt. It sits at the root of your website and gives instructions to the automated visitors that search engines send out to read your pages.

What it does

The file tells those visitors which parts of your site they should and should not look at. Most of the time it says very little, or simply welcomes everyone in. Used well, it keeps search engines away from pages that offer no value, such as internal admin areas or search result pages generated by your own site.

How it can go wrong

The risky part is how little it takes to block everything. A single line can tell search engines to stay out of your entire site. This often happens by accident. A site is built in a private test version where blocking search engines makes sense, then launched without removing the block. The site goes live, looks great, and quietly stays invisible.

If you want to understand why test versions exist, read what a staging site is.

How to check yours in a minute

Type your website address into your browser and add /robots.txt at the end. You will see a plain page of text.

• If you see a line that says "Disallow: /" under "User-agent: *", your whole site is being blocked. That needs fixing right away
• If the file is short, or lists only specific folders such as an admin area, that is usually fine
• If there is no file at all, search engines will simply read everything, which is also usually fine

Good things to know

• It is a request, not a lock. Well-behaved search engines follow it, but it does not protect private information. Anything truly private needs a password, not a robots.txt line
• Blocking a page does not always remove it from results. If other sites link to it, it can still appear without a description
• You can point to your sitemap here. Many sites add a line with the sitemap address. See do you need a sitemap

Check it after any big change

Look at this file after a redesign, a move to a new address, or a rebuild. Also look in Google Search Console, which will warn you if pages are blocked.

It is one of the cheapest checks in all of SEO, and a surprising number of businesses never do it.