Sphinn - Irony: If Google Can't Reach Your Robots.txt File, It Might Not List Your Site

7
Sphinn It!

Irony: If Google Can't Reach Your Robots.txt File, It Might Not List Your Site

Posted By: theGypsy 115 days ago
Topic Type: News Story (Jump to http://searchengineland.com)
Category: Google SEO

Search Engine Land - It is important to clarify that Google will still index your site if you do not have a robots.txt file. If you simply do not have the robots.txt file on your server, it does not mean Google won't index your site. As long as Google sees a normal server status response, such as a 404 not found, Google will index your site. It is when Google is not sure if you have a robots.txt file, where it plays it on the safe side.

1 Comment

Who Desphunn This (0)

Comments

from jay5r 114 days ago #

Votes: 0 | Vote:

+ -

I saw this a year an a half ago when we took over a site for a client from their inhouse web developers - nothing was indexed and a timeout on robots.txt was the reason why... At least at the time, Yahoo! did index the site. Personally I prefer Google's approach - better safe than sorry.

And the problem is one that they clearly show you if you use Google Webmaster Tools. That's how I found it back then...

I'd also mention that this problem seems to be found fairly frequently on .NET-based servers - I don't know why...