Published: Nov 20, 2007 - 07:17 pm
Story Found By: Sebastian 2876 Days ago
I dont know for sure which experimental crawler directives Google has implemented yet, but for example a line like
in your robots.txt will now deindex your complete Web site.
Better check your robots.txt and make sure it doesnt contain crawler directives belonging to robots meta tags respectively X-Robots-Tags.