[Project_owners] Bot ignoring robots.txt

Pete Collins pete at mozdev.org
Tue Oct 28 13:12:23 EST 2003


Ok, I added a mod rewrite rule for robots.txt.

It willl come up for every project subdomwain:

Eg:

http://jslib.mozdev.org/robots.txt

http://xprint.mozdev.org/robots.txt

http://ieview.mozdev.org/robots.txt

Let's hope this stops most crawlers from traversing the cgi scripts on 
mozdev.

--pete

-- 
Pete Collins
www.mozdev.org
www.mozdevgroup.com




More information about the Project_owners mailing list