[Project_owners] Bot ignoring robots.txt

Ben Bucksch ben.bucksch.news at beonex.com
Tue Oct 28 19:22:08 EST 2003


Pete Collins wrote:

> http://jslib.mozdev.org/robots.txt

Why are you disallowing crawling of projects/active.html? Seems like a 
pretty important page, if you want mozdev.org to be indexed on search 
services. Esp. because inactive.html seems to be allowed.



More information about the Project_owners mailing list