[Project_owners] Bot ignoring robots.txt

Pete Collins pete at mozdev.org
Tue Oct 28 13:27:12 EST 2003

Ben Bucksch wrote:
> Pete Collins wrote:
>> http://jslib.mozdev.org/robots.txt
> Why are you disallowing crawling of projects/active.html? Seems like a 
> pretty important page, if you want mozdev.org to be indexed on search 
> services. Esp. because inactive.html seems to be allowed.

It's a test. It is a CPU intensive page. Let's see how this works first, 
then we can tweak.


Pete Collins

More information about the Project_owners mailing list