[Project_owners] Bot ignoring robots.txt

Pete Collins pete at mozdev.org
Tue Oct 28 13:27:12 EST 2003


Ben Bucksch wrote:
> Pete Collins wrote:
> 
>> http://jslib.mozdev.org/robots.txt
> 
> 
> Why are you disallowing crawling of projects/active.html? Seems like a 
> pretty important page, if you want mozdev.org to be indexed on search 
> services. Esp. because inactive.html seems to be allowed.

It's a test. It is a CPU intensive page. Let's see how this works first, 
then we can tweak.

--pete


-- 
Pete Collins
www.mozdev.org
www.mozdevgroup.com




More information about the Project_owners mailing list