[Project_owners] Bot ignoring robots.txt
pete at mozdev.org
Tue Oct 28 13:27:12 EST 2003
Ben Bucksch wrote:
> Pete Collins wrote:
> Why are you disallowing crawling of projects/active.html? Seems like a
> pretty important page, if you want mozdev.org to be indexed on search
> services. Esp. because inactive.html seems to be allowed.
It's a test. It is a CPU intensive page. Let's see how this works first,
then we can tweak.
More information about the Project_owners