[Project_owners] Bot ignoring robots.txt

massey massey at stlouis-shopper.com
Wed Oct 29 09:03:17 EST 2003


Block those bot's IP's. Nothing else will work. If they are so rude as 
to ignore your robots.txt you should just worry about mozdev. There must 
be something about this kind of terrorism in the 'homeland security act' :-)

Ben Bucksch wrote:

> Pete Collins wrote:
>
>> Here is an example of a bot ignoring robots.txt just today:
>
>
> You said you have the robots.txt in the bugs/ dir. Does 
> xprint.mozdev.org/robots.txt also disallow bugs/?
>
> _______________________________________________
> Project_owners mailing list
> Project_owners at mozdev.org
> http://mozdev.org/mailman/listinfo/project_owners
>




More information about the Project_owners mailing list