[Project_owners] Bot ignoring robots.txt

Ben Bucksch ben.bucksch.news at beonex.com
Wed Oct 29 16:10:01 EST 2003


massey wrote:

> Block those bot's IP's. Nothing else will work. If they are so rude as 
> to ignore your robots.txt you should just worry about mozdev. There 
> must be something about this kind of terrorism in the 'homeland 
> security act' :-)

It seems that mozdev.org didn't have a valid robots.txt in the first 
place, it has to be in the root, to my knowledge. You can't just place 
it in some subfolder and expect the robot to obey it.

<politics>
And querying a public webserver is hardly terrorism. Nobody died here. I 
don't want to be hunted and tortured and locked away without trial by 
the US just because I asked for too many public files.
</politics>

Ben

> Ben Bucksch wrote:
>
>> You said you have the robots.txt in the bugs/ dir. Does 
>> xprint.mozdev.org/robots.txt also disallow bugs/?
>




More information about the Project_owners mailing list