[Project_owners] Bot ignoring robots.txt
ben.bucksch.news at beonex.com
Wed Oct 29 16:10:01 EST 2003
> Block those bot's IP's. Nothing else will work. If they are so rude as
> to ignore your robots.txt you should just worry about mozdev. There
> must be something about this kind of terrorism in the 'homeland
> security act' :-)
It seems that mozdev.org didn't have a valid robots.txt in the first
place, it has to be in the root, to my knowledge. You can't just place
it in some subfolder and expect the robot to obey it.
And querying a public webserver is hardly terrorism. Nobody died here. I
don't want to be hunted and tortured and locked away without trial by
the US just because I asked for too many public files.
> Ben Bucksch wrote:
>> You said you have the robots.txt in the bugs/ dir. Does
>> xprint.mozdev.org/robots.txt also disallow bugs/?
More information about the Project_owners