[Project_owners] Bot ignoring robots.txt
davidf at sjsoft.com
Wed Oct 29 17:04:11 EST 2003
> Block those bot's IP's. Nothing else will work. If they are so rude as
> to ignore your robots.txt you should just worry about mozdev. There
> must be something about this kind of terrorism in the 'homeland
> security act' :-)
In fact this may be a nice way to get rid of them.
You could put a link in robots.txt to a particular URL which nobody else
would access (linked from an invisible pixel on the main page). Then if
a robot accesses that address, automatically block it's IP address...
> Ben Bucksch wrote:
>> Pete Collins wrote:
>>> Here is an example of a bot ignoring robots.txt just today:
>> You said you have the robots.txt in the bugs/ dir. Does
>> xprint.mozdev.org/robots.txt also disallow bugs/?
More information about the Project_owners