[Project_owners] Bot ignoring robots.txt

David Fraser davidf at sjsoft.com
Wed Oct 29 17:04:11 EST 2003

massey wrote:

> Block those bot's IP's. Nothing else will work. If they are so rude as 
> to ignore your robots.txt you should just worry about mozdev. There 
> must be something about this kind of terrorism in the 'homeland 
> security act' :-)

In fact this may be a nice way to get rid of them.
You could put a link in robots.txt to a particular URL which nobody else 
would access (linked from an invisible pixel on the main page). Then if 
a robot accesses that address, automatically block it's IP address...


> Ben Bucksch wrote:
>> Pete Collins wrote:
>>> Here is an example of a bot ignoring robots.txt just today:
>> You said you have the robots.txt in the bugs/ dir. Does 
>> xprint.mozdev.org/robots.txt also disallow bugs/?

More information about the Project_owners mailing list