[Project_owners] Bot ignoring robots.txt

David Fraser davidf at sjsoft.com
Wed Oct 29 17:04:11 EST 2003


massey wrote:

> Block those bot's IP's. Nothing else will work. If they are so rude as 
> to ignore your robots.txt you should just worry about mozdev. There 
> must be something about this kind of terrorism in the 'homeland 
> security act' :-)

In fact this may be a nice way to get rid of them.
You could put a link in robots.txt to a particular URL which nobody else 
would access (linked from an invisible pixel on the main page). Then if 
a robot accesses that address, automatically block it's IP address...

David

>
> Ben Bucksch wrote:
>
>> Pete Collins wrote:
>>
>>> Here is an example of a bot ignoring robots.txt just today:
>>
>>
>>
>> You said you have the robots.txt in the bugs/ dir. Does 
>> xprint.mozdev.org/robots.txt also disallow bugs/?
>>




More information about the Project_owners mailing list