[Project_owners] Bot ignoring robots.txt
pete at mozdev.org
Wed Oct 29 10:01:37 EST 2003
> Block those bot's IP's. Nothing else will work. If they are so rude as
> to ignore your robots.txt you should just worry about mozdev. There must
> be something about this kind of terrorism in the 'homeland security act'
Tell me about it.
Blocking IP's is my last resort. It's like sticking fingers in a dam,
eventually it gets unmanageable.
The box seems a lot more responsive. So I think the changes I made
yesterday are having some affect.
More information about the Project_owners