[Project_owners] Bot ignoring robots.txt

Pete Collins pete at mozdev.org
Tue Oct 28 12:51:29 EST 2003

Ben Bucksch wrote:
> Pete Collins wrote:
>> Here is an example of a bot ignoring robots.txt just today:
> You said you have the robots.txt in the bugs/ dir. Does 
> xprint.mozdev.org/robots.txt also disallow bugs/?

Chris, I think we need to generate a robots.txt ghost file like source.html.

I know what to do . . .


Pete Collins

More information about the Project_owners mailing list