[Project_owners] Bot ignoring robots.txt
pete at mozdev.org
Tue Oct 28 12:51:29 EST 2003
Ben Bucksch wrote:
> Pete Collins wrote:
>> Here is an example of a bot ignoring robots.txt just today:
> You said you have the robots.txt in the bugs/ dir. Does
> xprint.mozdev.org/robots.txt also disallow bugs/?
Chris, I think we need to generate a robots.txt ghost file like source.html.
I know what to do . . .
More information about the Project_owners