[Project_owners] Bot ignoring robots.txt

Pete Collins pete at mozdev.org
Tue Oct 28 12:51:29 EST 2003


Ben Bucksch wrote:
> Pete Collins wrote:
> 
>> Here is an example of a bot ignoring robots.txt just today:
> 
> 
> You said you have the robots.txt in the bugs/ dir. Does 
> xprint.mozdev.org/robots.txt also disallow bugs/?

Chris, I think we need to generate a robots.txt ghost file like source.html.

I know what to do . . .

--pete

-- 
Pete Collins
www.mozdev.org
www.mozdevgroup.com




More information about the Project_owners mailing list