[Fasterfox] RE: Abusing sites with pre-fetching

[Tony Gentilcore]

PiK,

Thanks for the feedback.  Several other individuals have reported similar
experiences.

There is a bug filed for discussion here:
http://bugzilla.mozdev.org/show_bug.cgi?id=12051

I'm currently looking into a way to resolve this.  Right now I'm leaning
towards your robots.txt idea.

Tony

-----Original Message-----
From: Piotr Kubala [mailto:pik-emwac at sponsor.com.pl] 
Sent: Saturday, December 03, 2005 3:30 AM
To: fasterfox at mozdev.org
Subject: Abusing sites with pre-fetching

1. It is not true, that pre-fetching does not load dynamic content. It could
be
dynamic content with .htm extension too, and, in fact, there is.
2. In our site one user with pre-fetching behaves like hundreds of usual
users,
especially on full website map page - and we must pay for this.
3. Prefetcher behaves like a robot and should follow robot rules - both
written
in robots.txt and in <meta name="robots"> tag. I mean "nofollow" should stop
prefetcher. There are traps for rude robots - invisible links to pages
denied by
robots.txt. Usual user cannot see these links and does not visit these
traps.
Pre-fetcher try to access forbidden pages and, in effect, causes denial of
access for user.

I think pre-fetching SHOULD obey robot rules.
I would really appreciate indentifying prefetcher as special user-agent
(i.e.
"prefetcher") and possibility to create rules for it:

1. In robots.txt:

User-agent: Prefetcher
Disallow: ...
...

2. In META-tags:

<META name="prefetcher" content="..."

(index, noindex, follow, nofollow, cache, nocache)


I mean there are pages that pre-fetcher could prefetch but it could also
exist
pages forbidden to it. You cannot fix it with white- and blacklist in
prefetcher. You need a protocol webmasters could tell prefetcher how to not
be
rude.

PiK