I’ve had a situation creep up related to caching proxy servers used by (most notably .mil, .gov domains and large corporations like Intel).
These caching servers request say the index page and then the pre-fetch every link on the page and ignore all the rel=“nofollow” attributes of all the links. (all the various languages have anchor tags with rel=“nofollow”)
This is most problematic when a site has several installed languages. I.e. if a site has 8 installed languages and 12 links on a page (this would be very small for most sites) then there are 96 parallel GET requests being done to pre-fetch the data. And of course since cs-cart uses ajax on page loads (rather than per user action as it was designed/intended), there are actually a minimum of 2 more ajax page loads for every actual page requested. Hence there are now 8 X 12 X 3 (288 parallel requests for a single user looking at a merchant’s home page). Of course if there are lots of links then the load goes up even further.
So my question is: How does one tell the caching proxy servers NOT to pre-fetch data. It’s okay to cache what’s requested, but the pre-fetch can bring a server to its knees.
I can see hundreds of page requests for various languages for a single site all in process from the same IP address at the same time. I don’t want to reject their requests, but I’d like them to play nice.
Does anyone know of an Apache directive that will tell these proxies not to pre-fetch?