Since we seem to have two threads running in parallel on this topic, Golias's "Avoiding Spiders" thread and this, I want to draw attention to Rose's most recent post "across the way": "There are plenty of bad spiders out there that ignore robot tags. The only foolproof way to keep them out is to require a login." (I am certain Rose knows a whole lot more about such matters than I do....)
Clive
|