Thread: Copy a website
View Single Post
  #12   Report Post  
Posted to rec.crafts.metalworking
Ignoramus3863 Ignoramus3863 is offline
external usenet poster
 
Posts: 5
Default Copy a website

On 2008-09-01, Richard J Kinch wrote:
Ignoramus3863 writes:

I actually have some smarts in the server that can tell a bot from a
human.


Not a bot attempting to look human. Just bots that advertise their
botness, by honest design or flawed hacking.


Yes, a bot trying to look like a human (ie supplying Referer and
browser-like User-Agent, I can still detec that it is a bot).

The way I detect is is that there is a hidden link that humans cannot
see, and cannot click, but bots would follow it. The hidden link is
not permitted by robots.txt, so it catches all non-compliant bots.
--
Due to extreme spam originating from Google Groups, and their inattention
to spammers, I and many others block all articles originating
from Google Groups. If you want your postings to be seen by
more readers you will need to find a different means of
posting on Usenet.
http://improve-usenet.org/