LSTSRV-L Archives

LISTSERV Site Administrators' Forum

LSTSRV-L
Options: Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Topic: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Re: Google
Nate Eckstine <[log in to unmask]>
Wed, 31 Oct 2007 14:24:07 -0700
text/plain (40 lines) 
Try this.

Place a robots.txt file into the root of the web server with this text in it. Asterisks aren't in the file.

*************************************

# robots.txt for http://yourservernamehere

User-agent: *
Disallow: /

*************************************

This will stop all spiders and bots from searching anything on the server. The ones that respect the robots.txt file.

Search on google for robots.txt if you want to customize it.


--
Nate Eckstine (DoIT CityWide Webteam)

City of Seattle Department of Information Technology
700 Fifth Ave. Suite 2700
Seattle, WA 98124-4709   USA

Work (206) 733-9658
Fax (206) 684-0911

 


>>> Darren Evans-Young <[log in to unmask]> 10/31/2007 2:18 PM >>>
I just happened to notice in my logs a number X-LOGCHK entries. 48,000 to
be exact, since midnight. I traced the IP to Google. They are beating on
my server and I dont like it. What is the best way to stop it?
Others may want to check their logs too.

Darren Evans-Young
Univ of Alabama

ATOM RSS1 RSS2