Wed, 31 Oct 2007 14:33:46 -0700
|
My understanding of the file is that it has to be in the root directory of the webserver.
--
Nate Eckstine (DoIT)
City of Seattle Department of Information Technology
700 Fifth Ave. Suite 2700
Seattle, WA 98124-4709 USA
Work (206) 733-9658
Fax (206) 684-0911
>>> Darren Evans-Young <[log in to unmask]> 10/31/2007 2:30 PM >>>
On Wed, 31 Oct 2007, Nate Eckstine wrote:
>Try this.
>
>Place a robots.txt file into the root of the web server with this text in it. Asterisks aren't in the file.
>
>*************************************
>
># robots.txt for http://yourservernamehere
>
>User-agent: *
>Disallow: /
>
>*************************************
>
>This will stop all spiders and bots from searching anything on the server. The ones that respect the robots.txt file.
>
>Search on google for robots.txt if you want to customize it.
>
>
>--
>Nate Eckstine (DoIT CityWide Webteam)
I figured something like that would work. But I dont have access to the
web server doc root. Would putting this in Listserv's archives directory
prevent their bot from searching just that directory (and down)?
Darren
|
|
|