LSTSRV-L Archives

LISTSERV Site Administrators' Forum

LSTSRV-L
Options: Use Monospaced Font
Show Text Part by Default
Condense Mail Headers

Topic: [<< First] [< Prev] [Next >] [Last >>]

Parts/Attachments: text/plain (16 lines)
Print Reply
Mime-Version:
1.0
Content-Type:
text/plain; charset=US-ASCII
Date:
Tue, 29 May 2007 13:18:54 -0500
Content-Disposition:
inline
Reply-To:
LISTSERV site administrators' forum <[log in to unmask]>
Subject:
Re: Weird Authorization Message
From:
Andrew Bosch <[log in to unmask]>
In-Reply-To:
<[log in to unmask]>
Content-Transfer-Encoding:
quoted-printable
Sender:
LISTSERV site administrators' forum <[log in to unmask]>
 
The robots.txt syntax looks correct. You probably have an incorrect User-agent identifier. Try using "*" for the User-agent to see if the problem is really a web crawler; Google and other legitimate crawlers will stop. The "*" is a wild card for all user-agents.


>>> <[log in to unmask]> 5/29/2007 12:49 PM >>>
...
we've tried- perhaps we're doing it wrong. We put the following in
inetsrv.wwwroot

User-agent: googlebot
Disallow: /

Questions:
 am I doing the right thing?
 Will this stop the messages in the Listserv logs, or will they continue
 to show?

ATOM RSS1 RSS2