LSTSRV-L Archives

LISTSERV Site Administrators' Forum

LSTSRV-L

Options: Use Monospaced Font
Show Text Part by Default
Show All Mail Headers

Topic: [<< First] [< Prev] [Next >] [Last >>]

Print Reply
Michael Shannon <[log in to unmask]>
Thu, 6 May 2004 14:59:48 -0500
text/plain (54 lines)
>
>I wonder if switching the archives of your ACCOUNTABILITY list to
>private would prevent it from being included in Google's search
>results.

Yes, this will work as designed.  Since Private archives require a password
for access, Google's crawler won't be able to enter them.  There is a
solution for Public archives, though.  It involves the use of a robots.txt
file and/or META tags within the HTML code.

Information about using a robots.txt file can be seen on Google's own
website here <http://www.google.com/webmasters/faq.html#nocrawl>.
Robots.txt will work with compliant crawlers and gives you the opportunity
to instruct them as to which pages or directories you want indexed and
which ones you want left alone.  It's particularly useful in that one file
can be used to give instructions for your entire website.  Note that I
specify 'compliant crawlers'....some crawlers are non-compliant and won't
respect robots.txt.

You can also control compliant crawler behaviour at the individual page
level by using META tags.  You need to insert the tag between the page's
HEAD tags in the HTML code itself and looks something like this:

<meta name="robots" content="noindex;nofollow">

The parameter 'noindex' tells a compliant crawler to not index the page,
'nofollow' tells it to not follow any hyperlinks off of that page; they can
be used either individually or in combination.

For a list that has Public archives you can insert this tag into the HTML
by editing the WWW_INDEX template, which is found in each list's Template
Management area.  WWW_INDEX is used to build the list's first page, which
is also referenced from the Server Archives page and should help to
restrict crawling of the archives.  Two points to note are that this will
work with compliant crawlers only (as above) and it won't protect pages
further into the archives since it's only added to the listname.index page.

If you have a need to implement this across your entire site, or you have
too many lists to enable it individually, you can edit the WWW_INDEX page
found in the Server Management pages under the 'Customize Site-Wide Static
Web Pages and Banners' link.  Note that this will affect ALL lists; if you
only want to protect a few lists you'll have to edit them one-by-one as
described above.

--
Regards,
 Michael Shannon
 Senior Consultant, Asia Pacific Region           [log in to unmask]
 L-Soft international, Inc.                     http://www.lsoft.com

 Chazzozz!!

 LISTSERV (R) is a registered trademark of L-Soft.

ATOM RSS1 RSS2