Description of the changes for release 1.8c of LISTSERV(R)
----------------------------------------------------------
Copyright 1996,1997 L-Soft international, Inc.
January 20th, 1997
*************************************************************************
************************** List owner's notes ***************************
*************************************************************************
The release notes for version 1.8c of LISTSERV(R) have been split into
two documents: list owner's notes and LISTSERV maintainer's notes. The
present list owner's notes describe all the changes that list owners need
to be aware of, although in some cases list owners may be impacted by
changes described in the LISTSERV maintainer's section. For instance, the
availability of a new systemwide option may prompt the LISTSERV
maintainer to make a change affecting all the lists on the LISTSERV host.
Thus, list owners are advised to read the maintainer's notes as well.
**************
* Highlights *
**************
- New database functions (available for both VM and non-VM versions) with
the same degree of functionality as the original VM database functions,
and a more user-friendly syntax.
- (non-VM) Web archive interface allowing users to browse list archives
interactively. This includes a WWW interface to the new database
functions with online help.
- "Hands off" bounce processing allowing for accurate, fully automated
removal of undeliverable addresses without the list owner having to see
a single bounce again. This is accomplished through the delivery of
individually addressed list FAQs tagged with a special marker that
uniquely identifies the recipient.
- Support for "notary" delivery error format (RFC1891 et seq), the
oncoming Internet standard for delivery errors.
- The INDEX subscription option is now available for non-VM systems and a
new GETPOST command (also available under VM) has been introduced to
facilitate the retrieval of individual postings.
- (non-VM) Enhanced, hierarchical file server functions with sub-catalog
support. The LISTSERV administrator can now create catalogs for
individual lists, delegating file management operations to individual
list owners.
- Improved spam detector and spamming counter-measures.
- Support for "super-lists", allowing the hierarchical organization of
related lists without any cross-posting or duplication.
- New SUBJECTHDR subscription option and "Subject-Tag=" list header
keyword to insert the list name (or any other word) in the subject of
all messages coming from the list, on a per-subscriber basis.
- Comprehensive, 40-page user's guide and shorter "quick start" guide now
available online.
- Per-subscriber daily message quota.
***********************************
* Documentation: New user's guide *
***********************************
A new, 40-page user's guide is now available online. A plain text version
is included with the 1.8c update, replacing the old "INFO GENINTRO" and
"INFO PRESENTATION" documents. A shorter, "quick start" guide is also
available as "INFO QUICKSTART". You can also FTP a copy of these new
manuals from FTP.LSOFT.COM, CD DOCUMENTS. The files are available in a
variety of word processing formats - check the FTP server for more
information.
*************************************
* Usability: New database functions *
*************************************
Version 1.8c introduces a new set of database functions, available on all
supported operating systems. These new database functions were designed
to be more functional and easier to use than the old (VM) implementation,
and include a WWW interface (coupled to the web archive interface). Under
VM, the old and new database functions coexist and share the same data
files, with slightly different syntax and capabilities. The new functions
will be faster in most cases and should be used whenever possible.
For more information about the new database functions, please see the
User's guide.
The following checklist gives a brief overview of the differences between
old and new database functions:
- Searches are initiated using the new SEARCH command, rather than by
submitting a DATABASE SEARCH job. The syntax of the SEARCH command is
virtually identical to that of the SEARCH clause in database jobs. This
command automatically produces an index and, where applicable, context
information showing the search words as they appear in the posting.
Context information is not available for keyword searches (eg SEARCH
xxx WHERE SENDER CONTAINS JACK), or for negative searches (SEARCH NOT
XYZ IN xxx - all the postings containing XYZ have actually been
excluded, so you will never find XYZ in them).
- There is a new operator, NEAR, which is the default for the "locate"
clause. In other words, SEARCH JOE SMITH IN XYZ-L looks for JOE and
SMITH close to each other. You can still use AND explicitly. Note that
'a NEAR b NEAR c' is defined as '(a NEAR b) AND (b NEAR c)', so this
new operator is not fully commutative.
- While (on VM) you can use the new SEARCH command to search non-list
databases, such as BITEARN or PEERS, you cannot currently retrieve
edited results with the GETPOST command for these databases. You should
continue to use the old database functions for non-list databases.
- The new SEARCH command will only display up to 100 hits. To get the
next 100 hits, use a SINCE clause or an item range qualifier: SEARCH
xxx IN XYZ.9182- will display all hits with
****************************************************************
* Usability: New GETPOST command and INDEX subscription option *
****************************************************************
The INDEX subscription option is now available on non-VM systems. In
addition, a new command, GETPOST, has been added to facilitate the
retrieval of individual messages from the archives. The syntax of the
GETPOST command is:
GETPOST listname itemrange1 <itemrange2 <...>>
See the User's guide for a more detailed description of the GETPOST
command.
Note to VM list owners: depending on the configuration of your host
server, you may notice a slight change in the format of the messages
returned to subscribers who use the INDEX subscription option. This is
because LISTSERV now uses the GETPOST command in conjunction with INDEX
subscriptions, as this is more efficient than the database jobs used in
the previous version.
********************************************
* Usability: "Hands off" bounce processing *
********************************************
Delivery error notices (commonly called "bounces") are probably the
single largest obstacle to the effective operation of mailing lists by
non-technical list owners. Any public list with more than a handful of
subscribers is bound to be plagued with a number of daily bounces, many
of which were written by and for technical experts and make absolutely no
sense to the Internet novice. It may not even be clear what message and
above all what susbcriber the bounce is referring to. More often than
not, novice list owners end up discarding all these bounces unread
(usually by sending them to a separate account whose mail is never read),
because they simply do not know what to do to correct the problem.
Unfortunately, this is not a good situation at all. Here is what happens
for each invalid address in a mailing list:
1. The server hosting the mailing list, not knowing that the address is
invalid, attempts to deliver the message. This first delivery costs
just as much as a delivery to any other address.
2. The target host must then create a delivery error notice and send it
back to the server hosting the mailing list. This is usually an
expensive operation that, when multiplied by hundreds of daily
occurrences, can slow down the target server. While this may be
"someone else's problem", on the Internet a "good neighbour" is
expected not to cause other sites to waste undue resources. Just like
in real life, people do not speak highly of "bad neighbours", and this
is a factor to consider if the lists are run for PR purposes.
3. The server hosting the list undergoes the resource cost of receiving
this delivery error, and routing it to LISTSERV for processing.
4. LISTSERV determines that this is a delivery error in a format that it
is unable to decode, and forwards it to the list owner for manual
processing.
5. The server hosting the list delivers the message to the list owner.
All told, the resource cost at the host site for a bad address is about
4-5 times higher than for a working address. This means that if a list
should accumulate 20% of bad addresses through lack of bounce processing,
the cost of running the list will be roughly doubled, and the capacity of
the server will be halved. Since bad addresses do not go away on their
own, the number of bad addresses just keeps increasing with time, and
what initially looked like a minor and perfectly acceptable resource
surcharge quickly turns into a serious capacity problem.
The solution to this problem is automatic bounce processing, where the
computer processes the bounces without human intervention. LISTSERV has
had automatic bounce processing since 1992, however this process is not
100% accurate, owing mainly to the lack of a standard for the format of
delivery error messages (there is now an IETF standards-track document at
the "Proposed Standard" stage, but to date only a handful of products
have implemented it).
To improve the accuracy and effectiveness of automatic bounce processing,
LISTSERV can now optionally take a more active role in the detection of
invalid addresses. In addition to reacting to incoming bounces, some of
which are unfortunately not intelligible to computer programs, LISTSERV
can also be directed to "probe" the mailing list at regular intervals by
sending an individually addressed FAQ (or similar document) to the
subscribers. This FAQ contains a special marker which uniquely identifies
the subscriber, making it possible for LISTSERV to determine that mail
for this particular subscriber is bouncing, even when the delivery error
message itself is otherwise totally unintelligible. Upon receipt of a
bounced probe message, LISTSERV will schedule the delivery of additional
probes to the failing address, at an appropriate and controlled rate, to
monitor the address over the period of time requested in the
"Auto-Delete=" keyword for the list. Thus, bounced probes are acted upon
with exactly the same degree of leniency as ordinary delivery errors. By
sending additional probes as the need arises, LISTSERV can effectively
monitor bouncing addresses even on low-volume lists where there may not
be more than a few messages per week.
To enable active monitoring, the list owner must take the following
steps:
1. Obtain permission from the LISTSERV administrator, if you were not
already using the "Renewal=" feature. Probes incur the same system
overhead as a renewal message, and are in fact implemented as an
option to the "Renewal=" keyword. Since this overhead can be quite
significant on some operating systems or configurations, you should
obtain permission from the LISTSERV administrator before proceeding,
especially if yours is a large list or if you know that the mail host
is overloaded. The LISTSERV administrator may also be able to advise
you on the appropriate frequency (typically monthly) for the list
probes.
2. Retrieve and customize the PROBE1 administrative message, replacing
its contents with a FAQ, mini-newsletter, or other appropriate
message. This step is optional; the default PROBE1 template is fully
functional, if uninteresting. The new &DAYSEQ(n) feature (described in
the updated list owner's guide) can be used to create "rotating"
collections of FAQs that are both more informative and less "annoying"
to the subscribers than a monthly FAQ that is always the same.
3. Add a "Renewal=" keyword to the list header, specifying the "Probe"
option. For instance,
* Renewal= Yes,Monthly,Probe
directs LISTSERV to probe the subscribers on a monthly basis. Please
refer to the list owner's manual for more information on the
"Renewal=" keyword.
These steps do not affect the delivery of bounces to the list owners.
That is, you can activate list probing without suppressing the delivery
of unparseable bounces to your account. If on the other hand you do want
to completely delegate bounce processing to LISTSERV, you should perform
the following step:
4. Change the "Auto-Delete=" keyword in the list header to read:
* Auto-Delete= Yes,Full-Auto
(as opposed to "Yes,Semi-Auto" or "Yes,Manual"). When used in
conjunction with active probing, this will direct LISTSERV to suppress
the delivery of all bounces to the list owners.
It is not possible to suppress bounce delivery without enabling active
probing, because the accuracy of passive bounce processing is too low.
While this varies widely from one system to another, or even from one
list to another on a given system, passive bounce processing typically
catches between 50% and 80% of bad addresses. Active probing, on the
other hand, should have an accuracy in excess of 99%.
Supported mail servers
----------------------
Because LISTSERV receives its incoming mail (including bounces and in
particular the special probe bounces) from the mail server servicing the
SMTP port of the machine on which LISTSERV is running, active probing may
not work with all mail servers, or may require you to upgrade to a
certain revision level, as follows:
- VM: active probing is supported with all versions of LMail(TM).
- VMS: active probing is supported with version 1.1a of LSMTP(TM), and
with all versions of MX that include support for the LISTSERV
interface. PMDF(R) users should create a dedicated domain for LISTSERV
(such as LISTSERV.XYZ.COM) and add a rewrite rule to redirect all
traffic for that host to the LSV channel. This also simplifies the
creation of new lists (with this setup, it is no longer necessary to
define PMDF aliases for the lists).
- unix: the current versions of sendmail are not compatible with the list
probe, however a patch is available from FTP.LSOFT.COM in the
LISTSERV/unix/Contrib directory to implement the necessary changes to
sendmail.
- Windows: active probing is supported with version 1.1a of LSMTP and
with all versions of the SMTP listener.
In mixed environments where one mail product processes incoming mail
while another is responsible for outgoing mail, the incoming mail server
is the one that determines whether active probing is supported.
Users of version 1.0a of LSMTP will need to upgrade to version 1.1a
before enabling active probing, regardless of the operating system they
are using. L-Soft has decided to release 1.1a as a free upgrade, however
you must still obtain a (free) 1.1a license key through normal sales
channels before installing the new version.
Performance considerations
--------------------------
Probing has the same impact on the host server as renewal, and the same
considerations apply. In particular:
- If the server (or network) is already overloaded, you should refrain
from using probing until suitable resources are available.
- You should never activate probing or renewal on a large list without
first checking with the LISTSERV administrator.
- You should avoid using "Renewal=" specifications which force LISTSERV
to process all renewals on a particular day. If you order LISTSERV to
probe 30,000 people on a specific day, it will comply, but service may
be impacted. Using less specific syntax options (such as "3-Monthly" as
opposed to a series of explicit dates) makes it possible for LISTSERV
to spread the probing activity over a period of time. Thus, instead of
probing 30,000 people every 30 days, it can instead probe 1,000 people
every day, with much reduced impact on the host system.
- The first time you activate probing or renewal for a large one-way
list, most of the subscribers are likely to be overdue for probing,
especially if the list was imported from a database via a bulk ADD. In
this case it is best to change the "Renewal=" keyword during a weekend
or other period of reduced activity.
***********************************************************************
* (non-VM) Usability: Enhanced file server functions and sub-catalogs *
***********************************************************************
The file server functions have been enhanced in version 1.8c to
facilitate the management of file catalogs (see maintainer's release
notes), allow users to manipulate files using the file naming syntaxes
they are used to (DOS/Windows, unix, VMS or VM), and allow for the
creation of "sub-catalogs" whose management can be delegated to
individual list owners.
Support for popular file naming conventions
-------------------------------------------
To make the file server functions more intuitive to non-technical users,
LISTSERV now accepts file specifications in a variety of popular formats.
This means users can use the file naming conventions of the operating
system they are used to, and do not need to learn a new way to refer to
files. For instance, the file NEXT.MEETING in the WG3 sub-catalog can be
accessed as:
- /wg3/next.meeting
- \wg3\next.meeting
- WG3:NEXT.MEETING
- WG3:[000000]NEXT.MEETING
- WG3:<000000>NEXT.MEETING
- NEXT MEETING WG3
Note that this file can also be accessed as NEXT.MEETING (without
specifying the WG3 directory) unless there are several files by that name
in the LISTSERV file system.
LISTSERV will reply using the same syntax convention, as the users would
expect it.
Overview - delegating file management authority
-----------------------------------------------
The sub-catalog enhancement allows the LISTSERV administrator to delegate
file management authority in a controlled and secure manner. Multiple
list owners can be given the authority to maintain their own sub-catalog
in a predefined directory. With the LISTSERV-ISP add on (under
development), a quota can be imposed on the directory in question.
The procedure works as follows:
1. The LISTSERV administrator creates the sub-catalog and identifies the
directory where the files will be stored, and the person(s) who will
be in charge of managing it ("catalog owners").
2. The catalog owners use the GET and PUT commands to update their
catalog and register new files in their directory. Each file has its
own GET and PUT file access codes, allowing the catalog owners to
further delegate the management of individual files to third parties
("file owners").
3. The file owners manage the files in question using the GET and PUT
commands. Authorized users can then retrieve the files using the GET
command.
Creating a sub-catalog
----------------------
To create a sub-catalog, the LISTSERV administrator edits the file called
SITE.CATALOG (or site.catalog under unix) in LISTSERV's main directory
(the directory where SYSTEM.CATALOG/system.catalog is located - refer to
the list owner's guide for more information). A sub-catalog is defined as
follows:
TEST.CATALOG /home/lists/xyz/test.catalog ALL [log in to unmask]
(1) (2) (3) (4) (5)
Notes:
(1) The name must end in '.CATALOG', but otherwise it can be anything. In
particular, there does not need to be a list by that name.
(2) This is the directory where ALL the files defined in the sub-catalog
will be stored. DO NOT USE LISTSERV'S MAIN DIRECTORY FOR THIS
PURPOSE! The catalog owner will be given FULL ACCESS to all the files
in this directory, so make sure to create a new, empty directory. If
the sub-catalog is being set up for a list owner, it may be a good
idea to put the list archives and the sub-catalog in the same
directory.
(3) A file name must be provided for the sub-catalog file itself. This
name, however, does not need to match (1).
(4) This file access code controls the authority to INDEX the
sub-catalog. This will also be the default GET access code for all
the files registered in the sub-catalog.
(5) This file access code defines the catalog owner(s) and default file
owner(s) for all the files in the sub-catalog.
Note that there is no need to reboot LISTSERV after updating the
SITE.CATALOG file. Also, bear in mind that you are responsible for the
OS-level security of the directory you create for the catalog. The file
access codes in SITE.CATALOG only affect operations that go through
LISTSERV; it is your responsibility to make sure that other users of the
computer are given the appropriate access level to any directory you
create for LISTSERV's purposes.
Updating the sub-catalog
------------------------
Once the sub-catalog is created, the catalog owner(s) can register new
files using the following procedure (in this example, it will be assumed
that the sub-catalog is called TEST.CATALOG):
1. Send a GET TEST.CATALOG command to LISTSERV (or, if the catalog is
brand new, start from an empty file).
2. Register new file(s) in the catalog (see below).
3. Use the PUT TEST.CATALOG PW=XXXX command to store the updated catalog.
Alternatively, if the catalog owner has an account on the LISTSERV host
system and write access to the directory associated with the sub-catalog,
the file can be edited directly. Note however that, in that case, the
LISTSERV-ISP quota system will be inoperative as it has no control over
disk accesses which do not go through LISTSERV itself.
The format of sub-catalogs is similar to that of SITE.CATALOG:
MY.FILE my.file ALL [log in to unmask]
(1) (2) (3) (4)
Notes:
(1) This defines the name of the file as seen by LISTSERV users. That is,
the command to retrieve the file will be 'GET /test/my.file' (or 'GET
TEST:MY.FILE', 'GET MY FILE TEST', and in most cases just 'GET
MY.FILE').
(2) This defines the name of the actual disk file where the contents of
MY.FILE will be stored. Normally, you should specify the same as (1),
or just an equal sign (LISTSERV will then substitute the name you
provided for (1)). However, in some cases you may want to make a
particular file available under multiple names. This can be done by
registering multiple files (ie multiple values for (1)), and using
the same (2) value every time.
(3) This file access code determines who can order the file through a GET
command. See the list owner's guide for more information.
(4) This file access code determines who can update the file with the PUT
command. See the list owner's guide for more information.
Note: (2) defaults to the value of (1), and (3) and (4) default to the
GET and PUT access codes of the sub-catalog itself, respectively. So, in
most cases a sub-catalog entry will be as simple as:
MY.FILE
Additionally, comment lines (starting with an asterisk) or blank lines
can be interspersed with file definitions. These comments will be echoed
when the sub-catalog is indexed (see below), in sequence with the file
definitions. For instance, your catalog could read:
*
* Files for the XYZ sub-project
*
XYZ.AGENDA
XYZ.BUDGET
XYZ.PROPOSAL-1
XYZ.PROPOSAL-2
Indexing the sub-catalog
------------------------
If TEST.CATALOG is defined as:
TEST.CATALOG /home/lists/xyz/test.catalog xxx [log in to unmask]
Any user who matches the 'xxx' file access code is authorized to issue an
INDEX TEST command to get a formatted version of the catalog. For
compatibility with older versions of LISTSERV, GET TEST.FILELIST will
produce the same results. If there is a mailing list called TEST, a list
of the archive files will also be appended automatically.
Sub-directories
---------------
Sub-directories may be created within a sub-catalog, allowing files to be
partitioned hierarchically up to an unlimited depth. See the list owner's
guide for more information.
*************************************************
* (non-VM) Usability: New web archive interface *
*************************************************
Version 1.8c introduces a WWW archive interface that allows users to
browse and search list archives (the LISTSERV maintainer controls which
lists are or are not visible through this interface). Postings can be
organized by date, by topic or by author, and a search function with
online help is provided. URLs in the text of the postings are displayed
as 'live' hyperlinks. LISTSERV's WWW interface has the following
advantages over "hypermail" style web archiving:
- The information on the web is always up to date. New postings are shown
as soon as they are received.
- The postings can be organized in the manner that best suits the reader:
by date, by author, by topic, with or without table of contents, with
or without showing the author, etc.
- Only one copy of the information is kept, and in particular there is no
need to create an individual HTML file for each posting. This design
allows the interface to scale up gracefully to lists with hundreds of
thousands of archived postings, which would otherwise require hundreds
of thousands of individual HTML files, wasting disk space (each file
takes up at least one disk block) and stressing the file system past
reasonable limits.
- The search forms can be used to create search requests matching all
postings in the last X days. The resulting URL can then be bookmarked
and reloaded on a regular basis.
- List owners can customize the main page for their lists without any
intervention by the LISTSERV maintainer, by updating one of the mail
templates for their list. The LISTSERV maintainer can customize common
pages and header/trailer HTML statements by updating system mail
templates.
To take advantage of this new interface, you must first ensure that the
"Notebook=" options for your list are compatible with the WWW interface.
In most cases, you will not have to do anything, but certain options are
incompatible with the use of the WWW interface and may need to be
changed:
- The archive frequency must be WEEKLY, MONTHLY or YEARLY. SEPARATE and
SINGLE notebooks are not supported. L-Soft generally recommends
converting lists with SINGLE notebooks to YEARLY unless there is a
compelling reason to have all the messages in exactly one file.
- For optimal performance, the archive frequency may need to be adjusted
to produce an "adequate" number of topics and messages in each archival
period. The definition of "adequate" depends on your users, the kind of
equipment they have, and how they connect to the Internet. As a rule,
home users will prefer a larger number of smaller archives whereas
office users with large screens and T1 or better connectivity will
tolerate a larger table of contents.
- The archives must be public as there is no userid/password control in
the web archive interface.
- On most systems, the directory in which your list archives are kept
must be specified in absolute rather than symbolic form, or the WWW
interface may not be able to access it. Symbolic form is when the
directory name is a single letter, for instance "Notebook= Yes,L,
Monthly,Public" ("L" is the directory specification). In most cases,
your list header will read "Notebook=Yes,E:\LISTS\XYZ-L,Monthly,Public"
and you will not have to worry about this.
You must then ask the LISTSERV maintainer to enable your list for the WWW
interface. This may require the installation of a web server and of the
WWW interface code itself. You can then modify the WWW_INDEX mail
template to customize the main archive page for your list. See the list
owner's guide for more information on customizing mail templates.
*************************************
* Usability: Improved spam detector *
*************************************
The spam detector has been improved in version 1.8c, as follows:
- The spam detector is now able to catch the first few copies of a new
spam (in most cases). Previously, dozens of copies (out of the
thousands sent by the spammer) would be let through before enough data
was collected to identify the posting as a spam.
- The number of spam alert messages sent to the LISTSERV maintainer has
been reduced to one per offender.
- The spam database is trimmed more aggressively, saving storage and
processing time for sites with lots of mailing lists.
- The algorithms have been generally refined and improved, making the
spam detector react more promptly and more accurately. New detection
techniques have been introduced.
There are two major changes that list owners need to be aware of: spam
quarantine and "anonymous" spam alerts.
Spam quarantine
---------------
One of the most arduous problems the spam detector has to face is the
accurate detection of the first few copies of the spam. When the first
copy reaches the first LISTSERV server worldwide, it is just a posting
like any other. It will take repeated occurrences of this same posting
for LISTSERV to realize that it is in fact a spam. However, it is
desirable to block this very first copy as well, and this can only be
accomplished by introducing a delay in the processing of "suspicious"
messages. This "quarantine" gives the spam detector some time to gather
the necessary evidence to determine if the message is a spam or not. The
default value is 10 minutes, and can be changed by adding:
* Loopcheck= Spam-Delay(xxx)
in the list header (the value is in minutes). The LISTSERV maintainer can
also change the system default by adding a SPAM_DELAY variable to the
LISTSERV configuration with the desired value (also in minutes). A value
of zero disables this new feature.
The default value of 10 minutes is adequate in most cases; it can be
lowered on fast, large, active servers and may need to be increased on
servers with chronical backlogs. Currently, LISTSERV determines whether a
message is suspicious or not based on the sender's posting history. This
however may be changed in future versions to further improve the
efficiency of the spam detector.
"Anonymous" spam alerts
-----------------------
On occasion, you may receive a spam alert from LISTSERV where the
offender's e-mail address is replaced with the word "anonymous". These
alerts are generated by new detection algorithms where, for various
reasons, it may sometimes be desirable to hide the identity of the
potential offender, usually because there is a fair chance that the
posting is in fact legitimate for the particular lists to which it was
posted (for instance because these lists were configured to tolerate a
high degree of cross-posting). In this case, information about the text
of the message may be released and ultimately lead to a spamming alert
that will block further copies of this same message, while the identity
of the poster remains hidden.
****************************************
* Usability: Spamming counter-measures *
****************************************
The spamming industry has changed radically since the inception of the
LISTSERV spam detector in 1Q95. While basement spam companies remain
mostly unchanged, a number of larger companies have emerged, with enough
financing to purchase their own high-end servers and T1 or better
Internet connectivity. Instead of targeting mailing lists and usenet
groups, these companies use their own hardware and leased lines to send
mass-mailings directly to the users' mailboxes. The LISTSERV spam
detector cannot filter these messages because they do not go through
LISTSERV at all. The e-mail addresses, however, are often obtained from
mailing lists. A number of changes have been introduced in version 1.8c
to make it more difficult for spammers to collect information about
existing lists or e-mail addresses. Other changes may need to be
introduced by the list owner, possibly after a discussion among the
members of the list; these are described in this section as well.
LIST GLOBAL now requires a search string
----------------------------------------
The most popular source of list information among spammers, at least
where LISTSERV is concerned, was the LIST GLOBAL command, which returned
a conveniently formatted list of all the public LISTSERV lists. With over
10,000 public lists (DEC96), this file is now over 1M in size and exceeds
the message size limits of most Internet providers. It is too large to be
printed and read sequentially, and most requestors do not realize that
the file they are ordering is going to be that large. By April 1996, the
postmaster at LISTSERV.NET was receiving several complaints a day about
the "bad surprise" of having had one's mailbox crammed with "this useless
junk" and losing important mail due to a "mailbox full" error. The LIST
GLOBAL command was then changed (initially only on LISTSERV.NET) to
require a search string of at least three characters, and since then only
about a couple dozen complaints have been received from users who wanted
the full listing. About half of them later turned out to be spammers. By
and large, the users reacted positively to this change, which has been
incorporated in version 1.8c. Note that the list of lists can also be
searched interactively through the CataList(SM) search engine, available
(among others) at:
http://www.lsoft.com/lists/listref.html
The CataList actually offers more information than LIST GLOBAL, being
based on the full LISTS database (of which LIST GLOBAL is an excerpt).
The CataList engine has built-in restrictions to prevent abusive use.
LISTS command now restricted to local users
-------------------------------------------
Another popular method for collecting information about available lists
was to send a LISTS commands to individual LISTSERV (and non-LISTSERV)
sites. While this required more programming work for the spammer, it also
gave better results as the LISTS command would also list local lists
("Confidential= Service"), which outnumber public lists 3:1. The LISTS
command has been changed in version 1.8c to only return information to
local users, ie users whose hostname matches one of the patterns in the
LOCAL configuration variable. Typically, if the server is running on a
machine called LISTSERV.XYZ.EDU, the LOCAL variable will have been
defined as *.XYZ.EDU. This allows any user in the XYZ.EDU domain to send
a LISTS command to the server, while preventing spammers from getting any
useful information. To clarify:
- This change has no impact on local users, who can still access the same
information as before. It may be necessary for the LISTSERV maintainer
to review the setting of the LOCAL configuration variable to make sure
that all local domains are properly identified.
- This change prevents non-local users from obtaining information about
local ("Confidential= Service") lists. This change will completely
protect local lists from spam (both direct and indirect) as there will
be no way for the spammers to get information about them.
- "Confidential= Yes" lists remain completely hidden as before.
- Non-local users must now use the LIST GLOBAL command or the CataList to
search for public lists of interest.
The LIST GLOBAL command was also changed so that you can no longer ask
for a list of all the lists hosted on a particular server, as this would
defeat the purpose of the LISTS command change.
"Review=" now defaults to "Private"
-----------------------------------
The default setting for the "Review=" command was changed from "Public"
to "Private", to make it more difficult for spammers to retrieve
subscriber e-mail addresses. The list owner may of course override this
change by explicitly adding a "Review=" keyword to the list header.
L-Soft recommends reviewing the setting of the "Review=" keyword to
determine whether it adequately meets your spam-protection needs. If the
list is confidential (or at least local, ie "Confidential= Service"),
"Review= Private" ought to be sufficient to prevent the addresses from
being "stolen" by spammers, as it is very difficult to find out about the
existence of the list. For a public list, "Review= Owner" might be more
appropriate.
Compilation copyright
---------------------
The LIST GLOBAL command now inserts a copyright statement at the top of
the returned listing, as follows:
-------------------------------------------------------------------------
Copyright 1996 L-Soft international, Inc.
L-Soft international, Inc. owns the copyright to this compilation of
Internet mailing lists (the "Compilation") and hereby grants you the
right to copy the enclosed information for the sole purpose of
identifying, locating and subscribing to mailing lists of interest. Any
other usages of the Compilation, including, without limitation,
solicitation, tele-marketing, "spamming", "mail-bombing" and "spoofing"
are strictly prohibited.
-------------------------------------------------------------------------
A "compilation copyright" is a copyright that applies exclusively to the
compilation (directory, list, etc) on which the copyright is being
asserted, as opposed to the works listed in the compilation. For
instance, you could decide to make a list of all the records published by
pop singers born in Alaska from 1963 to 1964. You would then be allowed
to assert a copyright on this compilation, so that nobody could use it
without your permission. This does not, of course, mean that you own the
rights to any of the records in question, or that they may not appear in
other people's compilations of pop records. It just means that people
cannot copy your compilation without your permission. Similarly, L-Soft
does not claim any ownership rights to any of the lists present in LIST
GLOBAL. The copyright applies solely to the information that the server
is returning when you send the LIST GLOBAL command.
Without a copyright statement, anyone was able to use the LIST GLOBAL
data for any purpose, including spamming, mail-bombing, subscription
spoofing, and so forth. Many of these activities, while harmful to the
Internet community, are arguably not unlawful under many jurisdictions.
At any rate, the use of the LIST GLOBAL data in conjunction with these
activities would not, in itself, have broken any laws. Consequently,
people would be allowed to sell the LIST GLOBAL data in a variety of
formats adapted to, say, popular spamming programs, and this would have
constituted a perfectly legal (and thus unstoppable) activity. The
copyright statement makes it possible to challenge the use of the LIST
GLOBAL data independently from the legality of activities such as
spamming or subscription spoofing.
Conclusion
----------
The spamming counter-measures introduced in version 1.8c will provide the
following benefits to LISTSERV users:
- Local ("Confidential= Service") lists will become totally immune to
both direct and traditional spamming, as it will be impossible for
spammers to find out about their existence through programmable
methods. Note that local lists whose existence became known to spammers
prior to the installation of 1.8c may remain exposed. However, all new
lists will be protected.
- Public lists (especially those created after the installation of 1.8c)
will become much more resistant to spamming attacks, as it will be very
difficult for the spammers to gain knowledge of their existence. The
increased "Review=" protection will also make it harder for them to
collect membership data.
The dynamics of the spamming industry are simple: reach as many people as
possible, at a minimal cost. Direct mailings were not attempted until the
LISTSERV spam filter and the usenet "cancelbots" significantly reduced
the number of people who could be reached with the much cheaper list/news
spamming methods. The counter-measures introduced in version 1.8c will
significantly increase the spammers' costs for collecting membership
lists or other information about LISTSERV lists (and especially local
lists), making other targets comparatively attractive. Spamming is only
cost effective when the cost of acquisition of a particular e-mail
address is significantly less than with traditional (mundane) mailing
list brokering. Hiring people to surf the web in search of local lists,
then pose as an interested party and try to talk the list owner into
being allowed to join the list is a slow, unreliable and expensive
process - there are over 30,000 local LISTSERV lists with an average of
only 75 subscribers each. Running a simple program on a standard full
usenet feed, on the other hand, is and will remain a very cost effective
way to acquire massive amounts of e-mail addresses in a very short time.
It is unlikely that spammers will hire armies of "list investigators" to
engage in arguably fraudulous misrepresentations to collect a couple
million e-mail addresses when they can get better results legally with a
$2000 PC reading usenet news 24h a day.
***************************************************************
* Usability: MIME digests and other MIME-related enhancements *
***************************************************************
Version 1.8c introduces support for MIME digests, that is, list digests
formatted according to the specifications of the Internet MIME standards
(RFC1521/1522 et seq). These digests coexist with the traditional
(RFC1153) LISTSERV digests. Individual subscribers can select which type
of digest they wish to receive, and the list owner can set a default for
the list. In addition, when the list owner did not indicate any
preference through the "Default-Option=" keyword, LISTSERV automatically
activates MIME digests for SUBSCRIBE requests received from users with
MIME-capable mail readers. This effectively makes MIME digests the
default option while ensuring that users who do not have a MIME-capable
mail reader are not sent a digest that they cannot decode in a convenient
way.
The advantages of MIME digests over traditional ones are:
- Full support for all MIME enhancements: national/8-bit characters, file
transmission, multimedia attachments, etc.
- Advanced mail client support (in some mail products) for the digest
function. Typically, opening the digest will display a table of
contents showing all the available messages in the digest. Clicking on
one of the messages will then open it in a separate window. More
advanced functions (threading, sorting, etc) may be available in some
mail clients.
The disadvantages are:
- The MIME digest format is harder to read when using a mail client which
has no special support for MIME digests. In this case, it is usually
preferable to switch back to the traditional digest format (see below).
- Some users actually dislike the way their mail client provides specific
support for mail digests. Some clients, for instance, immediately burst
MIME digests into a collection of messages, which are entered
individually into the user's in-box. While this may seem to be a good
idea at first, it means that discarding a digest may actually require
hundreds of messages to be discarded individually.
Users who would rather continue to receive traditional digests can send a
SET * NOMIME command to LISTSERV to indicate their preference for
traditional digests for all the lists they are currently subscribed to.
This command does not have any effect on regular (MAIL) or INDEX
subscriptions and thus can be safely wildcarded. Similarly, users who
subscribed prior to the installation of 1.8c but would rather receive
MIME digests can do so with a SET * MIME command. Again, this only
affects digest subscriptions and will not cause non-digest subscriptions
to switch to digested mode.
List owners can add "Default-Options= MIME" to the list header to select
MIME digests as the default digest option for all new subscribers. As
usual, this keyword has no effect on existing subscribers (use SET
listname MIME FOR *@* to change existing subscriptions). Please note that
the MIME option only indicates a preference in the event that the users
should switch to digest mode on their own. Use "Default-Options=
MIME,DIGEST" to add all new subscribers with MIME digests (as opposed to
MAIL or INDEX). Conversely, "Default-Options= NOMIME" indicates that MIME
digests should only be activated when explicitly requested by the user
(through the SET listname MIME command).
While currently its only effect is to indicate a preference for MIME vs
traditional digests, the MIME subscription option may, in a future
version, control new MIME-related functionality.
Finally, MIME fields are now always written to the list archives, even
when "Notebook-Header= Short" is in effect.
****************************************
* Usability: Support for "super-lists" *
****************************************
In order to facilitate the organization and management of hierarchical
mailing list structures (such as lists mirroring the internal
organization of a business unit), version 1.8c introduces support for
"super-lists" ("super" as in the opposite of a "sub-list"). A super-list
is a container list that automatically includes all the subscribers in a
pre-defined set of sub-lists, recursively. Super-lists can be created
hierarchically to any depth.
The LISTSERV administrator creates a super-list in the same manner as any
other list, and inserts a "Sub-lists=" keyword in the list header to
identify the list as a super-list and provide a list of the sub-lists
whose membership information should be inherited automatically. All the
sub-lists in question must be on the same machine as the super-list. For
security reasons, only the LISTSERV administrator is allowed to add or
modify the "Sub-lists=" keyword.
In most respects, a super-list behaves exactly like an ordinary list. It
has its own list owner, its own list archives, its own set of list header
keywords, and so forth. It is even possible to subscribe to the
super-list directly, assuming the "Subscription=" keyword allows it. The
only difference between a super-list and a regular list is what happens
when you post to it. With the super-list, the membership of all the
sub-lists is added (recursively) to the list of people who have
subscribed to the super-list itself, and duplicates are suppressed. The
posting is then processed normally, archived based on the "Notebook="
keyword in the super-list, and so forth.
As noted above, users can subscribe to the super-list directly, and this
is in fact an important aspect of the operation of super-lists. If you
are subscribed to the super-list itself, the subscription options used to
deliver "super-messages" to you are taken from your subscription to the
super-list, just like with any other list. For instance, if you are
subscribed to the super-list with the NOMAIL option, you will not receive
any super-message, even if you are also subscribed to one of the
sub-lists without the NOMAIL option. By subscribing directly to the
super-list and setting the NOMAIL option, you are indicating that you do
not wish to receive any of the messages posted to the super-list and
explicitly overriding any subscription options in the sub-lists.
When you are subscribed to multiple sub-lists, but not to the super-list
itself, LISTSERV uses the following rules to determine which subscription
options should be used for super-messages:
1. NOMAIL subscriptions are ignored. You will get the super-message if
you have an active (non-NOMAIL) subscription to at least one sub-list.
The rationale for this rule is that posting to the super-list should
be equivalent to posting to each and every sub-list, minus the
duplicates. When posting to all the sub-lists, a single non-NOMAIL
subscription would be sufficient for the user to receive the message,
and consequently the message should be delivered when posted to the
super-list. The only way not to receive postings from the super-list
is to subscribe to the super-list directly and set yourself to NOMAIL.
2. The DIGEST and INDEX options are ignored and internally converted to
MAIL. The rationale is as follows:
- In most cases, super-lists will be used for administrative messages
or for announcements, rather than for active, ongoing discussions,
which would typically be carried out on the smaller sub-lists. Thus,
it seems improper to inherit the DIGEST or INDEX options from a
high-volume discussion sub-list and apply it to a low-volume
announcement super-list.
- This rule can be applied consistently regardless of the way the
super-list is configured. If the DIGEST or INDEX options were to be
retained when possible, there would still be cases where they would
have to be changed to NOMAIL because the super-list does not have
list archives or does not allow digests. This lack of consistency
would make it more difficult for non-technical users to work with
super-lists.
- Similarly, super-lists will normally be deployed to avoid duplicate
messages, ie in situations where users are subscribed to multiple
sub-lists, possibly with different options for each list. Inheriting
the DIGEST and INDEX options would require additional rules to
control the behaviour of the super-list in the presence of
conflicting subscription options (sub-list A set to DIGEST and
sub-list B set to INDEX or MAIL).
To receive super-messages in DIGEST or INDEX format, the users must
subscribe to the super-list directly and set the appropriate option.
Topics, if defined, are evaluated on a per-list basis. That is, for every
sub-list (and for the super-list as well, if appropriate), LISTSERV
determines whether the topic of the message is one that the user wanted
to see. If not, it acts as if the user were not subscribed to this
particular list. This mechanism works very well if all the sub-lists have
the same set of topics (or at least a well-defined set of common topics).
Collections of sub-lists with widely different sets of topics should be
avoided as there is no reasonable way to make super-list topics work in
this context.
Please refer to the 1.8c list owner's guide for more information about
super-lists.
*************************************************
* Usability: New SUBJECTHDR subscription option *
*************************************************
The new SUBJECTHDR subscription option directs LISTSERV to add the name
of the list in the "Subject:" field of every posting coming from the
list:
Subject: [XYZ-L] Question about ink mixing
With some mail programs, this can make it much easier to sort messages
coming from the list, for instance by using special filters to save all
the messages from a particular list to a designated folder. Replies are
handled automatically to avoid inserting multiple copies of the list
name.
Like all other subscription options, SUBJECTHDR can be turned on or off
by individual subscribers, or by the list owner on behalf of any (or all)
subscribers. The list owner can also define a default value using the
"Default-Options=" keyword.
By default, LISTSERV will insert the name of the list in the subject
field. In some cases, this may not be the most appropriate solution,
especially if the list name is very long. The list owner can change the
text of the insert using the "Subject-Tag=" keyword:
* Subject-Tag= FE
This is particularly useful for groups of related mailing lists.
**********************************************
* Usability: Support for parallel moderation *
**********************************************
In some cases, it may be useful to have a team of moderators working "in
parallel" on the same messages, for instance when it is critical that the
messages be approved or rejected as soon as possible. In this case, it is
counter-productive to have LISTSERV assign each message to a moderator in
the usual round-robin fashion; instead, all the moderators need to be
sent a copy of the message, and each of them should be able to approve
any message, without however causing duplicate copies to be posted to the
list. This is now possible, using the following keyword syntax:
* Send= Editor,Hold
* Moderator= All,moderator1,moderator2,...
If the moderator list starts with the word "All", approval requests for
incoming messages are sent to all the listed moderators. Any moderator
can then approve the message using the normal OK confirmation procedure.
If several moderators attempt to approve the same message, the first one
will succeed and subsequent requests will fail, stating that the message
was not found (because it was deleted after being approved and posted).
It is also possible to use "Moderator= All" without "Send= ...,Hold",
allowing the moderators to edit the messages before reposting them. In
this case, however, a separate synchronization mechanism needs to be used
to avoid duplicates, as two moderators are unlikely to make exactly the
same edits to the message. Even if LISTSERV were able to identify the two
submissions as being the same message, it would not know which one to
choose over the other.
**************************************************
* Usability: Support for anonymous subscriptions *
**************************************************
The SUBSCRIBE command has been enhanced to support the following syntax:
SUBSCRIBE listname ANONYMOUS
(as usual, the command is not case sensitive)
This indicates that the subscriber wishes to join the list anonymously,
ie without specifying a name. The CONCEAL subscription option is
automatically set, granting the subscriber the maximal level of
protection available.
An existing subscription can be made anonymous by sending a SUBSCRIBE
listname ANONYMOUS command. This will both delete the name from the list
and set the CONCEAL option. An anonymous subscription can be made non
anonymous by sending a SET listname NOCONCEAL command and a SUBSCRIBE
command with the desired name.
In addition, when a list operates with "Default-Options= CONCEAL",
LISTSERV no longer requires or expects a name to be specified on the
SUBSCRIBE command. If a name is provided, it is accepted and entered in
the list file, as before. If, however, no name was supplied either on the
SUBSCRIBE command or in the mail headers, LISTSERV accepts the
subscription (instead of replying with a description of the expected
syntax) and makes it anonymous.
*****************************************************
* Usability: Per-subscriber daily message threshold *
*****************************************************
The syntax of the "Daily-Threshold=" keyword has been extended to allow
the specification of a second, per-subscriber daily message threshold.
For instance,
* Daily-Threshold= 100,10
indicates that subscribers are allowed to post up to 10 messages per day
each. The 11th and following messages are returned to the sender, who
will have to resend them on the following day in order to have them
posted. The list owner and editors are exempt from this quota. The
default per-subscriber message quota is infinite.
The implementation of the global daily threshold (100 messages in the
present example) is unchanged from the previous version: the 101th and
following messages are queued instead of being returned, and will be
automatically reprocessed when the list owner sends a FREE command. The
decision to return messages exceeding the per-user threshold (instead of
queuing them) was based on the design objectives for this new function,
which were to provide a simple mechanism to educate outspoken subscribers
and make them understand the need to keep their contributions to a
reasonable volume. Queuing the messages would have encouraged a "It will
go through when it goes through" attitude and ultimately resulted in a
situation where the postings from hyperactive subscribers are distributed
several days after they are written, confusing the other subscribers and
raising questions about unexplained mail delays and problems with the
host system. Returning the message while requiring a retransmission, on
the other hand, provides instant feedback to the hyperactive user and
requires a conscious choice as to which postings should be resent the
next day.
*****************************************
* Usability: Miscellaneous enhancements *
*****************************************
- By default, the bottom banner is no longer appended to every single
message when composing a digest. Instead, a single copy is included in
the digest preamble. Please note that some digestification programs and
mail readers may not show the digest preamble at all, and that frequent
readers may skip the preamble out of habit. In general, there is no
guarantee that the bottom banner will be read at all unless it is
included on each and every message in the digest. To revert to the
previous behaviour (bottom banner on each and every message), add the
special "Bottom_banner" option to your "Digest=" keyword, as in:
* Digest= Yes,Monthly,Bottom_banner
The top banner is always included on each and every message as it is
normally used to insert a one-line copyright statement.
- LISTSERV now decodes MIME messages sent to the LISTSERV address (but
not those sent to the list addresses) before processing them. In most
cases, this will allow correct processing of LISTSERV requests which
used to be rejected by 1.8b, for instance because they were base64
encoded. In some cases, however, it can lead to LISTSERV rejecting MIME
messages with proprietary text-like formats when the requests
previously happened to work. LISTSERV requests should be sent in plain
text format whenever possible.
- "OK"-style confirmation messages can now be tailored, on a list per
list basis, using the new CONFIRM1 mail template.
- LISTSERV now keeps track of subscription dates. This information is
displayed by the QUERY command, when it is available. Since users who
subscribed before the installation of 1.8c do not have any known
subscription date, this information is not always present.
- When parsing list headers, LISTSERV now recognizes e-mail addresses and
processes them using a slightly different parser, which preserves the
case of the left-hand side of the address and, for BITNET sites,
removes any superfluous '.BITNET' suffix on the right-hand side. Thus,
it is no longer necessary to use double quotes to register a lower-case
address in the list header.
- The mail template processor now supports HTML-like variable closure, in
addition to the traditional LISTSERV closure (both methods are
supported concurrently; there is no need to select one over the other).
Here are examples with traditional and HTML closure:
Traditional: For more information, please send mail to &EMAIL or call
&PHONE.
HTML: For more information, please send mail to &EMAIL; or call
&PHONE;.
Previously, HTML writers who used HTML closure conventions would not
get the expected results. This change makes it easier for webmasters to
get the desired results the first time.
- New mail template variables: &LITE, &ISODATE, &DAYSEQ(n). &LITE has the
value 1 when running the new LISTSERV Lite product, and 0 otherwise.
This variable can be used to write generic templates that account for
the differences between the two products. &ISODATE returns today's date
in ISO format, ie yyyy-mm-dd. &DAYSEQ(n) is used to create FAQ
templates with rotating topics and is described in the 1.8c list
owner's guide.
- The "Renewal=" keyword now supports "xx-Daily" as a valid interval
specification. While the use of intervals of less than a week is and
remains inadvisable, FAQ templates with rotating topics may require the
selection of a very precise renewal interval (for congruence purposes),
which was not possibly with "xx-Weekly" granularity. Please refer to
the list owner's guide for a discussion of rotating FAQ support.
- The QUERY ... WITH command now supports topic selection. Examples:
QUERY XYZ-L FOR *@* WITH TOPICS: ADMIN FORUM
-> Shows all members subscribed to both ADMIN and FORUM topics.
QUERY XYZ-L FOR *@* WITH TOPICS: -ADMIN
-> Shows all members not subscribed to the ADMIN topic.
QUERY XYZ-L FOR *@* WITH TOPICS: ADMIN -TEST
-> Shows all members subscribed to the ADMIN topic, but not to the TEST
topic.
- The "List-Address=" keyword now allows the 'username' part (ie the name
of the list) to be redefined. Previously, only the host name could be
changed. It is important to note that the only effect of the
"List-Address=" keyword is to change the way the list identifies
itself in list postings, command replies, etc. It does not instruct the
mail system to create forwarding entries to support the new name, nor
does it establish the specified name as an alias for the list (use
"List-ID=" for this purpose). In general, you should not use this
keyword without first consulting with the LISTSERV maintainer.
- LISTSERV now requires confirmation (through the "OK" mechanism) when
commands are sent through a WWW browser, even if they apply to a list
whose security level or "Subscription=" keyword does not require
confirmation. This change was made both to avoid abuse and because many
occasional WWW users do not know their e-mail address or enter it
incorrectly, whereas people who use e-mail regularly do of course have
a working e-mail address in their configuration.
-------------------------------------------------------------------------
CataList is a service mark of L-Soft international, Inc.
LISTSERV is a registered trademark licensed to L-Soft international, Inc.
LSMTP is a trademark of L-Soft international, Inc.
LMAIL and L-SOFT are trademarks of L-Soft international.
PMDF is a registered trademark of Innosoft International, Inc.
Unix is a registered trademark of X/Open Company Limited.
DEC and VMS are trademarks of Digital Equipment Corporation.
Windows is a trademark of Microsoft corporation.
All other trademarks, both marked and not marked, are the property of
their respective owners.
-------------------------------------------------------------------------
|
