[. . .] >OK, the list is going to be fairly big in the future and we have to be sure >the auto-delete process is functioning properly. I have been sending about >20 mails to the TESTLIS with the above mentioned configuration. But none of >te bougus mail-addresses has been deleted..... This is strange... Help is >needed... The short answer is that in order for LISTSERV to process a bounce automatically the bounce must be in a format that LISTSERV understands. There are two or three formats that LISTSERV understands (most notably that described in RFC 1894) but many mailers don't use those formats. If the bogus addresses you are using to test the auto-delete function are all from the same domain then it's likely that the mailer responsible for handling mail for that domain is such a mailer. I'll include more information on the subject of how LISTSERV handles bounces at the end of this message. >A second issue is if it is advisable and/or possible to set the FROM >address-space with the lists own e-mail address?!?! http://www.lsoft.com/manuals/lsv-faq.stm#5.14 addresses this issue. LISTSERV won't make changes to the From: address, so the answer is basically no. The usual strategy is to set up a generic email address ([log in to unmask] or whatever) and use that as the return address in your email client when you send the mailing. Information on how LISTSERV handles bounces ------------------------------------------- One of the problems that everyone who runs a mailing list has to deal with is one of bad addresses; if your mailing list has a subscriber whose email address is invalid then every message that goes out results in a nondelivery notification (sometimes referred to as an 'NDN' or 'bounce') being generated, which for a high-traffic mailing list can result in a very large number of bounces. These bounces have to be dealt with somehow, and generally the responsibility for doing so falls on the person or people running the mailing list. Fortunately, LISTSERV has some tools which allow it to take some of the burden off of the listowner; unfortunately, none of these tools can be 100% effective. After reading this document you should have a better understanding of how LISTSERV handles nondelivery notifications and some of the issues involved. There are some things that are glossed over and simplified, however, so you may want to also look at some of the resources referred to in the final section of this document to get a fuller perspective. A Simple SMTP Transaction, and RFC821's "MAIL FROM:" vs. RFC822's "From:" ------------------------------------------------------------------------- Before discussing what happens when a message is undeliverable, though, let's look at what happens when a message is delivered successfully. There are several Internet standards dealing with getting email from one place to another. One of the most important is RFC 821, titled "Simple Mail Transfer Protocol" (or SMTP for short). This lays out how one mail server connects to another, what commands can be used, and what responses are allowed and what they mean. Generally the mail server that wants to deliver the message connects to port 25 of the receiving mail server and starts sending commands. (If you are not sure exactly what this means that's fine.) Here is a sample conversation between two mail servers; items preceded by '>>>' are being sent by the mail server that's sending the message, and those preceded by '<<<' are being sent by the receiving server. <<< 220 home.ease.lsoft.com (LSMTP for Windows NT v1.1b) ESMTP server ready >>> HELO earthlink.net <<< 250 home.ease.lsoft.com says hello to earthlink.net >>> MAIL FROM:<[log in to unmask]> <<< 250 Ok >>> RCPT TO:<[log in to unmask]> <<< 250 Ok >>> DATA <<< 354 Start mail input; end with <CRLF>.<CRLF> >>> From: [log in to unmask] >>> To: [log in to unmask] >>> Date: Fri, 18 Feb 2000 17:34:21 -0500 >>> >>> Thanks a lot! >>> . <<< 250 Ok, message queued (73 bytes) >>> QUIT <<< 221 home.ease.lsoft.com says goodbye Here's what happened above. The receiving SMTP server detected a connection attempt and announced itself ready to receive commands. The sending SMTP server identified itself as earthlink.net and the receiving SMTP server acknowledged that. Next the sending SMTP server issued the MAIL command (indicating it has mail to send) followed by the FROM:<return-path-address>. This return-address is extremely important because it instructs the receiving SMTP server where to send back any NDNs that are generated as a result of the transaction. Note that this may or may not be the same as the From: address that appears in the headers. (More on this later.) Next the sending SMTP server says who the message is being sent to (in the RCPT TO: line). If there were more than one recipient each recipient would get its own RCPT TO: line, but in this case there's only one. Finally the DATA command is issued. This tells the receiving SMTP server that the following lines are the actual message that's being sent. First any header lines the sending SMTP server wants to specify are sent, then a blank line, then the body of the message ("Thanks a lot!"). Finally a line consisting of a single period is sent, and that tells the receiving SMTP server that the DATA section is over. The receiving SMTP server reports back that the message was accepted, and the two servers say goodbye to each other and close the connection. Some things to notice: In this example the MAIL FROM: address was the same as the "From:" address given in the DATA section, and the RCPT TO: address was the same as in the To:. Neither of these identities has to be the case in general. For instance, if you send a message and "Bcc:" someone, they will definately get their own "RCPT TO:" line, but they will not appear in the message headers at all. Similarly, when a message is distributed by a LISTSERV-hosted mailing list, the From: address will be the address of the person who originally sent the message, but the MAIL FROM: will be something like "[log in to unmask]" (or wherever your mailing list is hosted). This makes it so that subscribers to the mailing list can see who sent the message, but any bounces that are generated by the mailing go to the owner- address rather than the original sender of the message (who probably would not be in a position to do anything about them). A metaphor may be of use here. When someone sends you a physical piece of mail, it comes in an envelope which includes the address of the person who sent the message and your address. When you receive the message you open and discard the envelope and read the letter itself. The letter may also contain information on who the message is from and who it is to, and this information may differ from the information that was on the envelope. If the letter can't be delivered for some reason it is stamped 'Return To Sender' (or similar) and returned to the sender's address on the envelope. Internet mail standards are based on a similar idea. The 'MAIL FROM:' and 'RCPT TO:' addresses are equivelent to the addresses on the envelope (and in fact you may sometimes see them referred to collectively as the 'SMTP envelope' or similar). The actual message headers and the body of the message are equivelent to the letter itself, and they should not be referred to at all when the message is being delivered (though sometimes broken mail servers do make reference to them). Throughout this document I will make a distinction between the address used in the MAIL FROM: and the one used in the From: of the same message. As the latter is specified in RFC 822, "Standard For The Format Of ARPA Internet Text Messages", I will sometimes refer to the "RFC 822 'From:' address" in distinction to the "RFC 821 'MAIL FROM:' address". LISTSERV's Automatic Bounce Detection ------------------------------------- As previously noted, LISTSERV always sends out mailing list messages with a MAIL FROM: pointing to 'owner-listname@[your site]'. Any bounces generated by a mailing should thus go to the owner-listname address. LISTSERV monitors the owner-listname address and any mail that is delivered there is assumed by LISTSERV to be a bounce. If the Auto-Del= setting is set to Full-Auto or Semi-Auto LISTSERV then attempts to understand the message. If it can understand the message it then acts on it, and if not it forwards it to the listowner. This works reasonably well, but there are a few pitfalls. 1) Bounces are not always sent to the correct address. According to Internet standards any bounces generated are supposed to go to the MAIL FROM: address, but there are mail servers out there that do not fully comply with the standards. The result is that you can expect to see some bounces go to virtually any email address in the headers of the message: The "From:", the "Sender:", the "Reply-To:", etc. One result of this is that occasionally bounces are sent to the mailing list itself. Fortunately LISTSERV has ways to detect when this is happening so that the bounces are not distributed to the mailing list; suspected bounces are instead forwarded to the owner. A more pernicious problem is that some bounces will go to addresses that have nothing to do with LISTSERV. Since LISTSERV can't process messages it never sees, this means that such bounces will have to be dealt with manually. Unfortunately since this is the result of mailers you have no control over misbehaving there isn't much that can be done about this problem apart from recognizing that it exists. 2) LISTSERV recognizes bounces that are in some standard formats. (Specifically it recognizes bounces in the format described in RFC 1894 and a couple of other formats.) Unfortunately, there are many non-standard formats around which LISTSERV cannot recognize--some nondelivery notifications don't even say what email address resulted in the bounce. Since LISTSERV doesn't understand these messages it forwards them to the listowner. (This is true even if you have Autodelete=Yes,Full-Auto set.) 3) Someone may sign up for your mailing list with one address and then set things up so that address forwards to a second address. (For instance someone may sign up as [log in to unmask] and then have that forward to [log in to unmask]) If something then goes wrong with the second address LISTSERV will be notified that that address is bad, but it won't be able to do anything about it as that address isn't signed up, even if the bounce is in a recognizable format. Depending on your Auto-Delete= settings LISTSERV will either forward such bounces to the listowner or will just add the address to the daily monitoring report. (Auto-Delete=Yes,Semi-Auto does the former, Auto-Delete=Yes,Full-Auto the latter.) 4) There are several different types of bounces. There are 'temporary' bounces which indicate that if you try sending your message again later it may get through, and 'permanent' bounces which indicate that you should not retry sending the message. LISTSERV gets the information on whether a bounce is temporary or permanent from return codes which are defined in RFC 821 and RFC 1893. So for instance a "550" bounce is a 'permanent' bounce that generally indicates that the recipient email address is unknown (e.g. it doesn't exist). The problem is that the codes may be used incorrectly. For instance, some sites report "Mailbox full"-type errors as 550 errors even though the problem will be cleared up as soon as the user cleans out his or her mail account. (Most sites will use the correct code for this, which is 552.) The key point is that as far as the standards are concerned (and so as far as LISTSERV is concerned as well) the numerical code is the only thing that matters, regardless of what any qualifying text may. Sites using the codes incorrectly may see their users incorrectly getting removed from mailing lists because of this. All the listowner can do is add the affected subscribers back and inform them of what the problem is that caused them to be deleted from the mailing list. Passive Probing --------------- One of the banes of large, active mailing list is the large volume of bounces which LISTSERV gets but can't interpret and so which must be handled manually by the listowner. Starting with LISTSERV 1.8d passive probing was introduced which allows LISTSERV to handle most bounces, even if they are not in a standard format. Here's how it works: Let's say that you have Auto-Delete=Yes,[. . .],Probe(30). When a mailing goes out, 1/30th of the subscribers are chosen randomly to be probed. Those subscribers are sent seperate messages, each of which has a customized MAIL FROM:. Then LISTSERV knows that any mail sent to one of the customized MAIL FROM:s is almost certainly a bounce related to the corresponding email address's subscription. It can then act on that knowledge even if the actual bounce is not in a format that it can understand. Here's a more concrete example: Let's say that the mailing list address is [log in to unmask] and one of the subscribers is [log in to unmask] The MAIL FROM: address would be set to something like owner-listname*badaddress**HOME*-EASE*-LSOFT*[log in to unmask] Now suppose that the subscribed bad address is bad. A bounce is generated and is sent to the MAIL FROM: address, in this case owner-listname*badaddress**HOME*-EASE*-LSOFT*[log in to unmask] LISTSERV sees that a message was sent to that address and extracts the name of the mailing list (listname in this case) and the offending address, which is [log in to unmask] It then deals with it according to the Auto-Delete= settings. This is a very powerful approach and using it results in a very substantial decrease in the amount of time the listowner has to spend dealing with bounces. (Active probing uses essentially the same mechanism to detect bounces.) There are some disadvantages, however: 1) Misbehaving mail servers that don't send to the correct address will continue to be a problem. 2) LISTSERV can now process bounces that aren't in a format it can understand, but when it does so it's essentially flying blind--any message sent to the MAIL FROM: address will be treated as a bounce. This includes autoresponses, 'Mailbox Full'-type messages, and nondelivery notifications warning of a temporary delivery problem. 3) The MAIL FROM: addresses that are used to probe are completely legal in that they fully comply with the relevent standards. However they are not the type of addresses that one runs into much on a day-to-day basis, and some mail servers may reject mail with such a MAIL FROM:. If this appears to be happening it is most commonly because the mail server has been configured to reject mail from addresses containing asterisks ('*'s), though it could also potentially be because of the length of the address. Often in this case if you explain the situation to the email administrator and convince him or her that you are not trying to do anything wrong they will modify the rules to fix this, but of course there is no guarantee. Where to Go for More Information -------------------------------- Both the List Owner's Manual and the Site Manager's Manual contain a lot of information on autodeletion and active and passive probing which is not discussed here. The manuals are available in a variety of formats from http://www.lsoft.com/info/default.asp?item=manuals Internet standards relating to SMTP can be found at http://www.imc.org/rfcs.html#hosttohost and those relating to the format of email messages, including headers, can be found at http://www.imc.org/rfcs.html#rfc822 . For the purposes of this document the most important standards are: RFC 821, "Simple Mail Transfer Protocol" ( available at http://www.imc.org/rfc821 ) RFC 822. "Standard For The Format Of ARPA Internet Text Messages" ( available at http://www.imc.org/rfc822 ) RFC 1123, "Requirements for Internet Hosts -- Application and Support" (in particular chapter 5, "Electronic Mail -- SMTP and RFC-822") ( available at http://www.imc.org/rfc1123 ) RFC 1893, "Enhanced Mail System Status Codes" ( available at http://www.imc.org/rfc1893 ) RFC 1894, "An Extensible Message Format for Delivery Status Notifications" ( available at http://www.imc.org/rfc1894 ) -- Jacob Haller, Technical Support L-Soft international, Inc http://www.lsoft.com/