Hi folks,
I recently provided a new home to a mailing list on my server running
LISTSERV LITE, but I only had it's archives in UNIX mailbox format,
so I had to find a way to convert them to notebooks. I found this
script on LSOFT's ftp server, but it wasn't perfect (it only
converted the very basic headers) and for some odd reason, the first
of the '='s in the separator line was lost, so it didn't work without
modification.
I fixed that, simplified output and added capabilites for extended
header conversion. The result is attached. I hope it's useful for
someone. Happy converting!
-- Peter 'Rattacresh' Backes, [log in to unmask]
TURN OFF AUTO-QUOTING OF THE WHOLE TEXT IF YOU REPLY!!!
#! /usr/bin/perl
#####################################################################
#
# Program Name: archive_migrate
#
# Function: migrates Listproc 6.0c format archive files to
# LISTSERV 1.8b minimal digest format notebooks
#
# Comment: In fact converts any UNIX mailbox format
# file to notebook format which can be read by
# virtually any LISTSERV version.
#
# Author: Peter 'Rattacresh' Backes, PLASMA ORGANIZATION
# ([log in to unmask])
# Based on work originally done by Steve Howie
# (CCS. University of Guelph)
# Based on work originally done by Norm Aleks
# ([log in to unmask])
#
# Logic:
# archive_mig will traverse each subdirectory in the
# $HOME/archives/lists directory, converting each archive
# file it finds into Listserv Minimal Digest format. It stores
# each converted archive file under a subdirectory of $newarch_root.
# This tree can then be moved en masse to it's new home, and the
# permissions / ownerships adjusted accordingly.
#
# When processing each directory, you will be prompted for the
# full list name for the "Sender:" field in the archive.
#
# If an error is encountered processing an archive, it will be
# skipped, a flag set and the archive processing will stop after
# the last archive in that directory
#
# You have the opportunity to skip directories if you wish
#
#####################################################################
require "pwd.pl"; # To get a half-assed accurate pwd function
$SIG{'INT'} = 'handler'; # Set up a handler to catch ctrl-C
print <<EOF;
**************************************************************
*
* Listproc -> Listserv Archive Migration Utility
*
**************************************************************
EOF
$newarch_root = "/var/tmp"; # Root for converted archives
$home_dir = `pwd`;
chop($home_dir);
opendir(THISDIR,"."); # Read all list directory names
$num = @alldirs = sort grep(-d, grep(!/^\./, readdir(THISDIR)));
closedir(THISDIR);
printf("\nFound a total of %d list archive directories. Continuing ..\n", $num);
&initpwd;
$err_flag = 0;
$list_tot = $skip_tot = $arch_tot = 0;
#
# Process each lists archive directory
#
foreach $i (@alldirs) {
if ($err_flag == 1) {
print "Errors encountered processing archives. Please check logs\n";
exit;
}
$location = "$home_dir\/$i";
next if ($location eq "/users/u/2/showie/archive"); #local mod
printf("\n>>>> Processing directory %s, press \"y\" to continue..", $i);
$yn = <STDIN>;
chop($yn);
if ($yn ne "y") {
print "Skipping directory $i ... \n";
$skip_tot++;
next;
}
$list_tot++;
print "Please enter the title of the group: ";
$oldlistaddr = <STDIN>;
chop($oldlistaddr);
print "Creating Directory ", $newarch_root."/".$i, " ...\n";
mkdir("$newarch_root/$i",0755); # Create new archive directory subtree
&chdir($location); # Move into next directory to process
#
# get a list of all archive files in the directory
#
opendir(THISDIR,".");
@newfiles = sort grep(/^[0-9]/,grep(!/^\./, readdir(THISDIR)));
closedir(THISDIR);
#
# Process each archive file inside the directory
#
foreach $archive (@newfiles) {
printf("Processing archive %s..", $archive);
$arch_tot++; # Bump total
$newname = $archive;
#
# Only rename and uncompress .z archives
#
if (/\.z$/) {
printf(".");
$newname =~ tr/z/Z/;
if (rename($archive,$newname) == 0) {
printf("Error renaming %s to %s, skipping this archive\n", $archive, $newname);
$err_flag = 1;
next;
}
printf(".");
$result = system("uncompress $newname");
if ($result != 0) {
printf("Error uncompressing %s, skipping this archive\n", $newname);
$err_flag = 1;
next;
}
chop($newname);
chop($newname);
}
printf(".");
$result = 0;
$result = &convert($newname);
if ($result == 1) {
printf("Error converting archive %s, skipping this archive\n", $newname);
$err_flag = 1;
next;
}
printf(" DONE\n");
}
}
printf("\n\nProcessed %d list(s) and %d archives. A total of %d lists were skipped. \n\n", $list_tot, $arch_tot, $skip_tot);
exit 0;
#
# Signal Handler for ctrl-C
#
sub handler {
print "User pressed CTRL-C, exiting..\n";
exit(0);
}
#
# Do the actual archive conversion
#
sub convert {
($archive_name) = @_;
open(NEWARCH, "> $newarch_root/$i/log$archive_name");
open(INFILE, $archive_name);
#print "\n\t\tCreating new Listserv archive: $newarch_root/$i/log$archive_name\n";
$body = 0;
$delm = 0;
while (<INFILE>) {
if ($body) {
if (/^From .* \d\d\d\d$/ ) {
$body = 0;
$delm = 0;
} else {
if ($delm) {
print NEWARCH "\n";
}
if (/^$/) {
$delm = 1;
} else {
$delm = 0;
print NEWARCH;
}
}
} else {
if (/^$/) {
$body = 1;
print NEWARCH "=========================================================================\n";
print NEWARCH "Date: $headerline{'DATE'}\n";
if ($headerline{'REPLY-TO'} ne "") {
print NEWARCH "Reply-To: $headerline{'REPLY-TO'}\n"; }
else {
print NEWARCH "Reply-To: $oldlistaddr\n"; }
print NEWARCH "Sender: $oldlistaddr\n";
if ($headerline{'FROM'} ne "") {
print NEWARCH "From: $headerline{'FROM'}\n"; }
if ($headerline{'ORGANIZATION'} ne "") {
print NEWARCH "Organization: $headerline{'ORGANIZATION'}\n"; }
if ($headerline{'SUBJECT'} ne "") {
print NEWARCH "Subject: $headerline{'SUBJECT'}\n"; }
if ($headerline{'IN-REPLY-TO'} ne "") {
print NEWARCH "In-Reply-To: $headerline{'IN-REPLY-TO'}\n"; }
if ($headerline{'MIME-VERSION'} ne "") {
print NEWARCH "MIME-Version: $headerline{'MIME-VERSION'}\n"; }
if ($headerline{'CONTENT-TYPE'} ne "") {
print NEWARCH "Content-type: $headerline{'CONTENT-TYPE'}\n"; }
if ($headerline{'CONTENT-TRANSFER-ENCODING'} ne "") {
print NEWARCH "Content-Transfer-Encoding: $headerline{'CONTENT-TRANSFER-ENCODING'}\n"; }
print NEWARCH "\n";
$headerline{'DATE'} = "";
$headerline{'REPLY-TO'} = "";
$headerline{'FROM'} = "";
$headerline{'ORGANIZATION'} = "";
$headerline{'SUBJECT'} = "";
$headerline{'IN-REPLY-TO'} = "";
$headerline{'MIME-VERSION'} = "";
$headerline{'CONTENT-TYPE'} = "";
$headerline{'CONTENT-TRANSFER-ENCODING'} = "";
} else {
chop;
if (! /^\s+(.*)/) {
$toprint = (($hdr,$info)=/^([a-zA-Z][a-zA-Z-]*)\:\s+(.*)/i);
$hdr =~ tr/a-z/A-Z/;
$headerline{$hdr} = $info if $toprint;
} else {
$headerline{$hdr} .= " $1" if $toprint;
}
}
}
}
close(NEWARCH);
close(INFILE);
return 0;
}
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
PLASMA ORGANIZATION DOC.MN.1 ARCH_MIG MEMO
(DESCRIPTION, USAGE)
Peter 'Rattacresh' Backes <[log in to unmask]>, Jul 03 2000, Rev 0
(c) 2000 PLASMA ORGANIZATION
Keywords: LISTSERV, archive, notebook, converting, UNIX mailbox,
mailing list, mailing list manager, MLM
Archive_migrate is a program which helps you converting your UNIX
mailbox format files into the notebook format, which are nowadays used
mainly in the context of LISTSERV archives. If you were using a
different MLM in the past which used the UNIX mailbox format as it's
archive format (notably Listproc) or if you just took over mailing list
hosting for a list you were a member of in the past and can't contact
the old owner to send you the old archives so you have to use what your
own mailbox folder contains, then this is the program you're looking
for.
Before you run the actual conversion script, order the original log
files as follows:
. Current directory
+ lstone-l Directory for the first list
| | 9701 UNIX mailbox format archive for Jan 1997
| | 9702 Feb 1997
| | ... Other archive formats, using the name convention
+ lsttwo-l Directory for the second list
| | ...
+ ... More list directories containing archives
The conversion script actually only cares about the first character of
the archive files; it must be a digit. After conversion, you will find
the files in LISTSERV name convention, <listname>.log<period>. You can
exploit this to use a different period scheme, like weekly or yearly
notebooks, as LISTSERV provides them. For further information about
this, consult section 8.10.3 of the Site Manager's Operations Manual
for LISTSERV 1.8d (reference number 9903-MD-01). To separate your
archives in monthly/weekly/yearly chunks just use your favourite mailer
that provides a feature to bulk move mails between UNIX mailbox
folders. I used mutt which did it's job without any problems.
When you've prepared the directory structure as described, change to
the directory that contains the list subdirectories and execute the
script from there (either copy the conversion program into this
directory or add the directory that contains it to your PATH). Don't
forget to give it +x mode first. Then run the script. It will show
something similar to the following, depending on your list names:
**************************************************************
*
* Listproc -> Listserv Archive Migration Utility
*
**************************************************************
Found a total of 1 list archive directories. Continuing ..
>>>> Processing directory ogde-l, press "y" to continue..
You can press 'y' to process this archive directory, something else to
skip it or send a ^C to terminate the script entirely. By the way, you
can press ^C at any time to emergency exit the script if it shows brain
disease.
After pressing 'y', you will be asked the following:
Please enter the title of the group:
arch_mig expects the RFC822 mailbox for your list to be entered. It is
recommended to give it a description of the list and the (old) list
email address in angle brackets. Be sure to use "double quotes" around
your list description if it contains one of the following characters:
()<>@,;:".[]\ Also be sure to escape double quotes inside double
quotes with a backspace:
German OG Mailing List <[log in to unmask]>
After hitting enter, arch_mig will process each archive and save it in
notebook format under a new directory. It will print a short status
message for each file:
Creating Directory /var/tmp/ogde-l ...
Processing archive 0001... DONE
Processing archive 0003... DONE
Processing archive 9802... DONE
Processing archive 9803... DONE
Processing archive 9804... DONE
Processing archive 9806... DONE
Processing archive 9807... DONE
Processing archive 9808... DONE
Processing archive 9809... DONE
Processing archive 9810... DONE
Processing archive 9811... DONE
Processing archive 9812... DONE
Processing archive 9901... DONE
Processing archive 9902... DONE
Processing archive 9903... DONE
Processing archive 9904... DONE
Processing archive 9905... DONE
Processing archive 9906... DONE
Processing archive 9907... DONE
Processing archive 9908... DONE
Processing archive 9909... DONE
Processing archive 9911... DONE
Processing archive 9912... DONE
When every file it found in the list directory was converted properly,
it possibly asks you for the the next one if there is one. Just
proceed as described above. After the last one being finished, it will
give you a summary of the work it's done. As we only converted
archives for one list in our case we would be shown this:
Processed 1 list(s) and 23 archives. A total of 0 lists were skipped.
Then you'll find yourself on the shell prompt again. All notebooks can
now be found under the directory /var/tmp/<listname>, for our example
list that's /var/tmp/ogde-l. You can change to /var/tmp, fix file
permissions if neccessary and either just move all directories to where
your LISTSERV list folders are stored or -- if they already exist for
your list -- move the contents of the directories to their appropriate
positions. Send a INDEX command for your list to LISTSERV and check
whether everything is OK. You can reboot LISTSERV or wait until
midnight if you want the archives to be available via web interface.
THE SOFTWARE IS PROVIDED ``AS IS'' WITHOUT EXPRESS OR IMPLIED WARRANTY
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT WILL
PLASMA OR ANY OTHER CONTRIBUTOR BE LIABLE FOR DIRECT, INCIDENTAL OR
CONSEQUENTIAL DAMAGES, EVEN IF PLASMA HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
|
