The LISTS database, originally designed with  the hope that some day with
a bit of luck  we might reach a thousand lists, had  become less and less
usable over the years as the number  of lists increased and the "one disk
file per list"  approach triggered non-linear behaviour in  CMS. The data
files could also get out of sync  with the database index if LISTSERV was
interrupted while  rebuilding its  index, which unfortunately  was rather
frequent,  due  to the  time  this  operation  (entirely coded  in  REXX)
normally took.
 
A new  driver has been written  for the LISTS database.  While the driver
solves  the  problems   mentioned  above  and  divides   the  disk  space
requirements for  the database  by a  factor of  2-3, it  is not  able to
automatically  convert  the files  produced  by  the old  driver.  Manual
intervention will be required from  sites running the LISTS database when
updating to  1.7f or running  the beta-test  code. The remainder  of this
message contains instructions for migrating to the new driver.
 
The first  thing to do is  to allocate a  new minidisk for the  new files
(you will be able  to remove the old minidisk after  the migration, or if
you are short on disk space you can  erase its contents and use it as new
minidisk). The recommended minimum is 15  3380 cylinders or about 9M. The
disk must be  accessed at a fixed filemode, for  performance reasons, and
its filemode must be entered in LOCAL SYSVARS under the name LISTS_FM:
 
                             LISTS_FM = 'G1'
 
You can (and  indeed should) do this  in advance, so that  your server is
ready when 1.7f is shipped. 1.7e will ignore this statement.
 
After installing 1.7f  or the beta version of the  LISTS driver, you will
need to take the following steps:
 
1. (Beta only) Delete the LISTS  DBNAMES and LISTS DBINDEX files from the
   old driver's disk (FILEDISK).
 
2. (Beta  only) Due  to other changes  you must also  do 'ERASE  * CACHE'
   before starting the server with the new code.
 
3. Once the server is started, it  will report a status of "empty so far"
   if you attempt to access the LISTS database. This of course is because
   the disk used by the new driver is empty. When your server is ready to
   accept new files,  you should contact me  and I will send  a number of
   jobs to  supply your server  with the data.  Note that we  are talking
   about a  total of  some 100,000  lines in jobs  of about  10,000 lines
   each. If your  line is slow or  backlogged, make sure to  tell me when
   you would like me to send the files.
 
Your  server will  reject a  number database  entries, claiming  they are
"outdated". This is normal, especially for  your own entries. The files I
have are a snapshot of the  database, which gets updated every night, and
some entries are indeed  outdated. Eventually, the normal synchronization
process will make  your new database converge towards the  same amount of
lists as GLOBLIST FILE.  If you want to load as  many entries as possible
no matter whether  they are outdated, you can ERASE  LUPDTIME FILE before
starting, but remember that GLOBLIST  FILE and the LISTS database entries
are updated  concurrently. If  you want LISTSERV  to leave  GLOBLIST FILE
unchanged, make a copy before you start and restore it afterwards.
 
Performance... You are going to save a  lot of CPU time and I/O, but this
still  doesn't  mean  9370-class   machines  can  realistically  run  the
database. The new driver loads updates to the LISTS database and GLOBLIST
FILE faster than  the old driver could update just  GLOBLIST FILE, but of
course this requires additional I/O's. I  haven't had the courage to time
the old index building  process vs the new one, but it  took less than 30
sec of TCPU on SEARN to build the entire index from scratch (meaning less
than 10 sec  on a 3090). The  procedure that updates the  index no longer
needs enormous amounts  of memory. However, there is nothing  that can be
done about the fact that the database  holds 6-7M of data, that the index
is over 1M, and that the subscribers  count is not supplied by the X-LUPD
protocol, which cannot be changed  without the risk of breaking LISTEARN,
but by a separate set of commands which causes the information to be in a
separate disk  file, from  which it must  be cross-referenced.  On SEARN,
which is  heavily I/O  bound with  a single disk  unit for  everything, a
"dumb" search  such as "search  compilers in  lists" takes 26  seconds of
wall clock time. I suspect a 3090-class machine would complete the search
in 5-6 seconds.
 
Anyway, contact me if you want to beta-test.
 
  Eric