I have just finished debugging generic input spool file manipulation
routines for release 1.7d (yes, d - no time to beta test this in time for
1.7c). These routines will make it possible to use formats other than
PUNCH with acceptable performance, and without having to stage the data
to a disk file and re-read it. They will also simplify the logic of
routines which deal with incoming files, as all the kludges for things
like PROFS or MVS can be put in a single place and then forgotten about.
Now, of course, there is this one problem that the description of the
Netdata format is at best unclear. Oh, the way data is split into
segments of up to 253 bytes so we can save one byte per segment for mail
files and lose a bunch of pairs of bytes (and all the cycles they take to
be reassembled) for binary files is pretty clear. The way text units are
assembled into control records is easy to understand. What the text units
mean, however, is not always clear, and there isn't a single line
describing what the data records contain, and under what conditions.
Looking at source code makes it even less clear what really belongs
where, it looks like the decoding process is a pretty happy-go-lucky
operation - we try not to worry too much about what the final file will
look like, it's probably fixable through some FILEDEF option or filemode
A4 anyway.
Supporting VM files is easy - one data record is one input record from
the original file, no prefix, no adjustment, no problem. Same thing for
JNET. There seems to be a potential however for MVS files where one data
record would be one disk block, or a record count followed by the actual
data, or many other 2-3 letter sequences whose meaning I can guess but
which I have never actually used. I have absolutely no intention of
supporting anything that native VM commands do not handle properly, but I
would like to make sure I remain compatible with 1.7c. In addition, I am
not sure how my code will react to some types of files which I have seen
described in the sources but never actually seen. So I would like to
solicit help from MVS sites, especially those without a decent mail
system ;-), and would like to receive sample files in Netdata format
exhibiting the following arcane properties:
- Two files in the deck, but one is a note so VM can process it.
- More than one file in the deck, in a combination VM won't handle.
- Some dsname with at least 3 components, like AA.BB.CC.DD.
- Some dsname of the type AA.XYZ.OUTPUT but with a INMTERM unit.
- Any record format you can think of which it is reasonable for end users
to try to send to LISTSERV. This may or may not include U, FB, VB and
VBS - I just don't know what normal editor sessions come up with.
- Your favourite mail system, as long as it is supported by 1.7b.
Finally, here are some crude benchmarks I did when I ran the program on
large files to make sure it produced correct output. The routines
basically extract and return strings mapping a logical record, one record
at a time, so I wrote a small program that extracts a record and writes
it to a disk file until EOF, to compare with the original and see how
long it takes. DISK format is particularly slow because it requires two
passes on the reader file - the LRECL is in the last card, and you can't
return the first record until you know it (DISK LOAD, on the other hand,
only needs to write the raw data to disk and sort out the mess
afterwards). CARD was added for the sake of completeness, because it was
easy to do and could be useful to have, and wasn't optimized at all.
Netdata is reasonably optimal on the other hand, I guess as optimal as it
can get in PASCAL. As expected, CARD is the fastest and the native CMS
commands are always slower, but this wasn't the case until I remembered
to set up my data buffers to allocate in blocks of 8k and not 128 bytes
:-)
Eric
+----------------+----------------+---------------+
| LSWPLIB MODULE | Native command | LSWISF+FIO |
| (6 recs, 220k) | VCPU TCPU I/O | VCPU TCPU I/O |
+----------------+----------------+---------------+
| DISK LOAD | 0.55/1.33 36 | 0.23/0.67 31 |
+----------------+----------------+---------------+
| CARD LOAD | 0.12/0.30 33 | 0.24/0.43 31 |
+----------------+----------------+---------------+
| DMSDDL RECEIVE | 0.34/0.87 12 | 0.13/0.30 31 |
+----------------+----------------+---------------+
+----------------+----------------+---------------+
| OSVSAM MACLIB | Native command | LSWISF+FIO |
| (968kb) | VCPU TCPU I/O | VCPU TCPU I/O |
+----------------+----------------+---------------+
| READCARD | 1.38/3.71 128 | 0.33/0.93 35 |
+----------------+----------------+---------------+
| DISK LOAD | 2.27/5.65 89 | 1.20/2.90 35 |
+----------------+----------------+---------------+
| CARD LOAD | 0.51/1.06 128 | 1.52/1.92 35 |
+----------------+----------------+---------------+
| DMSDDL RECEIVE | 2.47/4.87 124 | 0.85/1.45 35 |
+----------------+----------------+---------------+
|
