Discussion:
CRTL and RMS vs SSIO
Add Reply
Greg Tinkler
2021-10-06 02:06:29 UTC
Reply
Permalink
I notice that SSIO (beta) in included in an up coming V9.1 field test. So I read up on the issues it is trying to solve.

One concerning thing was to have CRTL (via SSIO) access directly to XFC. From an architectural point of view this is wrong at so many levels, but if that is what needs to happen then open it up so RMS and other code bases can use it.

The main reason stated was the need to do byte offset/count IO’s. Well lets solve that first, change RMS by adding SYS$READB and SYS$WRITEB. These would be useful to all code using RMS.
SYS$READB read from byte offset for count, return latest data from that byte range.
SYS$WRITEB write from byte offset for count, update latest copy of underlying blocks.

SYS$WRITEB needs to use latest copy of data, and could use the new SSIO interface to XFC but RMS has it's own methods for this.
It may seem like a big ask getting all the latest blocks, but if you think about it it only needs to re-read the last and first block if it does not already have the latest copy. Also no need if the offset starts at the beginning of a block, and it fills the last block.

By having these as part of RMS we want to ensure the blocks/buffers are coordinated so any other user of RMS will see the changes, and we get their changes.

This seems to be at the core of the CRTL issue, it does NOT use RMS, nor does it synchronize its blocks/buffers, leading to the lost update problem.

So with this ‘simple’ addition the CRTL should be altered to us RMS for all file IO.

An extra that could be added, if the file is RFM=fixed, and the C code uses it that way with the same record length then use the SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.

Anyway just my 2 cent worth.

gt down under
Stephen Hoffman
2021-10-06 03:09:14 UTC
Reply
Permalink
Post by Greg Tinkler
I notice that SSIO (beta) in included in an up coming V9.1 field test.
So I read up on the issues it is trying to solve.
One concerning thing was to have CRTL (via SSIO) access directly to
XFC. From an architectural point of view this is wrong at so many
levels,...
Off the top, some of the various existing stuff that breaks layering on
OpenVMS includes HBVS volume shadowing, MOUNT, and byte-range locking.

IP as a layered product is broken layering.

The C select() call is a fine mess of mis-layering.

The XQP design is mis-layering.

There are other examples.

There are examples of breaking layering to advantage, such as ZFS
else-platform.

All discussions of layering and esthetics aside, I presume the primary
purpose of the SSIO project is to permit porting PostgreSQL to OpenVMS,
posthaste.
--
Pure Personal Opinion | HoffmanLabs LLC
Greg Tinkler
2021-10-06 03:32:50 UTC
Reply
Permalink
Post by Stephen Hoffman
Off the top, some of the various existing stuff that breaks layering on
OpenVMS includes HBVS volume shadowing, MOUNT, and byte-range locking.
IP as a layered product is broken layering.
The C select() call is a fine mess of mis-layering.
The XQP design is mis-layering.
There are other examples.
There are examples of breaking layering to advantage, such as ZFS
else-platform.
All discussions of layering and esthetics aside, I presume the primary
purpose of the SSIO project is to permit porting PostgreSQL to OpenVMS,
posthaste.
Yup, exactly, hence get CRTL to use RMS which does work.

Re byte range locking, why not just use locking granularity (aka Rdb) to do the job. Very efficient and has worked for decades, and no need to change VMS DLM. Sure it may be nice to have an API that does this for us, but hey we are programmers.

gt
Stephen Hoffman
2021-10-06 15:09:20 UTC
Reply
Permalink
Post by Greg Tinkler
Yup, exactly, hence get CRTL to use RMS which does work.
For this case, RMS really doesn't work at all well. Says why right
there in the name, too. Record management, not stream management.

C and IP have both been tussling with mismatched assumptions within the
OpenVMS file system since the instantiation of C on OpenVMS, too.

Lately, I've been tussling with the record-oriented assumptions within
OpenVMS. Records just never got as far along as objects. And RMS
records are an unmitigated joy around upgrades and mixed-version
clusters.

The various stream-format files are one of the ensuing compromises here.
Post by Greg Tinkler
Re byte range locking, why not just use locking granularity (aka Rdb)
to do the job. Very efficient and has worked for decades, and no need
to change VMS DLM.
The use of Oracle Rdb isn't viable as a dependency for many folks, and
lock granularity doesn't work at all well for arbitrary and overlapping
locking ranges.
Post by Greg Tinkler
Sure it may be nice to have an API that does this for us, but hey we
are programmers.
I don't want us each writing and debugging and maintaining
range-locking code for what is part of the C standard library, but you
do you.

As much as I'd like a general range-locking solution here in DLM, and
with adding (better?) stream I/O support into RMS, and as much as I'd
like to see OO API support added, and IP integration, and app and app
security integration with sandboxes, packaging, and package management,
and a whole pile of other badly-needed work, I'd infer that the folks
at VSI really want PostgreSQL as an available database option soonest.

There's a very long history of "can-kicking" here and a whole lot of
that is almost inherent and inevitable with the upward-compatibility
goals for the platform, and with resulting miasma far less visible to
those of us that have used OpenVMS for the past decade or three or
more, but is front and center with any new developer looking at the
APIs, and with any wholly new 64-bit app work.
--
Pure Personal Opinion | HoffmanLabs LLC
Craig A. Berry
2021-10-06 12:40:07 UTC
Reply
Permalink
Post by Greg Tinkler
An extra that could be added, if the file is RFM=fixed, and the C
code uses it that way with the same record length then use the
SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.
I don't know the degree to which the current plan corresponds to the
original plan from a decade or so ago, but back then only stream files
were going to be supported by SSIO, which makes sense since the whole
point is locking byte ranges.
David Jones
2021-10-06 13:18:55 UTC
Reply
Permalink
Post by Craig A. Berry
Post by Greg Tinkler
An extra that could be added, if the file is RFM=fixed, and the C
code uses it that way with the same record length then use the
SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.
I don't know the degree to which the current plan corresponds to the
original plan from a decade or so ago, but back then only stream files
were going to be supported by SSIO, which makes sense since the whole
point is locking byte ranges.
Open source software ports often comes with the restriction that it only works
with stream-LF files. Maybe they should add flag to directory files that if set
only allows it to contain stream-LF or directory files.

I keep a stmlf.fdl file in my login directory to use for copying (i.e. convert/fdl=...)
text files to NFS shares.
John Dallman
2021-10-06 19:04:00 UTC
Reply
Permalink
Post by David Jones
Open source software ports often comes with the restriction that it
only works with stream-LF files. Maybe they should add a flag to
directory files that if set only allows it to contain stream-LF
or directory files.
People used to UNIX or Windows generally find the other VMS file types
baffling and confusing. I got used to the idea, but never made use of
them, since my employers already had fewer customers on VMS than they did
UNIX when I joined, and the disparity only increased.

John
Greg Tinkler
2021-10-07 01:25:57 UTC
Reply
Permalink
What a good conversation, some feedback.
Post by Arne Vajhøj
To be honest then I think the safest way to implement this is
to put lots of restrictions on when it is doable.
* No cluster support (announcement already states that!)
* Only FIX 512, STMLF and UDF are supported
* no mixing with traditional RMS calls
My point is SSIO seems to be focused on just PostgreSQL, whereas an RMS solution is much much easier to program, uses well tested code, and is already cluster ready putting the team ahead of the game and not building issues for the future.
Post by Arne Vajhøj
I've a database product, a rather old product. At the time it was
implemented it was rather useful. But there was a locking issue. The
DLM locks resource names. The database would support I/O transfers of 1
to 127 disk blocks. How would one lock 127 contiguous disk blocks? The
blunt force method could be taking out 127 locks, not an optimum
solution. Having numeric range locking back in 1984 would have been
quite useful.
Yup DLM uses resource names, but they can be hierarchical, like a B-Tree index. Also the resources need only exist when needed, removed it not. The the lock tree size depends on the lock contention.

This is why I made reference to Rdb, it uses this technique, and they are probably not the only ones. NB each level controls a range of resources and each level can have it’s own fan out factor. The depth and lowest level is always dependant on the applications requirements.

FYI I am pretty sure RMS uses RFA to lock a record, this is an implied range of 1 record.
Post by Arne Vajhøj
No matter what the disk can do then the VMS file system is still
block oriented and I believe the system services take block offsets
not byte offsets.
All disks are block based, even on Unix. With some SSD’s yes you can do byte transfers, but this should be left to the driver to optimise. Also with X86_64 it weill be virtualised so what the..
Post by Arne Vajhøj
For this case, RMS really doesn't work at all well. Says why right
there in the name, too. Record management, not stream management.
Well yes and no. If you think about it most Unix text IO is record, ie LF terminated, and binary is fixed records not necessarily the same length in the file.

RMS for $GET and $PUT are record based, but $READ and $WRITE are block based, missing is $READB and $WRITEB, not just for CRTL but useful for various applications.

RMS ISAM with fixed length records is a pain, I have long argued ISAM should support variable length records, don’t care if they are VFC or STMLF, I would allow for both as VFC could allow for binary variable length records.

Likewise the keys on an ISAM file should be able to be variable based on a separator e.g “,” or <tab> or a combination.
Post by Arne Vajhøj
The use of Oracle Rdb isn't viable as a dependency for many folks, and
lock granularity doesn't work at all well for arbitrary and overlapping
locking ranges.
I think you will be a B-Tree style dynamic resource tree, similar to what Rdb uses, will work well. Any ‘byte range’ implementation will need some index to find interesting locks, DLM uses hash which is as efficient as you can get.
Post by Arne Vajhøj
Post by Greg Tinkler
Sure it may be nice to have an API that does this for us, but hey we
are programmers.
I don't want us each writing and debugging and maintaining
range-locking code for what is part of the C standard library, but you
do you.
NO, quite the opposite. I believe there is a POSIX standard for a locking API, and as VMS, sorry OpenVMS, wishes to maintain its POSIX stamp it should use these API’s using DLM underneath. NB DLM is also already cluster based, but you know that.
Post by Arne Vajhøj
People used to UNIX or Windows generally find the other VMS file types
baffling and confusing.
I always wondered why the CRTL did not have some smarts to present a VFC records as STMLF and vise-versa, effectively hiding the internal record structures. This could be done via open using the VMS extension “rfm=STMLF” which should be the default unless it is a binary file “rfm=unf”. If the file is VFC then CRTL could to the translation. Wishful thinking.

gt down under
Arne Vajhøj
2021-10-07 01:48:24 UTC
Reply
Permalink
Post by Greg Tinkler
What a good conversation, some feedback.
Post by Arne Vajhøj
To be honest then I think the safest way to implement this is
to put lots of restrictions on when it is doable.
* No cluster support (announcement already states that!)
* Only FIX 512, STMLF and UDF are supported
* no mixing with traditional RMS calls
My point is SSIO seems to be focused on just PostgreSQL, whereas an
RMS solution is much much easier to program, uses well tested code,
and is already cluster ready putting the team ahead of the game and
not building issues for the future.
I very much doubt that a full RMS solution is much easier.

:-)
Post by Greg Tinkler
Post by Arne Vajhøj
For this case, RMS really doesn't work at all well. Says why right
there in the name, too. Record management, not stream management.
Well yes and no. If you think about it most Unix text IO is record,
ie LF terminated, and binary is fixed records not necessarily the
same length in the file.
RMS for $GET and $PUT are record based, but $READ and $WRITE are
block based, missing is $READB and $WRITEB, not just for CRTL but
useful for various applications.
RMS ISAM with fixed length records is a pain, I have long argued ISAM
should support variable length records, don’t care if they are VFC or
STMLF, I would allow for both as VFC could allow for binary variable
length records.
????

Index-sequential files and RMS API supports variable length.

Not all language API's on top of RMS does.
Post by Greg Tinkler
Post by Arne Vajhøj
The use of Oracle Rdb isn't viable as a dependency for many folks, and
lock granularity doesn't work at all well for arbitrary and overlapping
locking ranges.
I think you will be a B-Tree style dynamic resource tree, similar to
what Rdb uses, will work well. Any ‘byte range’ implementation will
need some index to find interesting locks, DLM uses hash which is as
efficient as you can get.
Hash is effective for finding exact matches but useless for finding
other matches aka "starting with". For those a tree is better.

Arne
Lawrence D’Oliveiro
2021-10-07 02:00:36 UTC
Reply
Permalink
Post by Greg Tinkler
All disks are block based, even on Unix.
The difference being, on *nix systems, the responsibility for blocking and deblocking is left to the filesystem layer. So if a file is n bytes long, and n mod «sector size» ≠ 0, the application never sees what is in the padding bytes, if any.

Some filesystems even implement “tail packing”, which means the leftover bits of multiple files can share the same block, all transparently to the application, minimizing fragmentation.

By the way, Linus Torvalds did apparently use a VMS system at some point. (Must have been after his Sinclair QL days.) Guess what reason he gave, when asked why he hated it ...
Post by Greg Tinkler
RMS ISAM with fixed length records is a pain, I have long argued ISAM should support
variable length records ...
Given that nowadays an SQL-based RDBMS like SQLite can offer full support for transactions, joins and subqueries (missing only more multi-user-type features like locking and replication), and yet still be resource-light enough to fit in your mobile phone, I would say the time for application developers to be grubbing about in ISAM files is past.
Dave Froble
2021-10-07 04:10:28 UTC
Reply
Permalink
Post by Greg Tinkler
What a good conversation, some feedback.
Post by Arne Vajhøj
To be honest then I think the safest way to implement this is
to put lots of restrictions on when it is doable.
* No cluster support (announcement already states that!)
* Only FIX 512, STMLF and UDF are supported
* no mixing with traditional RMS calls
My point is SSIO seems to be focused on just PostgreSQL, whereas an RMS solution is much much easier to program, uses well tested code, and is already cluster ready putting the team ahead of the game and not building issues for the future.
RMS is a bit too high level for what's being discussed.

But yeah, the real issue is that SSIO was aimed (it seems) at
PostgreSQL. In my opinion, that is poor software architecture and design.
Post by Greg Tinkler
Post by Arne Vajhøj
I've a database product, a rather old product. At the time it was
implemented it was rather useful. But there was a locking issue. The
DLM locks resource names. The database would support I/O transfers of 1
to 127 disk blocks. How would one lock 127 contiguous disk blocks? The
blunt force method could be taking out 127 locks, not an optimum
solution. Having numeric range locking back in 1984 would have been
quite useful.
Yup DLM uses resource names, but they can be hierarchical, like a B-Tree index. Also the resources need only exist when needed, removed it not. The the lock tree size depends on the lock contention.
Well the perceived issue is what happens when taking out locks, and at
some point there is a conflict. Say needing 127 blocks locked, and the
conflict is on the last block. That means 126 locks to be released, and
perhaps try again.

In reality, the large I/O buffer capability is rarely used, and then
it's usually with exclusive file access, which precludes the need for
block locks, just the file lock. For random access, single block
locking and I/O is good. Larger I/O buffers are usually used for
sequential access, both read only, and updating.
Post by Greg Tinkler
This is why I made reference to Rdb, it uses this technique, and they are probably not the only ones. NB each level controls a range of resources and each level can have it’s own fan out factor. The depth and lowest level is always dependant on the applications requirements.
FYI I am pretty sure RMS uses RFA to lock a record, this is an implied range of 1 record.
RMS has some interesting internals, basically below application usage.

Global buffers
Multiple buffers
Multi-block count

RMS can (I believe, it's been a long while) keep track of file usage,
and provide data from an RMS buffer to a user's buffer. No disk
activity required. Writes of course must go to disk. But even so, the
data can still be in the updated global buffers for use by multiple tasks.
Post by Greg Tinkler
Post by Arne Vajhøj
No matter what the disk can do then the VMS file system is still
block oriented and I believe the system services take block offsets
not byte offsets.
All disks are block based, even on Unix. With some SSD’s yes you can do byte transfers, but this should be left to the driver to optimise. Also with X86_64 it weill be virtualised so what the..
As long as storage is block oriented, then regardless of the numeric
range of bytes, all blocks encompassing the byte range will need to be
read, including locking, and written. This usually will include data
outside the byte range.
Post by Greg Tinkler
Post by Arne Vajhøj
For this case, RMS really doesn't work at all well. Says why right
there in the name, too. Record management, not stream management.
Ayep. RMS is record based.
Post by Greg Tinkler
Well yes and no. If you think about it most Unix text IO is record, ie LF terminated, and binary is fixed records not necessarily the same length in the file.
RMS for $GET and $PUT are record based, but $READ and $WRITE are block based, missing is $READB and $WRITEB, not just for CRTL but useful for various applications.
Forget RMS, I/O would be at the QIO level.
Post by Greg Tinkler
RMS ISAM with fixed length records is a pain, I have long argued ISAM should support variable length records, don’t care if they are VFC or STMLF, I would allow for both as VFC could allow for binary variable length records.
RMS keyed files can have variable record lengths.
RMS relative files require fixed length records. (if I remember correctly)
RMS sequential files can have variable record lengths.
Post by Greg Tinkler
Likewise the keys on an ISAM file should be able to be variable based on a separator e.g “,” or <tab> or a combination.
Post by Arne Vajhøj
The use of Oracle Rdb isn't viable as a dependency for many folks, and
lock granularity doesn't work at all well for arbitrary and overlapping
locking ranges.
I think you will be a B-Tree style dynamic resource tree, similar to what Rdb uses, will work well. Any ‘byte range’ implementation will need some index to find interesting locks, DLM uses hash which is as efficient as you can get.
Post by Arne Vajhøj
Post by Greg Tinkler
Sure it may be nice to have an API that does this for us, but hey we
are programmers.
I don't want us each writing and debugging and maintaining
range-locking code for what is part of the C standard library, but you
do you.
NO, quite the opposite. I believe there is a POSIX standard for a locking API, and as VMS, sorry OpenVMS, wishes to maintain its POSIX stamp it should use these API’s using DLM underneath. NB DLM is also already cluster based, but you know that.
Post by Arne Vajhøj
People used to UNIX or Windows generally find the other VMS file types
baffling and confusing.
That is because, without additional apps, Unix I/O is a stream of bytes.
There is no concept of records, such as that provided by RMS.

Frankly, (and yes, I'm biased), I find records reasonable, and a stream
of bytes baffling and confusing. Guess it's what one is used to.
Post by Greg Tinkler
I always wondered why the CRTL did not have some smarts to present a VFC records as STMLF and vise-versa, effectively hiding the internal record structures. This could be done via open using the VMS extension “rfm=STMLF” which should be the default unless it is a binary file “rfm=unf”. If the file is VFC then CRTL could to the translation. Wishful thinking.
I would suggest the use of "VMS" in the above, rather than "CRTL". That
is unless one considers the CRTL VMS ...
Post by Greg Tinkler
gt down under
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Lawrence D’Oliveiro
2021-10-07 07:54:51 UTC
Reply
Permalink
Post by Dave Froble
Frankly, (and yes, I'm biased), I find records reasonable, and a stream
of bytes baffling and confusing. Guess it's what one is used to.
Trouble is, there are many binary file formats that do not map easily to a simple sequence of records (of whatever delimitation). Consider the IFF family of file formats, for example: these are built out of chunks, and certain chunk types can contain other chunks.

For another example, consider file formats like TIFF and TTF, where there is a directory that identifies the location and size of the various major pieces. Oh, and PDF comes under this as well.

And then there are text-based format families, like XML, JSON, YAML, TOML ...
Simon Clubley
2021-10-07 12:26:36 UTC
Reply
Permalink
Post by Greg Tinkler
Post by Stephen Hoffman
For this case, RMS really doesn't work at all well. Says why right
there in the name, too. Record management, not stream management.
Well yes and no. If you think about it most Unix text IO is record, ie LF terminated, and binary is fixed records not necessarily the same length in the file.
How do you find byte 12,335,456 in a variable length RMS sequential file
without reading from the start of the file ?

That's why there are restrictions on RMS supported file formats in an
application in some cases.
Post by Greg Tinkler
I always wondered why the CRTL did not have some smarts to present a VFC
records as STMLF and vise-versa, effectively hiding the internal record
structures. This could be done via open using the VMS extension ?rfm=STMLF?
which should be the default unless it is a binary file ?rfm=unf?. If the file
is VFC then CRTL could to the translation. Wishful thinking.
This could not be the default. What if LF characters are part of the
existing data record itself ? You have just destroyed the meaning of
the file in that case.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Lawrence D’Oliveiro
2021-10-07 01:51:31 UTC
Reply
Permalink
Post by John Dallman
People used to UNIX or Windows generally find the other VMS file types
baffling and confusing.
One question I never saw answered (because I never came across examples of files to check it) was whether in “VFC” files, the record count included the fixed header or not? And was that the same or different in the on-disk format versus the in-memory RMS structure with the “RSZ” (“RAB$W_RSZ”?) field?

By the way, I knew FORTRAN carriage control is now an anachronism, but I didn’t realize that it is now considered so obsolete, that compilers won’t support it any more.
Arne Vajhøj
2021-10-07 01:59:41 UTC
Reply
Permalink
Post by Lawrence D’Oliveiro
Post by John Dallman
People used to UNIX or Windows generally find the other VMS file types
baffling and confusing.
One question I never saw answered (because I never came across
examples of files to check it) was whether in “VFC” files, the record
count included the fixed header or not? And was that the same or
different in the on-disk format versus the in-memory RMS structure
with the “RSZ” (“RAB$W_RSZ”?) field?
Try it!

$ open/write z.z z.z
$ write z.z "ABC"
$ close z.z
$ dir/full z.z

Directory DISK2:[ARNE]

z.z;1 File ID: (5295,236,0)
...
Record format: VFC, 2 byte header, maximum 0 bytes, longest 3 bytes
...
$ dump z.z

Dump of file DISK2:[ARNE]z.z;1 on 6-OCT-2021 21:54:39.48
File ID (5295,236,0) End of file block 1 / Allocated 16

Virtual block number 1 (00000001), 512 (0200) bytes

00000000 00000000 00000000 00000000 00000000 0000FFFF 00434241
8D010005 ....ABC......................... 000000

Arne
Simon Clubley
2021-10-07 12:12:55 UTC
Reply
Permalink
Post by John Dallman
Post by David Jones
Open source software ports often comes with the restriction that it
only works with stream-LF files. Maybe they should add a flag to
directory files that if set only allows it to contain stream-LF
or directory files.
People used to UNIX or Windows generally find the other VMS file types
baffling and confusing. I got used to the idea, but never made use of
them, since my employers already had fewer customers on VMS than they did
UNIX when I joined, and the disparity only increased.
That because asking Unix/Windows people to learn about VMS records and
file structures is like asking a VMS person to learn about how to work
with records and files on z/OS using traditional z/OS methods.

It is something so very, very, different from what they are used to.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Lawrence D’Oliveiro
2021-10-07 01:45:41 UTC
Reply
Permalink
Post by David Jones
Open source software ports often comes with the restriction that it only works
with stream-LF files.
I would say that’s partially true. Typically there are options to treat files as “text” or “binary”. A “binary” file is just a stream of arbitrary 8-bit bytes, which are supposed to be read or written without any imposition of record boundaries, sector-size rounding or special treatment of any byte values. A “text” file is assumed to be broken up into lines. It is true that LF is the traditional Unix line delimiter. But enlightened toolkits like Python are capable of reading text files in “universal newline” mode, so for example if you copy a text file created on MS-DOS (line delimiter = CR+LF, because CP/M did it that way, for no rational reason) in binary mode onto a Linux system, your Python text-processing script running on the latter can cope with it without a hiccup.
Dave Froble
2021-10-06 13:45:19 UTC
Reply
Permalink
Post by Craig A. Berry
Post by Greg Tinkler
An extra that could be added, if the file is RFM=fixed, and the C
code uses it that way with the same record length then use the
SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.
I don't know the degree to which the current plan corresponds to the
original plan from a decade or so ago, but back then only stream files
were going to be supported by SSIO, which makes sense since the whole
point is locking byte ranges.
It has been my impression that for quite some time at HP, work on
specific requests tended to be very specific to that request, and failed
to consider capabilities as general to VMS.

The approach to SSIO appears to be an example of this. Basically, do
the least required to achieve the specific result. In the case of SSIO
the result appears to be rather useless, at least so far.

For some years I've advocated a more general enhancement to the VMS DLM,
specifically, numeric range locking. Such would address a basic issue
I've had with the VMS DLM for a rather long time.

I've a database product, a rather old product. At the time it was
implemented it was rather useful. But there was a locking issue. The
DLM locks resource names. The database would support I/O transfers of 1
to 127 disk blocks. How would one lock 127 contiguous disk blocks? The
blunt force method could be taking out 127 locks, not an optimum
solution. Having numeric range locking back in 1984 would have been
quite useful.

I've also suggested in the past that a simple enhancement to the DLM,
specifically the addition of a "type of lock" with the capability of
adding logic for specific "types" would solve the locking part of SSIO
and do so as a part of VMS, not as part of the CRTL.

As for byte range I/O, I'm not sure what is and isn't possible with disk
drives. It has been my impression that only whole block transfers are
possible. Perhaps I've been wrong. Perhaps SSDs have more flexibility.

Not really an issue for me anymore.
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Arne Vajhøj
2021-10-06 14:37:05 UTC
Reply
Permalink
Post by Dave Froble
Post by Craig A. Berry
Post by Greg Tinkler
An extra that could be added, if the file is RFM=fixed, and the C
code  uses it that way with the same record length then use the
SYS$GET/SYS$PUT so it will play nicely with an RMS access to those files.
I don't know the degree to which the current plan corresponds to the
original plan from a decade or so ago, but back then only stream files
were going to be supported by SSIO, which makes sense since the whole
point is locking byte ranges.
It has been my impression that for quite some time at HP, work on
specific requests tended to be very specific to that request, and failed
to consider capabilities as general to VMS.
The approach to SSIO appears to be an example of this.  Basically, do
the least required to achieve the specific result.  In the case of SSIO
the result appears to be rather useless, at least so far.
General is better than specific.

When not considering resources.

My impression is that VSI engineering resources are very limited - and
several orders of magnitudes smaller than DEC 40 years ago.

So when they have the choice of solving something 80% for 200 hours of
effort or 100% for 1000 hours of effort then ...
Post by Dave Froble
For some years I've advocated a more general enhancement to the VMS DLM,
specifically, numeric range locking.  Such would address a basic issue
I've had with the VMS DLM for a rather long time.
I've also suggested in the past that a simple enhancement to the DLM,
specifically the addition of a "type of lock" with the capability of
adding logic for specific "types" would solve the locking part of SSIO
and do so as a part of VMS, not as part of the CRTL.
That would make sense to me.

But I do not count.
Post by Dave Froble
As for byte range I/O, I'm not sure what is and isn't possible with disk
drives.  It has been my impression that only whole block transfers are
possible.  Perhaps I've been wrong.  Perhaps SSDs have more flexibility.
No matter what the disk can do then the VMS file system is still
block oriented and I believe the system services take block offsets
not byte offsets.

Arne
Arne Vajhøj
2021-10-06 13:01:17 UTC
Reply
Permalink
Post by Greg Tinkler
I notice that SSIO (beta) in included in an up coming V9.1 field
test. So I read up on the issues it is trying to solve.
One concerning thing was to have CRTL (via SSIO) access directly to
XFC. From an architectural point of view this is wrong at so many
levels, but if that is what needs to happen then open it up so RMS
and other code bases can use it.
The main reason stated was the need to do byte offset/count IO’s.
Well lets solve that first, change RMS by adding SYS$READB and
SYS$WRITEB. These would be useful to all code using RMS. SYS$READB
read from byte offset for count, return latest data from that byte
range. SYS$WRITEB write from byte offset for count, update latest
copy of underlying blocks.
By having these as part of RMS we want to ensure the blocks/buffers
are coordinated so any other user of RMS will see the changes, and we
get their changes.
This seems to be at the core of the CRTL issue, it does NOT use RMS,
nor does it synchronize its blocks/buffers, leading to the lost
update problem.
So with this ‘simple’ addition the CRTL should be altered to us RMS for all file IO.
An extra that could be added, if the file is RFM=fixed, and the C
code uses it that way with the same record length then use the
SYS$GET/SYS$PUT so it will play nicely with an RMS access to those
files.
To be honest then I think the safest way to implement this is
to put lots of restrictions on when it is doable.

Examples:
* No cluster support (announcement already states that!)
* Only FIX 512, STMLF and UDF are supported
* no mixing with traditional RMS calls

Some applications coming over from *nix most known PostgreSQL needs
this. But trying to cover all types of cases would be a lot of
work.

Arne
Loading...