Discussion:
VMS enhancement suggestion: Add a "read regardless" file open option.
Add Reply
Simon Clubley
2020-11-09 13:30:23 UTC
Reply
Permalink
On RSTS/E, you can view the contents of a file opened for write by
specifying mode 4096 as in:

pip filename.dat/mo:4096

What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?

How useful would others here find this capability ?

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Jan-Erik Söderholm
2020-11-09 13:49:07 UTC
Reply
Permalink
Post by Simon Clubley
On RSTS/E, you can view the contents of a file opened for write by
pip filename.dat/mo:4096
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
How useful would others here find this capability ?
Simon.
How usefull? I'd say it would be usefull, simply.

Our detached applications logs messages to log files which
are opened/written/closed for each line written. So they
can always be TYPE'ed. But with some overhead for each line
written, of course.
Stephen Hoffman
2020-11-09 15:30:00 UTC
Reply
Permalink
Post by Simon Clubley
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
How useful would others here find this capability ?
Not all that useful, as I'm not usually looking to access what can be
inconsistent data.

For override access, use CONVERT /SHARE when access to a sharing-locked
log file is required. Or BACKUP /ALLOW=DATA_INCONSISTENCIES, err,
BACKUP /IGNORE=INTERLOCK, otherwise. Or one of the previously-discussed
tools.

For me, adding an OPEN /SHARE=OVERRIDE, and/or TYPE /SHARE, and/or TYPE
/SHARE /TAIL option or otherwise exposing FIB$M_NOLOCK isn't all that
interesting, as I just don't have to deal with reading locked files.

Not for very long, if at all.

With batch, use SET OUTPUT to increase the write frequency, and/or
redirect the interesting stuff to a log elsewhere. Elsewhere with
sharing enabled. Logging app data into batch is... it works.

That batch processing default-caches 30 seconds of data per write is an
interesting optimization from a previous era of I/O performance. This
given batch output is a negligible fraction of typical SSD and NVMe
bandwidth. But I digress.

Or write your own app logs, as you can set the sharing access to those
as required. Or variously better for storing logging data rather than
mixing together logging data and app data as is typical with how some
folks use batch logs, writing the log data to a database, or to a
syslog server, or... and write the app data to your own files or
databases or servers.

Put differently, I usually fix whatever I'm needing access into, if
it's blocking that access. And yes, it's also possible to intercept and
shim the non-shareable open API call into a shareable open I/O call.
I've not had to resort to that in many years, though. I typically have
the app source. I've fixed countless numbers of apps over the years
that have had unnecessary file locks specified at open. Apps written in
C are notorious for this too, due to the non-sharing defaults on fopen
et al.

This whole area reeks of old designs, and an inability to look under
the proximate rocks, with an inability to actually fix the problematic
app code (for reasons), leading to ever-more-complex work-arounds in
OpenVMS itself.

*Ponders whether somebody will then eventually request a
DOUBLE-SECRET-NON-SHAREABLE-FILE option that can then prevent
overriding and accessing a non-shareable file. That might seem
humorous. But there have been similar requests made elsewhere. Object
access controls that can selectively override privileges is one case
that's been discussed around here.*

I'm not sure what sort of change might be appropriate here. TYPE /SHARE
is probably the smallest and thus most manageable and most likely, but
it seems there are rather larger re-designs available here. OpenVMS
batch and job scheduling and the class scheduler and logging and
operator communications—all features related to keeping production
running and for recovering from production failures—are somewhere
between problematic and absent, and these particular older designs are
probably less reflective of where many of us find ourselves now working.
--
Pure Personal Opinion | HoffmanLabs LLC
Simon Clubley
2020-11-09 18:17:31 UTC
Reply
Permalink
Post by Stephen Hoffman
Post by Simon Clubley
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
How useful would others here find this capability ?
Not all that useful, as I'm not usually looking to access what can be
inconsistent data.
Stephen, have you ever worked in commercial data processing ?

There are plenty of uses for such a facility, none of which involve
data integrity issues.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Stephen Hoffman
2020-11-09 19:38:22 UTC
Reply
Permalink
Post by Simon Clubley
Post by Stephen Hoffman
Post by Simon Clubley
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
How useful would others here find this capability ?
Not all that useful, as I'm not usually looking to access what can be
inconsistent data.
Stephen, have you ever worked in commercial data processing ?
Yes.

I'm familiar with commercial data processing code, having written and
maintained and refactored and enhanced that, and with the development
and management and administration and politics of same.

Also familiar with the benefits and trade-offs that can arise with
enhancement requests, too. And familiar with requests that are
fundamentally escalations of local political and policy limits and
workarounds, too.
Post by Simon Clubley
There are plenty of uses for such a facility, none of which involve
data integrity issues.
Some interlocks exist for a reason. Some don't. Some exist because
developers didn't know better and/or didn't care and/or accepted poor
defaults.

And have worked with entities that absolutely could not change and
could not upgrade their configurations for reasons, and some of those
for decades. Then management toggled the organizational write-lock bit
off somewhere, and, wow, all those previously-impossible upgrades and
source code fixes and the rest of the backlog all arrived. Some
enhancements are attempts to work around a developer- or management-led
policy, and for these sorts of cases I'm more skeptical about
enhancements. Those developers and management folks may well have
reasons for those policies, and often do. Whether those reasons and
related justifications are valid under broader review is another
discussion, and one best held among local staff.

If the TYPE with FIB$M_NOLOCK is implemented, I would then foresee a
few requests arriving for double-interlocks to prevent this override
from working, because somebody had some reason to want to prevent
access to data in progress. (I've used this to stage data, for
instance.) But then FIB$M_NOLOCK does currently require system
privilege or control access, so that's potentially more tractable than
some of these cases.



Inevitable Digression 1: a slightly less inelegant hack might allow
specific file open operation defaults to be overridden, akin to how
default protections can be specified for file operations.

Inevitable Digression 2: some folks use TYPE to review indexed file
data. I've met that usage in production more than once, too.



TL;DR: I'd prefer to re-code the app (slightly) to allow sharing,
rather than adding FIB$M_NOLOCK to TYPE. From the vendor side of this,
I'd prefer work toward the bigger issues latent and lurking within
OpenVMS be addressed—job scheduling, operator communications, batch
logging, system logging and filtering and automation, etc. This whole
area of OpenVMS—displaying log files, scheduling and rescheduling and
handling errors, etc—is not far enough past 1980 for my own production
preferences.
--
Pure Personal Opinion | HoffmanLabs LLC
V***@SendSpamHere.ORG
2020-11-10 02:25:36 UTC
Reply
Permalink
Post by Simon Clubley
Post by Stephen Hoffman
Post by Simon Clubley
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
How useful would others here find this capability ?
Not all that useful, as I'm not usually looking to access what can be
inconsistent data.
Stephen, have you ever worked in commercial data processing ?
ROTFLMFAO!
Post by Simon Clubley
There are plenty of uses for such a facility, none of which involve
data integrity issues.
It'd be much simpler to add shared-read to those log files. ;)
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
Dave Froble
2020-11-09 19:02:48 UTC
Reply
Permalink
Post by Simon Clubley
On RSTS/E, you can view the contents of a file opened for write by
pip filename.dat/mo:4096
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
How useful would others here find this capability ?
Simon.
Hmmm ....

Seems like VAX Basic has a "read regardless" lock qualifier ...

Might that be helpful?

And of course RMS would have the capability.

As for the TYPE utility, haven't looked ...
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Simon Clubley
2020-11-10 13:25:30 UTC
Reply
Permalink
Post by Dave Froble
Post by Simon Clubley
On RSTS/E, you can view the contents of a file opened for write by
pip filename.dat/mo:4096
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
How useful would others here find this capability ?
Simon.
Hmmm ....
Seems like VAX Basic has a "read regardless" lock qualifier ...
I wasn't aware of that. Does it allow you to open a sequential file for
reading while the file is also in the process of being created by another
program ? You can clearly do it for batch log files but what about programs
that don't specify that writing mode when creating a sequential file ?
Post by Dave Froble
Might that be helpful?
And of course RMS would have the capability.
Interesting. One of the issues that was raised when I moved from RSTS/E
to VMS in the early 1990s was that a VMS version of mode 4096 didn't
exist and I wasn't aware it had been added in the meantime.
Post by Dave Froble
As for the TYPE utility, haven't looked ...
I just did in case I missed something there as well and couldn't find
anything. If that capability does exist in RMS, then I change my
request to adding a qualifier to TYPE to use it. That should be something
that would be very easy to add in that case.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Craig A. Berry
2020-11-10 15:06:25 UTC
Reply
Permalink
Post by Simon Clubley
Post by Dave Froble
Seems like VAX Basic has a "read regardless" lock qualifier ...
I wasn't aware of that. Does it allow you to open a sequential file for
reading while the file is also in the process of being created by another
program ? You can clearly do it for batch log files but what about programs
that don't specify that writing mode when creating a sequential file ?
Post by Dave Froble
Might that be helpful?
And of course RMS would have the capability.
Interesting. One of the issues that was raised when I moved from RSTS/E
to VMS in the early 1990s was that a VMS version of mode 4096 didn't
exist and I wasn't aware it had been added in the meantime.
Search for "OpenVMS RMS RAB$V_RRL" and see how it works. That's
record-level, though, not file-level.
Jim
2020-11-10 21:48:36 UTC
Reply
Permalink
Post by Simon Clubley
Post by Dave Froble
Seems like VAX Basic has a "read regardless" lock qualifier ...
I wasn't aware of that. Does it allow you to open a sequential file for
reading while the file is also in the process of being created by another
program ? You can clearly do it for batch log files but what about programs
that don't specify that writing mode when creating a sequential file ?
Post by Dave Froble
Might that be helpful?
And of course RMS would have the capability.
Interesting. One of the issues that was raised when I moved from RSTS/E
to VMS in the early 1990s was that a VMS version of mode 4096 didn't
exist and I wasn't aware it had been added in the meantime.
Search for "OpenVMS RMS RAB$V_RRL" and see how it works. That's
record-level, though, not file-level.
For file level access - search for fib$v_nolock or fib$m_nolock
Craig A. Berry
2020-11-10 23:16:24 UTC
Reply
Permalink
Post by Jim
Post by Simon Clubley
Post by Dave Froble
Seems like VAX Basic has a "read regardless" lock qualifier ...
I wasn't aware of that. Does it allow you to open a sequential file for
reading while the file is also in the process of being created by another
program ? You can clearly do it for batch log files but what about programs
that don't specify that writing mode when creating a sequential file ?
Post by Dave Froble
Might that be helpful?
And of course RMS would have the capability.
Interesting. One of the issues that was raised when I moved from RSTS/E
to VMS in the early 1990s was that a VMS version of mode 4096 didn't
exist and I wasn't aware it had been added in the meantime.
Search for "OpenVMS RMS RAB$V_RRL" and see how it works. That's
record-level, though, not file-level.
For file level access - search for fib$v_nolock or fib$m_nolock
Ah, yes, I either forgot or didn't know about that. But it looks like
that's ACP-level only and doesn't fit Dave Froble's statement that I was
responding to, i.e., that whatever BASIC does, "RMS would have the
capability" because with FIB$M_NOLOCK you have to do a $QIO to open the
file. Hein has a nice write-up here:

<https://community.hpe.com/t5/operating-system-openvms/open-file-with-no-locks-enabled-so-that-a-2nd-process-can-access/td-p/7011048#.X6sbMy9h1TY>

and I'm going to take the liberty of copy-pasting his program below
since who knows how long those HPE forum posts will still be around.

(N.B. I have never seen nor used RSTS/E and have no idea what mode 4096
was.)

/*
** read_locked.c Hein van den Heuvel, July 2006
**
** This program can be used to copy the allocated blocks from a locked file
** Much like BACKUP/IGNORE=INTERLOCK is use the ACP NOLOCK access to over-
** ride file interlock. This requires SYSPRV or CONTROL ACCESS to the file.
** Unlike BACKUP this program simply reads all ALLCOATED blocks as the
** EOF information is likely not to be updated by the active writer (zero).
** The program copies MOST file attributes, but not all as its intended
** use is limited to copy live log files and such.
**
** You may need to use SET FILE/ATTR=(EBK:x,FFB:y) on the output as needed.
**
** If the code format looks a little odd and dated... it is!
** This program is loosly basde on some old example which where floating
** around at Digital Equipment such as SET_EXTENT.
** Suspected authors: Barry Dysert, Guenther Froehlin, me?...
**
** Enjoy!
** Hein, HvdH Perfromance Consulting
**
*/

#define MAXBLOCKS 120
#define MAXBYTES MAXBLOCKS*512

#include atrdef
/*
** libr/extr=fatdef/out=fatdef.h sys$library:sys$lib_c.tlb
*/
#include "fatdef.h"
#include fibdef
#include iodef
#include rms
#include stdio
#include stdlib
#include string

int sys$create(), sys$connect(), sys$write(), sys$close();
int sys$create(), sys$parse(), sys$search();
int sys$assign(), SYS$QIOW(), lib$stop();

main(argc,argv)
int argc;
char *argv[];
{
static char buf[MAXBYTES];
static char *usage = "Usage: $ read_locked name output_name\n";
static char esa[256], rsa[256];
static int status, channel, bytes, vbn=1;
static int file_hbk, file_nbytes, spec_nbytes;
static struct FAB fab, out;
static struct RAB rab;
static struct NAM nam;
static struct
{
short status;
char not_used[6];
} iosb;
static struct
{
short count;
short not_used;
void *address;
} fibdes;
static struct fibdef fib;
static struct atrdef atr[2];
FAT attributes;
struct { int len; char *addr; } devnam_desc;

/******************************************************************************/

/* Verify that we've been properly invoked */

if (argc != 3) printf("%s",usage), exit(1);

/* Use RMS to parse the file so that we get a FID of the QIO */

fab = cc$rms_fab;
fab.fab$l_fna = argv[1];
fab.fab$b_fns = strlen (argv[1]);
fab.fab$l_nam = &nam;

nam = cc$rms_nam;
nam.nam$l_esa = esa;
nam.nam$b_ess = sizeof (esa) - 1;
nam.nam$l_rsa = rsa;
nam.nam$b_rss = sizeof (rsa) - 1;

out = cc$rms_fab;
out.fab$b_fac = FAB$M_BIO | FAB$M_PUT;
out.fab$l_fna = argv[2];
out.fab$b_fns = strlen (argv[2]);

rab = cc$rms_rab;
rab.rab$l_fab = &out;
rab.rab$l_rbf = buf;
rab.rab$w_rsz = sizeof(buf);

if (((status=sys$parse(&fab)) & 1) != 1) lib$stop(status);
if (((status=sys$search(&fab)) & 1) != 1) lib$stop(status);

/* Get a channel for QIO access */

devnam_desc.addr = nam.nam$l_dev;
devnam_desc.len = nam.nam$b_dev;
if (((status=sys$assign(&devnam_desc,&channel,0,0,0)) & 1) != 1)
lib$stop(status);

/* Set up the structures required for the ACP interface */

fibdes.count = sizeof(fib);
fibdes.address = &fib;

fib.fib$l_acctl = FIB$M_NOLOCK;
fib.fib$w_fid_num = nam.nam$w_fid[0];
fib.fib$w_fid_seq = nam.nam$w_fid[1];
fib.fib$w_fid_rvn = nam.nam$w_fid[2];

atr[0].atr$w_type = ATR$C_RECATTR;
atr[0].atr$w_size = ATR$S_RECATTR;
atr[0].atr$l_addr = &attributes;
atr[1].atr$w_type = 0;
atr[1].atr$w_size = 0;

/* Get the file's current attributes, such as hi-block, rfm,... */

status =
SYS$QIOW(0,channel,IO$_ACCESS|IO$M_ACCESS,&iosb,0,0,&fibdes,0,0,0,&atr,0);
if ((status & 1) != 1) lib$stop(status);
if ((iosb.status & 1) != 1) lib$stop(iosb.status);

/* Validate the specified values (e.g. can't extend the file) */

file_hbk = attributes.fat$w_hiblkl + (attributes.fat$w_hiblkh << 16);
file_nbytes = file_hbk*512;

out.fab$l_alq = file_hbk;
out.fab$b_rat = attributes.fat$b_rattrib;
out.fab$b_rfm = attributes.fat$b_rtype;
out.fab$w_mrs = attributes.fat$w_maxrec; /* LRL=fat$w_rsize ?? */
out.fab$b_fsz = attributes.fat$b_vfcsize;
status = sys$create ( &out ) ;
if (status & 1) status = sys$connect ( &rab );
while ((status & 1) && (file_nbytes > 0)) {

if (file_nbytes >= MAXBYTES) {
bytes = MAXBYTES;
file_nbytes -= MAXBYTES;
} else {
bytes = file_nbytes;
rab.rab$w_rsz = file_nbytes;
file_nbytes = 0;
}

status =
SYS$QIOW(0,channel,IO$_READVBLK,&iosb,0,0,&buf,bytes,vbn,0,0,0,0);
vbn += MAXBLOCKS;
if (status & 1) status = iosb.status;
if (status & 1) status = sys$write ( &rab );
}

/* Release the files */

(void) SYS$QIOW(0,channel,IO$_DEACCESS,&iosb,0,0,&fibdes,0,0,0,0,0);
(void) sys$close ( &out );
return status;
}
Dave Froble
2020-11-11 00:55:12 UTC
Reply
Permalink
Post by Craig A. Berry
Post by Jim
Post by Simon Clubley
Post by Dave Froble
Seems like VAX Basic has a "read regardless" lock qualifier ...
I wasn't aware of that. Does it allow you to open a sequential file for
reading while the file is also in the process of being created by another
program ? You can clearly do it for batch log files but what about programs
that don't specify that writing mode when creating a sequential file ?
Post by Dave Froble
Might that be helpful?
And of course RMS would have the capability.
Interesting. One of the issues that was raised when I moved from RSTS/E
to VMS in the early 1990s was that a VMS version of mode 4096 didn't
exist and I wasn't aware it had been added in the meantime.
Search for "OpenVMS RMS RAB$V_RRL" and see how it works. That's
record-level, though, not file-level.
For file level access - search for fib$v_nolock or fib$m_nolock
Ah, yes, I either forgot or didn't know about that. But it looks like
that's ACP-level only and doesn't fit Dave Froble's statement that I was
responding to, i.e., that whatever BASIC does, "RMS would have the
capability" because with FIB$M_NOLOCK you have to do a $QIO to open the
http://h30266.www3.hpe.com/odl/vax/progtool/cpqbasic39/bas_ref_021.htm#index_x_1706

Read about the USEROPEN in a file OPEN statement. Basically (sic) the
FAB and RAB (I believe) are populated, then the USEROPEN routine,
provided by the programmer, can set or clear FAB and RAB fields, and
then do the actual file open.

http://h30266.www3.hpe.com/odl/vax/progtool/cpqbasic39/bas_ref_015.htm#index_x_1228

Read about the GET statement, and the REGARDLESS lock clause.

Basic(ally) (sic), it's in there ....
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Hein RMS van den Heuvel
2020-11-11 06:16:51 UTC
Reply
Permalink
Read about the USEROPEN in a file OPEN statement. Basically (sic) the
FAB and RAB (I believe) are populated, then the USEROPEN routine,
provided by the programmer, can set or clear FAB and RAB fields, and
then do the actual file open.
Basic(ally) (sic), it's in there ....
Noop. Sorry. A program doesn't get that far!
As others indicated, the RRL (read Regardless of Lock) option is RECORD level for your GETs, but RMS will fail the open when the writer did not explicitly grant permission.

Some replies refer to BATCH log file and for those indeed increasing the output_rate will help and indeed DCL could be improved for all to enjoy. Fine, but nothing in the Original Post suggested to me that batch job logs were the main or only concern.

The relative usefulness of the NO_LOCK XQP option can be tested by using BACKUP/IGNORE=INTERLOCK as copy tool.
You may find that the 'log' writer often does NOT force RMS to update the EOF, leaving it at zero and therefor it will NOT copy anything! My silly program, which Craig kindly found and re-posted here may be able to return more data for some usages.

For occasional/exceptional usage Wizards, and apprentice wizards, would of course use ANALYZE/SYSTEM to just read the last record written straight in user process spaces. Roughly -
SET PROCESS XXX
SHOW PROC/CHAN !-- helpful to match the FAB with the file through STV when the name is no longer connected to the fab; SHOW PROC/RMS=FAB ! Find the file desired looking at FNM/DNM and find file number in IFI - for tricky cases: RMS=(FAB,FWA)
SHOW PROC/RMS=(NOIFB:a, RAB) ! guess 'a' from relative file number, or from above, NOIFB selects one file only, without displaying IFAB ... which you might need to find actual in memory EOF
EXAM ufb;usz ! Display last record using RAB pointer/size. -
If that's too dynamic, then you may want to look at the last buffer(s).
SHOW PROCS/RMS=BDBSUM will show start addresses and VBN ranges for the last buffers.

If would require a fair bit of Kernel mode code to turn the above into a program, not impossible (and easier than writing a reliable intercept - right Brian!) but nobody has done this in a semi productized fashion best I know. Hmmm....

Hein.
V***@SendSpamHere.ORG
2020-11-11 22:02:59 UTC
Reply
Permalink
Post by Hein RMS van den Heuvel
Read about the USEROPEN in a file OPEN statement. Basically (sic) the
FAB and RAB (I believe) are populated, then the USEROPEN routine,
provided by the programmer, can set or clear FAB and RAB fields, and
then do the actual file open.
Basic(ally) (sic), it's in there ....
Noop. Sorry. A program doesn't get that far!
As others indicated, the RRL (read Regardless of Lock) option is RECORD level for your GETs, but RMS will fail the open when the writer did not explicitly grant permission.
Some replies refer to BATCH log file and for those indeed increasing the output_rate will help and indeed DCL could be improved for all to enjoy. Fine, but nothing in the Original Post suggested to me that batch job logs were the main or only concern.
The relative usefulness of the NO_LOCK XQP option can be tested by using BACKUP/IGNORE=INTERLOCK as copy tool.
You may find that the 'log' writer often does NOT force RMS to update the EOF, leaving it at zero and therefor it will NOT copy anything! My silly program, which Craig kindly found and re-posted here may be able to return more data for some usages.
For occasional/exceptional usage Wizards, and apprentice wizards, would of course use ANALYZE/SYSTEM to just read the last record written straight in user process spaces. Roughly -
SET PROCESS XXX
SHOW PROC/CHAN !-- helpful to match the FAB with the file through STV when the name is no longer connected to the fab; SHOW PROC/RMS=FAB ! Find the file desired looking at FNM/DNM and find file number in IFI - for tricky cases: RMS=(FAB,FWA)
SHOW PROC/RMS=(NOIFB:a, RAB) ! guess 'a' from relative file number, or from above, NOIFB selects one file only, without displaying IFAB ... which you might need to find actual in memory EOF
EXAM ufb;usz ! Display last record using RAB pointer/size. -
If that's too dynamic, then you may want to look at the last buffer(s).
SHOW PROCS/RMS=BDBSUM will show start addresses and VBN ranges for the last buffers.
If would require a fair bit of Kernel mode code to turn the above into a program, not impossible (and easier than writing a reliable intercept - right Brian!) but nobody has done this in a semi productized fashion best I know. Hmmm....
Yeah, a spare time project.
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
Dave Froble
2020-11-12 03:45:59 UTC
Reply
Permalink
Post by Hein RMS van den Heuvel
Read about the USEROPEN in a file OPEN statement. Basically (sic) the
FAB and RAB (I believe) are populated, then the USEROPEN routine,
provided by the programmer, can set or clear FAB and RAB fields, and
then do the actual file open.
Basic(ally) (sic), it's in there ....
Noop. Sorry. A program doesn't get that far!
As others indicated, the RRL (read Regardless of Lock) option is RECORD level for your GETs, but RMS will fail the open when the writer did not explicitly grant permission.
I stand corrected, I think. Could not find anything in the FAB
definition to bypass locking.

To do this, one would have to use QIO in the useropen routine, not
something I've ever tried, not sure it would work. I believe Craig
mentioned using QIO to access the file, thus bypassing the DLM.

Maybe I should stick to finishing the airplane ....
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Simon Clubley
2020-11-11 19:05:25 UTC
Reply
Permalink
Post by Craig A. Berry
(N.B. I have never seen nor used RSTS/E and have no idea what mode 4096
was.)
On RSTS/E, when you opened or created a file, you could specify
a bitmap-encoded 16-bit integer as a mode argument.

For example, when you created a file, you could set a bit which indicated
that you wanted the file created as a contiguous file.

When you opened a file, you could include 4096 in the mode argument to
indicate that you still wanted to open the file even if it was still
being created by another program.

The program behind various directory commands on RSTS/E was called PIP
and you could (1) invoke it directly and (2) specify a mode argument
when PIP was directly invoked. This meant you could type:

pip filename.dat/mo:4096

from the command line to examine a file even if it was still being
created by another program.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Simon Clubley
2020-11-11 19:11:09 UTC
Reply
Permalink
Post by Simon Clubley
Post by Craig A. Berry
(N.B. I have never seen nor used RSTS/E and have no idea what mode 4096
was.)
On RSTS/E, when you opened or created a file, you could specify
a bitmap-encoded 16-bit integer as a mode argument.
For example, when you created a file, you could set a bit which indicated
that you wanted the file created as a contiguous file.
When you opened a file, you could include 4096 in the mode argument to
indicate that you still wanted to open the file even if it was still
being created by another program.
The program behind various directory commands on RSTS/E was called PIP
That should say "behind various file and directory commands". The command
below was used to display the contents of filename.dat on the user's terminal.
Post by Simon Clubley
and you could (1) invoke it directly and (2) specify a mode argument
pip filename.dat/mo:4096
from the command line to examine a file even if it was still being
created by another program.
Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Stephen Hoffman
2020-11-10 17:19:05 UTC
Reply
Permalink
Post by Simon Clubley
Post by Dave Froble
Seems like VAX Basic has a "read regardless" lock qualifier ...
I wasn't aware of that. Does it allow you to open a sequential file for
reading while the file is also in the process of being created by
another program ? You can clearly do it for batch log files but what
about programs that don't specify that writing mode when creating a
sequential file ?
C has access to RRL, too.
But RRL doesn't help with a locked file.
RRL is applicable when a file is already open and already accessed, and
a record lock is to be overridden.
--
Pure Personal Opinion | HoffmanLabs LLC
Simon Clubley
2020-11-10 18:33:07 UTC
Reply
Permalink
Post by Stephen Hoffman
Post by Simon Clubley
Post by Dave Froble
Seems like VAX Basic has a "read regardless" lock qualifier ...
I wasn't aware of that. Does it allow you to open a sequential file for
reading while the file is also in the process of being created by
another program ? You can clearly do it for batch log files but what
about programs that don't specify that writing mode when creating a
sequential file ?
C has access to RRL, too.
But RRL doesn't help with a locked file.
RRL is applicable when a file is already open and already accessed, and
a record lock is to be overridden.
I thought David was talking about some option to get access to the
file contents while the file is being created, especially given that
he came from RSTS/E, so I assume he remembers what mode 4096 did on RSTS/E.

Thanks for explaining.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Dave Froble
2020-11-10 22:21:40 UTC
Reply
Permalink
Post by Simon Clubley
Post by Stephen Hoffman
Post by Simon Clubley
Post by Dave Froble
Seems like VAX Basic has a "read regardless" lock qualifier ...
I wasn't aware of that. Does it allow you to open a sequential file for
reading while the file is also in the process of being created by
another program ? You can clearly do it for batch log files but what
about programs that don't specify that writing mode when creating a
sequential file ?
C has access to RRL, too.
But RRL doesn't help with a locked file.
RRL is applicable when a file is already open and already accessed, and
a record lock is to be overridden.
I thought David was talking about some option to get access to the
file contents while the file is being created, especially given that
he came from RSTS/E, so I assume he remembers what mode 4096 did on RSTS/E.
Thanks for explaining.
Simon.
RMS definitely has provision for "open regardless" and "read
regardless". Whether or not they are used is another matter. The
capability exists.
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Tom Wade
2020-11-09 22:35:29 UTC
Reply
Permalink
Post by Simon Clubley
On RSTS/E, you can view the contents of a file opened for write by
pip filename.dat/mo:4096
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
If you want to read files that are locked by another process, check out
the Ralf utility at www.tomwade.eu/software

Ralf is written as a callable utility, but has a command line PEEK
[/page] program that displays a locked file. We used it extensively to
examine PMDF message files that were being processed (and therefore locked).


Tom Wade
tom dot wade at tomwade dot eu
Hein RMS van den Heuvel
2020-11-10 01:46:50 UTC
Reply
Permalink
Post by Tom Wade
Post by Simon Clubley
On RSTS/E, you can view the contents of a file opened for write by
pip filename.dat/mo:4096
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
If you want to read files that are locked by another process, check out
the Ralf utility at www.tomwade.eu/software
Ralf is written as a callable utility, but has a command line PEEK
[/page] program that displays a locked file. We used it extensively to
examine PMDF message files that were being processed (and therefore locked).
Tom Wade
tom dot wade at tomwade dot eu
Well, it will only be partially useful as many such files write-no-share file are written by RMS or an RTL actively buffering data to be written in incomplete chucks.
RMS by default could have an 8KB or 16 KB buffer only written when full. The most recent record will only exist in process memory. RMS has a minor backdoor to try an flush on exit, but I don't think there is a way to jiggle that conditions. So it could all be very disappointing.

You can verify with BACK/IGNORE=INTERLOCK whether it would or would not sufficiently solve a good part of the business needs.
I hope it is clear it will not at all be what folks expect and very hard to explain.

To properly solve this and similar problem you really need a system buffer.
RMS/VMS engineering spend upwards of 2 manyears to define 'stream' file access back in the 90ies but nothing practical transpired.
I believe most solution still ended up with the applications needing to 'do' something, which is the very thing that we all want to avoid.

As Brian says it may be easier hack the applications (Patch!) to initialize the fabs with sharing option and take the 'hit' of the locking overhead.

Cheers,
Hein
geze...@rlgsc.com
2020-11-10 04:47:49 UTC
Reply
Permalink
Post by Hein RMS van den Heuvel
Post by Tom Wade
Post by Simon Clubley
On RSTS/E, you can view the contents of a file opened for write by
pip filename.dat/mo:4096
What would be involved in adding a "read regardless" file open option
to VMS which would allow the opening of files for read only even if
they are already open for write, and then adding a qualifier to $ TYPE
to use this new option ?
If you want to read files that are locked by another process, check out
the Ralf utility at www.tomwade.eu/software
Ralf is written as a callable utility, but has a command line PEEK
[/page] program that displays a locked file. We used it extensively to
examine PMDF message files that were being processed (and therefore locked).
Tom Wade
tom dot wade at tomwade dot eu
Well, it will only be partially useful as many such files write-no-share file are written by RMS or an RTL actively buffering data to be written in incomplete chucks.
RMS by default could have an 8KB or 16 KB buffer only written when full. The most recent record will only exist in process memory. RMS has a minor backdoor to try an flush on exit, but I don't think there is a way to jiggle that conditions. So it could all be very disappointing.
You can verify with BACK/IGNORE=INTERLOCK whether it would or would not sufficiently solve a good part of the business needs.
I hope it is clear it will not at all be what folks expect and very hard to explain.
To properly solve this and similar problem you really need a system buffer.
RMS/VMS engineering spend upwards of 2 manyears to define 'stream' file access back in the 90ies but nothing practical transpired.
I believe most solution still ended up with the applications needing to 'do' something, which is the very thing that we all want to avoid.
As Brian says it may be easier hack the applications (Patch!) to initialize the fabs with sharing option and take the 'hit' of the locking overhead.
Cheers,
Hein
Hein,

I concur with You, Hoff, and Mike. It is difficult to well-define the desired functionality,

However, the OUTPUT_RATE on batch log files is a reasonable compromise. It is not too complex to implement an equivalent functionality, using a timer AST to flush out buffers and update end-of-file in an orderly fashion.

The drawback is that it is a library/source code change, and cannot be imposed after the fact on a running or pre-built image, although one could write an intercept for the various RMS calls.

- Bob Gezelter, http://www.rlgsc.com
V***@SendSpamHere.ORG
2020-11-10 16:21:57 UTC
Reply
Permalink
On 2020-11-09 13:30, Simon Clubley wrote:=20
On RSTS/E, you can view the contents of a file opened for write by=20
specifying mode 4096 as in:=20
=20
pip filename.dat/mo:4096=20
=20
What would be involved in adding a "read regardless" file open option=
=20
to VMS which would allow the opening of files for read only even if=20
they are already open for write, and then adding a qualifier to $ TYPE=
=20
to use this new option ?
If you want to read files that are locked by another process, check out=
=20
the Ralf utility at www.tomwade.eu/software=20
=20
Ralf is written as a callable utility, but has a command line PEEK=20
[/page] program that displays a locked file. We used it extensively to=20
examine PMDF message files that were being processed (and therefore locke=
d).=20
=20
=20
Tom Wade=20
tom dot wade at tomwade dot eu
Well, it will only be partially useful as many such files write-no-share fi=
le are written by RMS or an RTL actively buffering data to be written in in=
complete chucks.
RMS by default could have an 8KB or 16 KB buffer only written when full. T=
he most recent record will only exist in process memory. RMS has a minor ba=
ckdoor to try an flush on exit, but I don't think there is a way to jiggle =
that conditions. So it could all be very disappointing.
You can verify with BACK/IGNORE=3DINTERLOCK whether it would or would not s=
ufficiently solve a good part of the business needs.
I hope it is clear it will not at all be what folks expect and very hard to=
explain.
To properly solve this and similar problem you really need a system buffer.
RMS/VMS engineering spend upwards of 2 manyears to define 'stream' file ac=
cess back in the 90ies but nothing practical transpired.
I believe most solution still ended up with the applications needing to 'do=
' something, which is the very thing that we all want to avoid.
As Brian says it may be easier hack the applications (Patch!) to initialize=
the fabs with sharing option and take the 'hit' of the locking overhead.
... or simply correct the application(s) writing the logs to have read share.

Patching the application(s) would require locating the associated FAB which I
would have little problem doing but others may. I could probably hack RMS to
do it but I think I've done enough RMS hacking for more than one lifetime. ;)
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
Chris Townley
2020-11-10 15:43:40 UTC
Reply
Permalink
Post by V***@SendSpamHere.ORG
On 2020-11-09 13:30, Simon Clubley wrote:=20
On RSTS/E, you can view the contents of a file opened for write by=20
specifying mode 4096 as in:=20
=20
pip filename.dat/mo:4096=20
=20
What would be involved in adding a "read regardless" file open option=
=20
to VMS which would allow the opening of files for read only even if=20
they are already open for write, and then adding a qualifier to $ TYPE=
=20
to use this new option ?
If you want to read files that are locked by another process, check out=
=20
the Ralf utility at www.tomwade.eu/software=20
=20
Ralf is written as a callable utility, but has a command line PEEK=20
[/page] program that displays a locked file. We used it extensively to=20
examine PMDF message files that were being processed (and therefore locke=
d).=20
=20
=20
Tom Wade=20
tom dot wade at tomwade dot eu
Well, it will only be partially useful as many such files write-no-share fi=
le are written by RMS or an RTL actively buffering data to be written in in=
complete chucks.
RMS by default could have an 8KB or 16 KB buffer only written when full. T=
he most recent record will only exist in process memory. RMS has a minor ba=
ckdoor to try an flush on exit, but I don't think there is a way to jiggle =
that conditions. So it could all be very disappointing.
You can verify with BACK/IGNORE=3DINTERLOCK whether it would or would not s=
ufficiently solve a good part of the business needs.
I hope it is clear it will not at all be what folks expect and very hard to=
explain.
To properly solve this and similar problem you really need a system buffer.
RMS/VMS engineering spend upwards of 2 manyears to define 'stream' file ac=
cess back in the 90ies but nothing practical transpired.
I believe most solution still ended up with the applications needing to 'do=
' something, which is the very thing that we all want to avoid.
As Brian says it may be easier hack the applications (Patch!) to initialize=
the fabs with sharing option and take the 'hit' of the locking overhead.
... or simply correct the application(s) writing the logs to have read share.
Patching the application(s) would require locating the associated FAB which I
would have little problem doing but others may. I could probably hack RMS to
do it but I think I've done enough RMS hacking for more than one lifetime. ;)
Woudln't it be possible to change the batch processor to make log files
readable?

Chris
geze...@rlgsc.com
2020-11-10 15:54:57 UTC
Reply
Permalink
Post by Chris Townley
Post by V***@SendSpamHere.ORG
On 2020-11-09 13:30, Simon Clubley wrote:=20
On RSTS/E, you can view the contents of a file opened for write by=20
specifying mode 4096 as in:=20
=20
pip filename.dat/mo:4096=20
=20
What would be involved in adding a "read regardless" file open option=
=20
to VMS which would allow the opening of files for read only even if=20
they are already open for write, and then adding a qualifier to $ TYPE=
=20
to use this new option ?
If you want to read files that are locked by another process, check out=
=20
the Ralf utility at www.tomwade.eu/software=20
=20
Ralf is written as a callable utility, but has a command line PEEK=20
[/page] program that displays a locked file. We used it extensively to=20
examine PMDF message files that were being processed (and therefore locke=
d).=20
=20
=20
Tom Wade=20
tom dot wade at tomwade dot eu
Well, it will only be partially useful as many such files write-no-share fi=
le are written by RMS or an RTL actively buffering data to be written in in=
complete chucks.
RMS by default could have an 8KB or 16 KB buffer only written when full. T=
he most recent record will only exist in process memory. RMS has a minor ba=
ckdoor to try an flush on exit, but I don't think there is a way to jiggle =
that conditions. So it could all be very disappointing.
You can verify with BACK/IGNORE=3DINTERLOCK whether it would or would not s=
ufficiently solve a good part of the business needs.
I hope it is clear it will not at all be what folks expect and very hard to=
explain.
To properly solve this and similar problem you really need a system buffer.
RMS/VMS engineering spend upwards of 2 manyears to define 'stream' file ac=
cess back in the 90ies but nothing practical transpired.
I believe most solution still ended up with the applications needing to 'do=
' something, which is the very thing that we all want to avoid.
As Brian says it may be easier hack the applications (Patch!) to initialize=
the fabs with sharing option and take the 'hit' of the locking overhead.
... or simply correct the application(s) writing the logs to have read share.
Patching the application(s) would require locating the associated FAB which I
would have little problem doing but others may. I could probably hack RMS to
do it but I think I've done enough RMS hacking for more than one lifetime. ;)
Woudln't it be possible to change the batch processor to make log files
readable?
Chris
Chris,

Last time I checked, they were. The frequency of writing can be reset using the SET OUTPUT_RATE command (default is one minute interval).

- Bob Gezelter, http://www.rlgsc.com
Chris Townley
2020-11-10 16:14:40 UTC
Reply
Permalink
Post by ***@rlgsc.com
Post by Chris Townley
Post by V***@SendSpamHere.ORG
On 2020-11-09 13:30, Simon Clubley wrote:=20
On RSTS/E, you can view the contents of a file opened for write by=20
specifying mode 4096 as in:=20
=20
pip filename.dat/mo:4096=20
=20
What would be involved in adding a "read regardless" file open option=
=20
to VMS which would allow the opening of files for read only even if=20
they are already open for write, and then adding a qualifier to $ TYPE=
=20
to use this new option ?
If you want to read files that are locked by another process, check out=
=20
the Ralf utility at www.tomwade.eu/software=20
=20
Ralf is written as a callable utility, but has a command line PEEK=20
[/page] program that displays a locked file. We used it extensively to=20
examine PMDF message files that were being processed (and therefore locke=
d).=20
=20
=20
Tom Wade=20
tom dot wade at tomwade dot eu
Well, it will only be partially useful as many such files write-no-share fi=
le are written by RMS or an RTL actively buffering data to be written in in=
complete chucks.
RMS by default could have an 8KB or 16 KB buffer only written when full. T=
he most recent record will only exist in process memory. RMS has a minor ba=
ckdoor to try an flush on exit, but I don't think there is a way to jiggle =
that conditions. So it could all be very disappointing.
You can verify with BACK/IGNORE=3DINTERLOCK whether it would or would not s=
ufficiently solve a good part of the business needs.
I hope it is clear it will not at all be what folks expect and very hard to=
explain.
To properly solve this and similar problem you really need a system buffer.
RMS/VMS engineering spend upwards of 2 manyears to define 'stream' file ac=
cess back in the 90ies but nothing practical transpired.
I believe most solution still ended up with the applications needing to 'do=
' something, which is the very thing that we all want to avoid.
As Brian says it may be easier hack the applications (Patch!) to initialize=
the fabs with sharing option and take the 'hit' of the locking overhead.
... or simply correct the application(s) writing the logs to have read share.
Patching the application(s) would require locating the associated FAB which I
would have little problem doing but others may. I could probably hack RMS to
do it but I think I've done enough RMS hacking for more than one lifetime. ;)
Woudln't it be possible to change the batch processor to make log files
readable?
Chris
Chris,
Last time I checked, they were. The frequency of writing can be reset using the SET OUTPUT_RATE command (default is one minute interval).
- Bob Gezelter, http://www.rlgsc.com
So why cannot EDT read it?


Chris
Simon Clubley
2020-11-10 18:29:49 UTC
Reply
Permalink
Post by ***@rlgsc.com
Last time I checked, they were. The frequency of writing can be reset using the SET OUTPUT_RATE command (default is one minute interval).
Before everyone thinks that log files are the only thing you might
want to look at in this way, don't forget that there are a good range
of sequential files you might want to look at while they are being
created by some program that takes a long time to run.

Setting a log output rate will not help you in that case and neither
will changing the source code if you don't have access to the source
code for the program.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Jan-Erik Söderholm
2020-11-10 22:57:08 UTC
Reply
Permalink
Post by Simon Clubley
Post by ***@rlgsc.com
Last time I checked, they were. The frequency of writing can be reset using the SET OUTPUT_RATE command (default is one minute interval).
Before everyone thinks that log files are the only thing you might
want to look at in this way,...
And besides, what is the definition of a "log file" here?

Batch log files created by the quemgr for a job running on a batch
queue is just one sort of "log file". And those files are readable
(at least by TYPE) just fine.

I do not expect to be able to open a batch log file from a *running*
batch job in EDT, why would one want to do that? And it is simple to
do a TYPE/OUTPUT=<some file> and open *that* file in EDT.

If you want to know how far the job has come, a type of the batch
log is in most cases just fine.

Then there are "log files" that can we written by programming code
in some application. And that can have any number of quirks I guess,
depending on how that log file was opened.

For our main detached (Cobol) applications we have a C-routine that
we call with one log file entry (line) as one of the parameters, and
that file is written in a way so that it is always TYPE'able. The
detached process can also have a SYS$OUTPUT file, but that is not
readable until the process exits. That is why this seprate log file
was created. So we try to not DISPLAY anything from the Cobol source
but call this C logging routine instead.
Post by Simon Clubley
don't forget that there are a good range
of sequential files you might want to look at while they are being
created by some program that takes a long time to run.
If it takes a "long time" to run, I expect it to run in batch. And in
that case, the appliation can print a progress message to sys$output
that can we watched by TYPE'ing the normal batch log file.

10.000 records written to output file...
20.000 records written to output file...
30.000 records written to output file...
...
...
Post by Simon Clubley
Setting a log output rate will not help you in that case and neither
will changing the source code if you don't have access to the source
code for the program.
Simon.
Stephen Hoffman
2020-11-10 17:50:20 UTC
Reply
Permalink
Post by Chris Townley
Woudln't it be possible to change the batch processor to make log files
readable?
Yes. That's entirely possible. The change itself wouldn't be difficult.

There'll be some increase in system overhead associated, though that
increase is probably negligible in most environments.

That API change will break apps, as there are apps that are dependent
on encountering locked log files.

With any sufficiently complex software product, "no good deed goes
unpunished", q.v. Hyrum's Law, q.v. https://xkcd.com/1172/
--
Pure Personal Opinion | HoffmanLabs LLC
Michael Moroney
2020-11-10 02:10:31 UTC
Reply
Permalink
As others mentioned, the best way is to do $ SET OUTPUT_RATE <small#> for the batch job
you're looking at. If you can't do it in the submitted file, put it in the login.com
of the account under which the job is submitted.

I suppose another way is some $cmexec/$cmkrnl hack which pokes whatever field corresponds
to $ set output_rate or has the file opened in a sharable mode.
Loading...