Discussion:
Command to show process rms file opens?
(too old to reply)
David Hittner
2020-07-30 12:52:15 UTC
Permalink
What is the command to show all RMS files being opened while the process executes commands? I recall seeing this in an "openvms internals" presentation somewhere, but can't remember what facility does this or how to invoke it.

I have an MMS build that runs 2 hours 21 minutes, and would like to see if there's anything that I could do to speed up the file access, such as INSTALLing frequently accessed files and images, moving files to faster access media, etc. But in order to generate a solution to the the problem I need to know exactly which files are being accessed.
Arne Vajhøj
2020-07-30 13:00:30 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the
process executes commands? I recall seeing this in an "openvms
internals" presentation somewhere, but can't remember what facility
does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to
see if there's anything that I could do to speed up the file access,
such as INSTALLing frequently accessed files and images, moving files
to faster access media, etc. But in order to generate a solution to
the the problem I need to know exactly which files are being
accessed.
Is it:

$ SET WATCH FILE /CLASS=MAJOR

you are looking for?

Arne
David Hittner
2020-07-31 14:03:17 UTC
Permalink
Post by Arne Vajhøj
$ SET WATCH FILE /CLASS=MAJOR
you are looking for?
Arne
$ SET WATCH FILE was exactly what I was looking for.
Thanks!
David
Robert A. Brooks
2020-07-30 13:02:59 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the process
executes commands? I recall seeing this in an "openvms internals"
presentation somewhere, but can't remember what facility does this or how to
invoke it.
$ SET WATCH FILE /CLASS = MAJOR
Post by David Hittner
I have an MMS build that runs 2 hours 21 minutes, and would like to see if
there's anything that I could do to speed up the file access, such as
INSTALLing frequently accessed files and images, moving files to faster
access media, etc. But in order to generate a solution to the the problem I
need to know exactly which files are being accessed.
Change the build NOT to create any compiler listings and linker
maps. If that makes any appreciable difference, then it is clearly disk
I/O that's the issue. If it is, consider setting the disk volume(s)
to be /NOHIGHWATER, assuming that's a safe thing to do in your environment.
--
-- Rob
David Hittner
2020-07-31 14:10:48 UTC
Permalink
Post by Robert A. Brooks
Change the build NOT to create any compiler listings and linker
maps. If that makes any appreciable difference, then it is clearly disk
I/O that's the issue. If it is, consider setting the disk volume(s)
to be /NOHIGHWATER, assuming that's a safe thing to do in your environment.
--
-- Rob
I thought that I was not creating listing files and maps, but after a careful review, I found that I was still creating map files.
I will fix DESCRIP.MMS to not create map files in the next few days and see how much that affects the build.

I set /NOHIGHWATER on the volume as suggested since I am the only user of the system. Build times changed as follows:
HIGHWATER: 2:21:00
NOHIGHWATER: 1:59:00 (-15%)

Thanks for the suggestion!
David
Bob Gezelter
2020-07-31 18:21:28 UTC
Permalink
Post by David Hittner
Post by Robert A. Brooks
Change the build NOT to create any compiler listings and linker
maps. If that makes any appreciable difference, then it is clearly disk
I/O that's the issue. If it is, consider setting the disk volume(s)
to be /NOHIGHWATER, assuming that's a safe thing to do in your environment.
--
-- Rob
I thought that I was not creating listing files and maps, but after a careful review, I found that I was still creating map files.
I will fix DESCRIP.MMS to not create map files in the next few days and see how much that affects the build.
HIGHWATER: 2:21:00
NOHIGHWATER: 1:59:00 (-15%)
Thanks for the suggestion!
David
David,

I concur. RAID-5 is probably not the best choice. Since one is running a build, all of the outputs are recreatable.

Mirroring might be a performance assist. But not RAID-5.

- Bob Gezelter, http://www.rlgsc.com
V***@SendSpamHere.ORG
2020-08-01 09:50:11 UTC
Permalink
Post by David Hittner
Post by Robert A. Brooks
Change the build NOT to create any compiler listings and linker
maps. If that makes any appreciable difference, then it is clearly disk
I/O that's the issue. If it is, consider setting the disk volume(s)
to be /NOHIGHWATER, assuming that's a safe thing to do in your environment.
--
-- Rob
I thought that I was not creating listing files and maps, but after a careful review, I found that I was still creating map files.
I will fix DESCRIP.MMS to not create map files in the next few days and see how much that affects the build.
I would never NOT create .MAPs and .LIStings. YMMV.
Post by David Hittner
HIGHWATER: 2:21:00
NOHIGHWATER: 1:59:00 (-15%)
;)
--
VAXman- A Bored Certified VMS Kernel Mode Hacker VAXman(at)TMESIS(dot)ORG

I speak to machines with the voice of humanity.
Stephen Hoffman
2020-07-30 13:57:01 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the
process executes commands? I recall seeing this in an "openvms
internals" presentation somewhere, but can't remember what facility
does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to see
if there's anything that I could do to speed up the file access, such
as INSTALLing frequently accessed files and images, moving files to
faster access media, etc. But in order to generate a solution to the
the problem I need to know exactly which files are being accessed.
0: Use SSD.
1: Use recent MMK, as that tends to work better than all but recent
MMS. And use SSD.
2: Use MMK (or MMS) logging, and set up build parallelism, if you have
the cores and the HDD spindles. And use SSD.
3: Use SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. Did I mention using SSD?
4: If you're on HDD, use SSD. If you can't use SSD, don't saturate
glacial HDD spindles. And use SSD.
5: INSTALL and parallelism helps mask part of the glacial HDD speeds,
where SSD is massively faster than HDD.
6: I'd suggest using SSD here, too. Too many OpenVMS systems are still
using HDD.
--
Pure Personal Opinion | HoffmanLabs LLC
Simon Clubley
2020-07-30 17:24:13 UTC
Permalink
Post by Stephen Hoffman
6: I'd suggest using SSD here, too. Too many OpenVMS systems are still
using HDD.
1) When current SSDs fail, what is the sudden failure without warning rate
of those drives due to hardware component failure when compared to HDD units ?

2) When implementing hardware redundancy to handle the above issue, do
you mix enough different SSD manufacturers into your drive arrays so
that you can have every single drive from one manufacturer suddenly and
permanently fail without warning and still not suffer any data loss ?

SSDs are very useful for certain applications. They also have failure
modes that traditional drives do not.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
IanD
2020-07-30 17:44:06 UTC
Permalink
There's a number of detailed studies of SSD drives versus HDD reliability

https://www.usenix.org › filesPDF
A Study of SSD Reliability in Large Scale Enterprise Storage Deployments - Usenix

There is no definitive answer is one better than the other, horses for courses yet again it seems
Bob Gezelter
2020-07-30 18:45:43 UTC
Permalink
Post by IanD
There's a number of detailed studies of SSD drives versus HDD reliability
https://www.usenix.org › filesPDF
A Study of SSD Reliability in Large Scale Enterprise Storage Deployments - Usenix
There is no definitive answer is one better than the other, horses for courses yet again it seems
Ian,

Quite.

However, I must re-emphasize my previous comment: Do the science first.

I say that for a simple reason. I hate to be the person who spends funds for a hardware change, only to find that the problem is elsewhere. A Britishism I heard summarizes this well: Awkward that.

In my many years of practice, I have seen far too many situations where a performance problem was, in truth, nothing worse than an unconsidered, or inadequately in retrospect, considered quota or parameter.

WADR, switching HDD for SSD eliminates latency, and arm movement. No dispute. But performance may not increase if the working set was set at the wrong value. Similarly, we have seen more than a few episodes of poorly structured directories causing a performance problem on comp.os.vms.

Doing the science first ensures that one does not expend monetary or political capital unnecessarily.

- Bob Gezelter, http://www.rlgsc.com
Stephen Hoffman
2020-07-30 19:55:37 UTC
Permalink
Post by IanD
There's a number of detailed studies of SSD drives versus HDD reliability
https://www.usenix.org › filesPDF
A Study of SSD Reliability in Large Scale Enterprise Storage
Deployments - Usenix
Direct: https://www.usenix.org/system/files/fast20-maneas.pdf
Post by IanD
There is no definitive answer is one better than the other, horses for
courses yet again it seems
Ayup.

All storage can and will inevitably fail, which means backups, RAID for
access-critical data, and potentially more. HDDs throw EDC-related
READERROR errors, too.

Wear-monitoring and age on SSD, too, same as tracking hard disk errors.

The I/O performance boost from HDD to SSD is somewhere between
substantial and massive.

Here's a high-level intro to some of the considerations, for those
interested: https://www.backblaze.com/blog/how-reliable-are-ssds/

Here are some Google findings from a few years ago:
http://0b4af6cdc2f0c5998459-c0245c5c937c5dedcca3f1764ecc9b2f.r43.cf2.rackcdn.com/23105-fast16-papers-schroeder.pdf

For the highlights, skip to the summary on page 79; about twelve pages
in—the paper is not 79 pages long.

"Understanding endurance and performance characteristics of HP solid
state drives" (old info!)
https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c03312456

HPE has an SSD selector tool, as they have a number of
differently-targeted SSDs...
https://ssd.hpe.com

And just like everything else in computing, there are better-quality
devices, and there's also junk.

And yes, there are bugs:
https://www.theregister.com/2020/03/25/hpe_ssd_death_fix/
--
Pure Personal Opinion | HoffmanLabs LLC
Simon Clubley
2020-07-31 12:29:56 UTC
Permalink
Post by IanD
There's a number of detailed studies of SSD drives versus HDD reliability
https://www.usenix.org ? filesPDF
A Study of SSD Reliability in Large Scale Enterprise Storage Deployments - Usenix
There is no definitive answer is one better than the other, horses for courses yet again it seems
Both individual SSDs and HDDs fail over time and you need to plan for
that via backups and online redundancy. It is somewhat alarming to read
reports about individual SSDs failing hard without any chance of data
recovery but that should be covered by your backup and online redundancy
procedures.

What _is_ a MAJOR concern for me with SSDs however is how an entire batch
of them can all suddenly fail permanently at pretty much the same time
through no fault of your own, rendering your online redundancy procedures
pretty such useless and making you vulnerable to data loss if even you
implement something like RAID or HBVS for your SSD devices.

There is no way that those types of bugs should exist in a device as
important as a SSD and the SSD firmware should be written in a way
that allows you access to your data even if such a bug gets through
testing and is triggered.

The only way around this at the moment is to make sure that you mix
a variety of different manufacturers and different ages of SSDs in
your drive arrays.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Craig A. Berry
2020-07-30 21:56:29 UTC
Permalink
Post by Stephen Hoffman
Post by David Hittner
What is the command to show all RMS files being opened while the
process executes commands? I recall seeing this in an "openvms
internals" presentation somewhere, but can't remember what facility
does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to
see if there's anything that I could do to speed up the file access,
such as INSTALLing frequently accessed files and images, moving files
to faster access media, etc. But in order to generate a solution to
the the problem I need to know exactly which files are being accessed.
In addition to the SET WATCH FILE already mentioned, you can see what's
currently open with SHOW DEVICE/FILE and, in SDA, SHOW PROCESS/CHANNEL.
Post by Stephen Hoffman
0: Use SSD.
Or DECram for some or all of the build, depending on the size of a dirty
build tree and how much memory you have. You didn't say what language,
but if it's C or C++, the name mangler database is a very hot file and
IIRC there is a way to point it elsewhere. Putting .OLB and/or .TLB
files referenced by the build on a RAM disk might also make a difference.
Post by Stephen Hoffman
1: Use recent MMK, as that tends to work better than all but recent MMS.
And use SSD.
2: Use MMK (or MMS) logging, and set up build parallelism, if you have
the cores and the HDD spindles. And use SSD.
Consider contributing a /PARALLEL=N feature to MMK equivalent to make
-j. Most build systems these days expect to be able to keep multiple
cores busy, but you can't do that on VMS that I'm aware of. This
assuming that disk I/O is not your only problem, which may be wrong.
Post by Stephen Hoffman
3: Use SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. Did I mention using SSD?
4: If you're on HDD, use SSD. If you can't use SSD, don't saturate
glacial HDD spindles. And use SSD.
5: INSTALL and parallelism helps mask part of the glacial HDD speeds,
where SSD is massively faster than HDD.
6: I'd suggest using SSD here, too. Too many OpenVMS systems are still
using HDD.
Terry Kennedy
2020-07-31 08:54:00 UTC
Permalink
Post by Stephen Hoffman
0: Use SSD.
I admit to not being up on current HP offerings, but don't you need to have a quite recent Itanium system to have SAS/SATA support? And are there HP qualified SSDs available for that hardware? Otherwise we're looking at either external storage (FC SAN) or hard-to-obtain and unsupported controllers (per recent discussions in c.o.v) or out-of-production adapter cards from parallel SCSI or PATA to whatever interfaces your SSDs have. Given the unsupported nature of those, or the cost to switch to FC SAN that supports SSDs, it would seem prudent to do some research to make sure that throwing SSDs at the problem will improve performance enough to be worthwhile (see my earlier replies fo reasons it might not be).

Once VSI releases production grade x86 VMS, a host with some NVMe storage (some of which is capable of 4Gbyte/second data rates) passed through to VMS in a guest VM should give us a better idea of what modern VMS on modern hardware can accomplish. That's one good justification for running in a VM instead of on bare metal- the hypervisor can make the real hardware available to the VM guest(s) while mapping it to a device type supported by the client OS. It helps if the client OS has some understanding of the actual hardware, even if not the exact model, so it can send Trim/Unmap commands to SSDs to take advantage of the performance benefits. I have seen firsthand how well device type remapping works - in my testing of the AlphaVM Pro emulator I was able to map a physical SATA disk to a parallel SCSI drive on the emulated Alpha, even though the command sets are utterly different.
David Hittner
2020-07-31 14:42:42 UTC
Permalink
On Thursday, July 30, 2020 at 9:57:04 AM UTC-4, Stephen Hoffman wrote:
.
Post by Stephen Hoffman
0: Use SSD.
1: Use recent MMK, as that tends to work better than all but recent
MMS. And use SSD.
2: Use MMK (or MMS) logging, and set up build parallelism, if you have
the cores and the HDD spindles. And use SSD.
3: Use SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. Did I mention using SSD?
4: If you're on HDD, use SSD. If you can't use SSD, don't saturate
glacial HDD spindles. And use SSD.
5: INSTALL and parallelism helps mask part of the glacial HDD speeds,
where SSD is massively faster than HDD.
6: I'd suggest using SSD here, too. Too many OpenVMS systems are still
using HDD.
--
Pure Personal Opinion | HoffmanLabs LLC
Your suggestion of SSD is duly noted. LOL. Hilarious overemphasis!
I'll see if I can find a low-cost SSD acceptable to the controller and OpenVMS for an experiment.
Caveat: Will use the recommended analysis from Bob Gezelter first.

I didn't know that you could set up build parallelism in MMS. I'll have to look up how to do that.
Sadly, MMK will not work with my DESCRIP.MMS build. I'm corresponding with the MMK maintainer about it.

I do plan on INSTALLing any tools used in the build that are not installed to see if the speed will increase.

The system in question is:
Integrity rx2660, 2x 2C CPUs, hyperthreading enabled (8 virtual CPUs)
P800 SAS Controller w/ 512 cache, write cache currently disabled due to low battery voltage
7x 146GB 10K SAS drives in a RAID-5 array
2x logical drives built on the RAID-5 array, System drive is 25%, Data drive is 75%
HPE OpenVMS 8.4 U7
HPE MMS 12.8
Oracle RDB 7.2.5
Most current HPE FORTRAN, FORMS, C
Build sources (2000+ files): FORTRAN, FORMS, C, SQL/Fortran, SQL/C, SQL/Module

Thanks for the comments and food for thought!
David
Chris Scheers
2020-07-31 16:45:25 UTC
Permalink
Post by David Hittner
.
Post by Stephen Hoffman
0: Use SSD.
1: Use recent MMK, as that tends to work better than all but recent
MMS. And use SSD.
2: Use MMK (or MMS) logging, and set up build parallelism, if you have
the cores and the HDD spindles. And use SSD.
3: Use SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. Did I mention using SSD?
4: If you're on HDD, use SSD. If you can't use SSD, don't saturate
glacial HDD spindles. And use SSD.
5: INSTALL and parallelism helps mask part of the glacial HDD speeds,
where SSD is massively faster than HDD.
6: I'd suggest using SSD here, too. Too many OpenVMS systems are still
using HDD.
--
Pure Personal Opinion | HoffmanLabs LLC
Your suggestion of SSD is duly noted. LOL. Hilarious overemphasis!
I'll see if I can find a low-cost SSD acceptable to the controller and OpenVMS for an experiment.
Caveat: Will use the recommended analysis from Bob Gezelter first.
I didn't know that you could set up build parallelism in MMS. I'll have to look up how to do that.
Sadly, MMK will not work with my DESCRIP.MMS build. I'm corresponding with the MMK maintainer about it.
I do plan on INSTALLing any tools used in the build that are not installed to see if the speed will increase.
Integrity rx2660, 2x 2C CPUs, hyperthreading enabled (8 virtual CPUs)
P800 SAS Controller w/ 512 cache, write cache currently disabled due to low battery voltage
7x 146GB 10K SAS drives in a RAID-5 array
2x logical drives built on the RAID-5 array, System drive is 25%, Data drive is 75%
HPE OpenVMS 8.4 U7
HPE MMS 12.8
Oracle RDB 7.2.5
Most current HPE FORTRAN, FORMS, C
Build sources (2000+ files): FORTRAN, FORMS, C, SQL/Fortran, SQL/C, SQL/Module
Thanks for the comments and food for thought!
David
I understand the pain of the P800 battery.

However, the lack of the write cache on the P800 will hurt you big time.

Also, the HP 300GB (and even 600GB) drives are very inexpensive these
days. They are also considerably faster (even without cache) than the
146GB drives.

You could replace your drives with 8x300GB drives configured as shadowed
stripe sets (not RAID-5) and you would have much better performance and
still have redundancy.

Good luck!
--
-----------------------------------------------------------------------
Chris Scheers, Applied Synergy, Inc.

Voice: 817-237-3360 Internet: ***@applied-synergy.com
Fax: 817-237-3074
Stephen Hoffman
2020-07-31 22:42:15 UTC
Permalink
Post by Chris Scheers
Post by David Hittner
Integrity rx2660, 2x 2C CPUs, hyperthreading enabled (8 virtual CPUs)
P800 SAS Controller w/ 512 cache, write cache currently disabled due to
low battery voltage
7x 146GB 10K SAS drives in a RAID-5 array
That's a very big performance hit right there, non-cached RAID-5 HDD.
Post by Chris Scheers
Post by David Hittner
2x logical drives built on the RAID-5 array, System drive is 25%, Data drive is 75%
HPE OpenVMS 8.4 U7
HPE MMS 12.8
...
I understand the pain of the P800 battery.
However, the lack of the write cache on the P800 will hurt you big time.
Also, the HP 300GB (and even 600GB) drives are very inexpensive these
days. They are also considerably faster (even without cache) than the
146GB drives.
You could replace your drives with 8x300GB drives configured as
shadowed stripe sets (not RAID-5) and you would have much better
performance and still have redundancy.
Yep.

Going price for an HPE SFF 240 GB SATA SSD is USD$350. Might be able to
find those cheaper, that was a _very_ quick look.

But I'd fix the dead cache battery and the use of RAID-5 first. Fix
those issues, and add a shelf of SSDs maybe demoting the HDDs to
archival and backup use, and this box will be vastly faster.

For approximately no software effort.

Yeah, this could be CPU or memory constraints too, and T4 or SPM or
otherwise can help with that determination, but non-cached HDDs on
RAID-5 are almost certainly going to be I/O saturated.

MONITOR DISK/ITEM=QUEUE past 0.5 is bad news, and I'll wager that
you're far past 0.5 during a build.
--
Pure Personal Opinion | HoffmanLabs LLC
Bob Gezelter
2020-07-30 16:00:26 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the process executes commands? I recall seeing this in an "openvms internals" presentation somewhere, but can't remember what facility does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to see if there's anything that I could do to speed up the file access, such as INSTALLing frequently accessed files and images, moving files to faster access media, etc. But in order to generate a solution to the the problem I need to know exactly which files are being accessed.
David,

I would suggest starting with MONITOR (or run T4 which runs MONITOR to gather data) to see what the bottleneck is.

There are many performance knobs that be adjusted, but the operative question is: What is the bottleneck?

It could be I/O. It could be paging? It could (less likely) be CPU saturation? WADR, do not speculate. Measure utilization and make the required adjustments.

- Bob Gezelter, http://www.rlgsc.com
Terry Kennedy
2020-07-31 08:32:21 UTC
Permalink
Post by Bob Gezelter
I would suggest starting with MONITOR (or run T4 which runs MONITOR to gather data) to see what the bottleneck is.
There are many performance knobs that be adjusted, but the operative question is: What is the bottleneck?
It could be I/O. It could be paging? It could (less likely) be CPU saturation? WADR, do not speculate. Measure utilization and make the required adjustments.
I seem to recall a DEC product named something like "Performance Data Collector" that added profiling to apps at a much more fine-grained level than what you get from MONITOR or WATCH. However, I don't know if it made the transition from VAX to Alpha or IA64. It may have required the application be built with profiling support, which would make it useless if you are trying to profile system compilers/utilities. Also, I think it was one of the first products in the Great Selloff and I don't know where it ended up or if is still available.

Some performance problems are just VMSisms. As an example, Digital Canada contracted me to write a MOP server for RSTS/E. I wrote it entirely in BP2 except for a tiny Macro-11 part to handle the few things not available in BP2, like enabling Ethernet multicast groups. It turned out to be faster on an 11/23 than on some high-end VAX like an 8800, so they asked me to add a configurable delay before the RSTS/E system would respond, to give the VAX a chance. https://www.glaver.org/ftp/tmk-software/mop-server

I also used to evaluate preproduction hardware for 3rd parties. One such product was an early 256MB SSD, built out of DRAM memory boards with a hard drive to handle nonvolatile save/restore. Even connected directly to the 8650's backplane, it didn't provide a lot of performance improvement, even fordisk-intensiveworkloads. Running 4BSD on the same or lesser model VAX often provided a 3x or more speedup, even without the SSD. But of course you give up all of the nice VMS features. This was all on VAX era hardware and may not be relevant any more, but as an anecdote when Itanium systems came out I compared performance of the admittedly low-end RX2620 VMS to some PCs I got from a dumpster dive running FreeBSD and the PCs outperformed the Itanium by a huge amount, while using a fraction of the power. Based on that, I recommended that applications that used a bunch of VMS-specific features should remain on VMS, but everything else should be moved to commodity hardware/OS.

It will be interesting to compare the future production ready release of VSI's x86 port, using optimized versions of compilers, with FreeBSD or Linux on the same hardware. Since they will all be using the LLVM backend, this should provide the most detailed "apple's to apples" performance comparison to date.
Simon Clubley
2020-07-31 12:39:00 UTC
Permalink
Post by Terry Kennedy
It will be interesting to compare the future production ready release of VSI's x86 port, using optimized versions of compilers, with FreeBSD or Linux on the same hardware. Since they will all be using the LLVM backend, this should provide the most detailed "apple's to apples" performance comparison to date.
While it's not compiler related, the performance data I will be interested
in seeing is how much overhead there is with mapping the KESU modes to an
architecture with only KU modes, both when PCID is and is not available.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
David Hittner
2020-07-31 15:10:30 UTC
Permalink
Post by Bob Gezelter
I would suggest starting with MONITOR (or run T4 which runs MONITOR to gather data) to see what the bottleneck is.
There are many performance knobs that be adjusted, but the operative question is: What is the bottleneck?
It could be I/O. It could be paging? It could (less likely) be CPU saturation? WADR, do not speculate. Measure utilization and make the required adjustments.
- Bob Gezelter, http://www.rlgsc.com
Thanks for the reminder to do an analysis and gather detailed metrics before haphazardly changing things.

I would tend to think that it's an I/O bottleneck, because a slower rx2620 with user files on a dual-channel external 8GB FC RAID array could execute this build in 45 minutes, but I will check. I copied the username from the default SYSTEM account (HPE OpenVMS 8.4 U7) after setting up the system, so it should have fairly decent working set sizes - but again, I'll check.

It could just be that I need to fix the write cache on the RAID controller by getting new batteries to get the I/O speed back up.

David
Bob Gezelter
2020-07-31 15:21:41 UTC
Permalink
Post by David Hittner
Post by Bob Gezelter
I would suggest starting with MONITOR (or run T4 which runs MONITOR to gather data) to see what the bottleneck is.
There are many performance knobs that be adjusted, but the operative question is: What is the bottleneck?
It could be I/O. It could be paging? It could (less likely) be CPU saturation? WADR, do not speculate. Measure utilization and make the required adjustments.
- Bob Gezelter, http://www.rlgsc.com
Thanks for the reminder to do an analysis and gather detailed metrics before haphazardly changing things.
I would tend to think that it's an I/O bottleneck, because a slower rx2620 with user files on a dual-channel external 8GB FC RAID array could execute this build in 45 minutes, but I will check. I copied the username from the default SYSTEM account (HPE OpenVMS 8.4 U7) after setting up the system, so it should have fairly decent working set sizes - but again, I'll check.
It could just be that I need to fix the write cache on the RAID controller by getting new batteries to get the I/O speed back up.
David
David,

While you are at it, check the RMS defaults. File extends are very elapsed time extensive. Also, increasing buffering factors for sequential files can have large impacts, particularly on high-activity disks. Read ahead and write behind can yield major improvements in elapsed time.

Disk speed (or more embraceivly, mass storage performance) can cover a multitude of configuration choices. In my experience, many cases benefit from tuning. Many years ago, I amused many with what one could get out of MV 3100-38 with a single disk (or for that matter, an 11/750) with proper tuning.

- Bob Gezelter, http://www.rlgsc.com
Roy Omond
2020-07-31 15:58:24 UTC
Permalink
Post by David Hittner
It could just be that I need to fix the write cache on the RAID controller by getting new batteries to get the I/O speed back up.
David
I see you posted your configuration after I had sent my response ... ok.

Your disabled write cache on the RAID controller is going to hit you
very hard, especially for your RAID-5 configuration.

You definitely need to get the new batteries installed, and the write
cache back in operation.
Roy Omond
2020-07-31 14:52:26 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the process executes commands? I recall seeing this in an "openvms internals" presentation somewhere, but can't remember what facility does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to see if there's anything that I could do to speed up the file access, such as INSTALLing frequently accessed files and images, moving files to faster access media, etc. But in order to generate a solution to the the problem I need to know exactly which files are being accessed.
You've already received a number of useful suggestions ...

Re "moving files to faster access media" - read on.

However you don't describe your system configuration in any way, so
what I write here may or may not be pertinent to your environment.

I'd say that most VMS users and even most system managers are unaware
that VMS explicitly disables disk caching for directly-connected SCSI
disks (yeah, I know many will not be using such disks, but many will).

At a few sites I've supported over the years until I retired at the
beginning of this month, I've achieved *huge* I/O performance
improvements by *enabling* on-board disk caching (this only applies
to SCSI-connected disks). Seriously *huge* improvements ...

I am well aware that VMS likes to be sure that I/O writes have reached
the actual platter before considering the I/O write to be complete.
But this comes at a cost. I would guess that most system managers and
power users would readily accept the tiny risk (for some subset of
disks) associated with the possibility that the disk might hiccup just
when a write has been acknowledged by the disk but before the data are
actually on the platter.

Here's how to enable the on-board disk cache (note: the disk must have
already been mounted, since VMS disables it explicitly on mounting):

Example (an 18 Gbyte Compaq disk):

$ mcr sys$etc:scsi_mode -devname $2$dkb400 -page 8 -offset 0e 14 -mount
-devtype "bf018863b4"

The important "bit" is the "4" at byte offset 0e in page 8 of the SCSI
mode pages. Setting the bit on enables the on-board disk cache.

I don't have access to such a system at this moment, so I can't provide
any benchmark test results - any volunteers ? My observations, though,
have always been "wow, that's very impressive".

Note also that has often been this "caution" that has given rise to the
perception that VMS I/O is slow (other systems have been running with
disk caching enabled).
Hein RMS van den Heuvel
2020-08-03 14:38:34 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the process executes commands?
Many good answers have been given.
SET WATCH FILE/CLASS=MAJOR specifically to the question, and more good answer towards the real problem you are trying to solve.

In a later reply you indicated an Integrity server, therefor a version of VMS from this century with XFC enabled. The XFC should take care of the caching of the input files, such as Object libraries, text libraries and include files.
You can verify the effectiveness of the XFC and get more insight into the hotfiles by using $SHOW MEM/CACHE=(TOPQIO=50,VOLUME=xxx)
Note, since you XFC has probably been collecting data for a long time, you may want to run this twice 1) just before the build and 2) well into the build and find the incremental numbers by subtraction. - or just reboot before a test build.

RMS defaults typically don't do too much but why not try:
$ SET RMS /BUF=4/BLOCK=64 ! Possibly some more read-ahead, stay under XFC max IO.
More likely to make a noticeable change is:
$ SET RMS / EXTEND=1000 ! filesystem default is a piddly 5, RMS will ask at least for a buffer (16 or 32 block) at a time, but the actual minimum is the volume CLUSTER SIZE which for a typical raid device is in the 100+ range. Specifying anything less than cluster size will not have an effect.

You have to decide whether you actually ever used the listings or maps, or think you can recreate them exactly as needed with a rebuild. If you've never used them, then don't make them! The fastest IO is an IO never done.

DECram should be considered for intermediate files.
Notably object files which are moved to a library or just linked and deleted.
You may want to build everything onto DECram and then use BACKUP to copy out what you need on successful build.

And yes, you have plenty of resources, so go parallel preferably within the (MMS) tool, or by splitting up the build if dependencies allow that.

Cheers,
Hein.
Roy Omond
2020-08-03 16:03:07 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the process executes commands?
[snip]
You can verify the effectiveness of the XFC and get more insight into the hotfiles by using $SHOW MEM/CACHE=(TOPQIO=50,VOLUME=xxx)
Note, since you XFC has probably been collecting data for a long time, you may want to run this twice 1) just before the build and 2) well into the build and find the incremental numbers by subtraction. - or just reboot before a test build.
[snip]
If David has sufficient privileges to reboot, he'll find it easier to just:

$ set cache/reset

Otherwise agreed with all that Hein wrote.
Hein RMS van den Heuvel
2020-08-03 21:04:52 UTC
Permalink
Post by Roy Omond
$ set cache/reset
Otherwise agreed with all that Hein wrote.
I wish. Last time I looked (OpenVMS 8.4) that only clears the highest level counters, down to the per IO size histogram.
The SET CACHE/RESET does not clear the per file data, not even the per volume data.

Instead of reboot, dismount/remount will of course clear any XFC counters for the volume.

Note - during a quick test before writing this I could not get $SHOW MEM/CACHE=(TOPQIO=50,VOLUME=DATA) to work for my 'data' drive. It just returned emptiness. $SHOW MEM/CACHE=(VOLUME=DATA) did show (moving) counters.
It worked fine asking about the file itself: $SHOW MEM/CACHE=(FILE=DATA:[X].X).
Hmm, odd. Plain Alpha 8.4 on FreeAXP.

Hein
^P
2020-08-05 14:39:22 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the process executes commands? I recall seeing this in an "openvms internals" presentation somewhere, but can't remember what facility does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to see if there's anything that I could do to speed up the file access, such as INSTALLing frequently accessed files and images, moving files to faster access media, etc. But in order to generate a solution to the the problem I need to know exactly which files are being accessed.
Maybe you're looking for; enter SDA, SET PROC to the PID of the process you wish to study, then SHOW PROC/CHANNEL if I remember correct

^P
Simon Clubley
2020-08-08 00:25:44 UTC
Permalink
Post by ^P
Post by David Hittner
What is the command to show all RMS files being opened while the process executes commands? I recall seeing this in an "openvms internals" presentation somewhere, but can't remember what facility does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to see if there's anything that I could do to speed up the file access, such as INSTALLing frequently accessed files and images, moving files to faster access media, etc. But in order to generate a solution to the the problem I need to know exactly which files are being accessed.
Maybe you're looking for; enter SDA, SET PROC to the PID of the process you wish to study, then SHOW PROC/CHANNEL if I remember correct
^P
VMS really needs a VMS version of RSTS/E's systat/w supplied as part of
VMS and for it to include the locking information as well (as systat/w
did IIRC).

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Dave Froble
2020-08-08 01:55:16 UTC
Permalink
Post by Simon Clubley
Post by ^P
Post by David Hittner
What is the command to show all RMS files being opened while the process executes commands? I recall seeing this in an "openvms internals" presentation somewhere, but can't remember what facility does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to see if there's anything that I could do to speed up the file access, such as INSTALLing frequently accessed files and images, moving files to faster access media, etc. But in order to generate a solution to the the problem I need to know exactly which files are being accessed.
Maybe you're looking for; enter SDA, SET PROC to the PID of the process you wish to study, then SHOW PROC/CHANNEL if I remember correct
^P
VMS really needs a VMS version of RSTS/E's systat/w supplied as part of
VMS and for it to include the locking information as well (as systat/w
did IIRC).
Simon.
Simon,

I believe that I already have some programs that do some of what you ask.

It's been a very long time since RSTS for me. Like 1978. What? 42 years?

If you'd care to come up with some specs, what info should be
selectable, how it should be displayed, I might find the energy to dust
off some of my old stuff, come up with a utility you might like, donate
it to VSI, or if they didn't want it, make it available similar to DECUS
stuff.

VMS aleady has some things in the MONITOR utility.
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Simon Clubley
2020-08-08 02:10:50 UTC
Permalink
Post by Dave Froble
Post by Simon Clubley
VMS really needs a VMS version of RSTS/E's systat/w supplied as part of
VMS and for it to include the locking information as well (as systat/w
did IIRC).
Simon,
I believe that I already have some programs that do some of what you ask.
It's been a very long time since RSTS for me. Like 1978. What? 42 years?
If you'd care to come up with some specs, what info should be
selectable, how it should be displayed, I might find the energy to dust
off some of my old stuff, come up with a utility you might like, donate
it to VSI, or if they didn't want it, make it available similar to DECUS
stuff.
VMS aleady has some things in the MONITOR utility.
I did a long-lost version of this for a former employer and it required
an intimate understanding of VMS data structure internals as you needed
to walk through them to get the required information.

Basically sys/w was the command to list the open files, who had
them open (and at what block). My version included the current
locks information for each file which was taken from the RMS locks
in the DLM (IIRC).

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Dave Froble
2020-08-08 02:34:40 UTC
Permalink
Post by Simon Clubley
Post by Dave Froble
Post by Simon Clubley
VMS really needs a VMS version of RSTS/E's systat/w supplied as part of
VMS and for it to include the locking information as well (as systat/w
did IIRC).
Simon,
I believe that I already have some programs that do some of what you ask.
It's been a very long time since RSTS for me. Like 1978. What? 42 years?
If you'd care to come up with some specs, what info should be
selectable, how it should be displayed, I might find the energy to dust
off some of my old stuff, come up with a utility you might like, donate
it to VSI, or if they didn't want it, make it available similar to DECUS
stuff.
VMS aleady has some things in the MONITOR utility.
I did a long-lost version of this for a former employer and it required
an intimate understanding of VMS data structure internals as you needed
to walk through them to get the required information.
Been there, done that.
Post by Simon Clubley
Basically sys/w was the command to list the open files, who had
them open (and at what block). My version included the current
locks information for each file which was taken from the RMS locks
in the DLM (IIRC).
Simon.
Go to freeware, 6 or 8 I think, and look for RMS_LOCKS

But that's just locking, not what else is happening is a process.

Original purpose was to search for deadlocks ...
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Stephen Hoffman
2020-08-08 23:47:04 UTC
Permalink
Post by Simon Clubley
VMS really needs a VMS version of RSTS/E's systat/w supplied as part of
VMS and for it to include the locking information as well (as systat/w
did IIRC).
DECamds / Availability Manager is handy for chasing locks around a host
or around a cluster.

In the absence of that, the SDA LCK extension can be helpful.
--
Pure Personal Opinion | HoffmanLabs LLC
David Hittner
2020-09-09 17:32:15 UTC
Permalink
To finish up this topic, I'll put in a summary of the information that most significantly impacted my MMS build times.

It turns out that I didn't really need the command to show files accessed. While interesting, the excessive build time problem was really (lack of) raw IO performance, as many might have guessed.

While setting the volume to NOHIGHWATER helped significantly, it isn't necessarily worth the security cost. As Hoff pointed out, "4. Use SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. Did I mention using SSD?" made the most difference.

Thanks to all for the advice. It cost me $600 for a pair of SSDs, but what a difference it makes to the MMS build time!

-------------

Original Hardware: rx2660, 7x 146GB 10K 3G SAS drives, in RAID5 split 25%/75% into two logical drives (DKB0, DKB1) on a P800 3G controller with write cache disabled due to bad batteries.

Final Hardware: rx2660, 2x 200GB 6G SSD in RAID1 (DKB0), 5x 146GB 10K 3G SAS drives in RAID5 (DKB1) on a P810 6G controller with (capacitor-based) write cache enabled.

MMS build: approximately 3500 source files, generating approximately 3700 intermediate and executable files.

Original build time (on HDD DKB1): 02:21:00.00
$SET VOL DKB1: /NOHIGHWATER: 01:59.00.00
Rebuilt DKB0 with SSD and DKB1
with HDD on P800, moved home
directory from DKB1 to DKB0(SSD),
reset all drives to /HIGHWATER: 00:27:18.00
Replaced P800 with P810, changes
3G max to 6G max, and enables
write cache: 00:24:00.00

Comparison with reference rx2620
system with all storage on
external 8G fiber channel
MSA2024 HDD arrays: 00:23.30.00
Stephen Hoffman
2020-09-09 18:05:05 UTC
Permalink
Post by David Hittner
While setting the volume to NOHIGHWATER helped significantly, it isn't
necessarily worth the security cost. As Hoff pointed out, "4. Use SSD.
SSD. SSD. SSD. SSD. SSD. SSD. SSD. SSD. Did I mention using SSD?" made
the most difference.
FWIW: The highwater setting matters on OpenVMS in a different way in
this context, as the OpenVMS I/O implemetation lacks TRIM support, and
the Highwater sector erasure is what then triggers the TRIM in some SSD
firmware implementations.

Far from all SSD firmware implements the erased-sector-is-TRIM
processing, though.

https://en.wikipedia.org/wiki/Trim_(computing)
--
Pure Personal Opinion | HoffmanLabs LLC
Jon Schneider
2020-09-10 09:41:23 UTC
Permalink
Post by David Hittner
Final Hardware: rx2660, 2x 200GB 6G SSD in RAID1 (DKB0), 5x 146GB 10K 3G SAS drives in RAID5 (DKB1) on a P810 6G controller with (capacitor-based) write cache enabled.
I'd just like to make the point that all dynamic RAM, all flash as well
as older technologies like EEPROM/EPROM and others are capacitor based.

Unless you mean some volatile (probably capacitor based) memory powered
by capacitor as opposed to battery.

At leat we know it's not core.

Jon
e***@gmail.com
2020-09-10 09:55:43 UTC
Permalink
Post by Jon Schneider
Post by David Hittner
Final Hardware: rx2660, 2x 200GB 6G SSD in RAID1 (DKB0), 5x 146GB 10K 3G SAS drives in RAID5 (DKB1) on a P810 6G controller with (capacitor-based) write cache enabled.
Unless you mean some volatile (probably capacitor based) memory powered
by capacitor as opposed to battery.
The P810 RAID controller has RAM on board. It'll happily run read caching as is. It prefers/requires (can't recall which) protection of the memory to enable write caching.

There are two options that give you this Battery Backed Write Cache and Flash Backed Write Cache. The first just keeps the memory hot, the second dumps memory to flash in the event of power failure and is capacitor based.
Edgar Ulloa
2020-09-11 02:05:57 UTC
Permalink
Post by David Hittner
What is the command to show all RMS files being opened while the process executes commands? I recall seeing this in an "openvms internals" presentation somewhere, but can't remember what facility does this or how to invoke it.
I have an MMS build that runs 2 hours 21 minutes, and would like to see if there's anything that I could do to speed up the file access, such as INSTALLing frequently accessed files and images, moving files to faster access media, etc. But in order to generate a solution to the the problem I need to know exactly which files are being accessed.
Hi

Try this

while the process executes commands?

ana/sys
sda>set proc/id=xxxxx
sad>sh proc/chann

---------------------------------------------------------

$create mio.dat
$set file/statistics mio.dat
$monitor rms/file=mio.dat/item=all
$ SET RMS_DEFAULT /BLOCK_COUNT=127 /BUFFER_COUNT=255 /INDEXED

$ana/sys
Sda>show rms
SDA>SET RMS=(IFAB=1,CCB,WCB)
SDA>SHOW RMS

$ANALYZE/SYSTEM ...
Sda>SET PROC/IN=xxx
Sda>SHOW PROC/RMS=(RAB,FAB,BDBSUM).

Regards

Loading...