Discussion:
Wide area cluster, metro area network, seeking info
(too old to reply)
Rich Jordan
2021-06-08 22:28:49 UTC
Permalink
We are looking at the possibility of putting VMS boxes in two locations, with Integrity boxes running VSI VMS. This is the very beginning of the research on the possibility of clustering those two servers instead of just having them networked. Probably have to be master/slave since only two nodes and no shared storage.

After reviewing the various cluster docs, they seem to be focused on older technologies like SoNET and DS3 using FDDI bridges (which would allow shared storage). The prospect has a metropolitan area network but I do not have any specs on that as yet.

Are there available docs relevant to running a distributed VMS cluster over a metro area network or fast/big enough VPN tunnel? Or is that just the straight cluster over IP configuration in the docs (which we've never used) that we need to concentrate on?

Thanks
Arne Vajhøj
2021-06-08 23:12:26 UTC
Permalink
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two
locations, with Integrity boxes running VSI VMS. This is the very
beginning of the research on the possibility of clustering those two
servers instead of just having them networked. Probably have to be
master/slave since only two nodes and no shared storage.
You mean node A 2 votes and node B 1 vote?
Post by Rich Jordan
After reviewing the various cluster docs, they seem to be focused on
older technologies like SoNET and DS3 using FDDI bridges (which would
allow shared storage). The prospect has a metropolitan area network
but I do not have any specs on that as yet.
Are there available docs relevant to running a distributed VMS
cluster over a metro area network or fast/big enough VPN tunnel? Or
is that just the straight cluster over IP configuration in the docs
(which we've never used) that we need to concentrate on?
First you need to find out what protocols they can support. Most
likely IP only. That means you need to look at IP.

Then you need to ask a lot of questions about latency. Average,
maximum, distribution etc.. Because high latency can kill the
project.

Arne
dthi...@gmail.com
2021-06-09 00:49:07 UTC
Permalink
Post by Arne Vajhøj
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two
locations, with Integrity boxes running VSI VMS. This is the very
beginning of the research on the possibility of clustering those two
servers instead of just having them networked. Probably have to be
master/slave since only two nodes and no shared storage.
You mean node A 2 votes and node B 1 vote?
Post by Rich Jordan
After reviewing the various cluster docs, they seem to be focused on
older technologies like SoNET and DS3 using FDDI bridges (which would
allow shared storage). The prospect has a metropolitan area network
but I do not have any specs on that as yet.
Are there available docs relevant to running a distributed VMS
cluster over a metro area network or fast/big enough VPN tunnel? Or
is that just the straight cluster over IP configuration in the docs
(which we've never used) that we need to concentrate on?
First you need to find out what protocols they can support. Most
likely IP only. That means you need to look at IP.
Then you need to ask a lot of questions about latency. Average,
maximum, distribution etc.. Because high latency can kill the
project.
Arne
This might a good time to call in VSI or <another OpenVMS consulting group> to consult about creating a successful and stable wide area cluster. :-)

With Integrity systems, you basically have only one cluster communications channel: Ethernet. So you have a choice of a LAN Cluster (LAVC) or a WAN Cluster (Cluster-over-IP).

With the LAN Cluster, you have to _bridge_ the sites together with fairly expensive network equipment or create a poor-man's VPN tunnel, which then means your cluster has a secondary dependency upon the stability of the tunneling/bridging systems. The MAN (Metropolitan Area Network) technologies can fit into the LAN cluster category, although they don't have to. With Cluster-over-IP, you can use existing internet routing between the two sites, although you do have to open the SCS communications port(s) in your firewalls. A MAN is still an option for Cluster-over-IP, although a rather expensive one compared to the cost of a standard ISP Point-Of-Presence.

As Arne pointed out, latency can be a big problem for a wide area cluster, regardless of whether you choose LAN or WAN. The cluster design book comments about factoring the latency between your sites when setting cluster parameters. HPE/VSI recommended keeping the distance between clustered sites to under 500 miles due to latency, although OpenVMS engineering at HPE India did forge a planet-wide OpenVMS Cluster-over-IP centered in Banglore, India. They noted that the cluster parameters had to be tweaked heavily to support the excessive distance latency.

Regarding cluster votes and quorum: You might find that purchasing a cheap second-hand Integrity located at a third site to provide a tie-breaking vote could enhance your cluster stability. :-)


David
Phillip Helbig (undress to reply)
2021-06-09 05:04:55 UTC
Permalink
We are looking at the possibility of putting VMS boxes in two locations, wi=
th Integrity boxes running VSI VMS. This is the very beginning of the rese=
arch on the possibility of clustering those two servers instead of just hav=
ing them networked. Probably have to be master/slave since only two nodes =
and no shared storage.
If you do a cluster, do it right. What advantage does master/slave give
you? Note that a third node (at a third location) can be a really small
box doing essentially nothing.

If there is no shared storage (hence you hint at there can be no quorum
disk), use HBVS with (at least) one member at each location. With a
third (quorum) node and HBVS, your cluster will continue to work if one
node is lost. With master/slave, it won't, so what's the point?
Stephen Hoffman
2021-06-09 18:31:30 UTC
Permalink
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two
locations, with Integrity boxes running VSI VMS. This is the very
beginning of the research on the possibility of clustering those two
servers instead of just having them networked. Probably have to be
master/slave since only two nodes and no shared storage.
After reviewing the various cluster docs, they seem to be focused on
older technologies like SoNET and DS3 using FDDI bridges (which would
allow shared storage). The prospect has a metropolitan area network
but I do not have any specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster
over a metro area network or fast/big enough VPN tunnel? Or is that
just the straight cluster over IP configuration in the docs (which
we've never used) that we need to concentrate on?
Once the costs are understood and acceptable, then the design
discussions and trade-offs can progress.

Get the prices, first. Before starting to investigate the design in any
detail, get a quote from VSI for clustering and likely also for
software RAID-1 (HBVS), and determine whether those costs are
sustainable, and whether SaaS licensing or traditional licensing is
preferred. Prior to the advent of SaaS licensing, the cluster license
purchase prices alone ended more than a few of these project
discussions. Also for this case, get a quote for metropolitan Ethernet
via SDN or MPLS or point-to-point microwave or whatever other similar
connect the local comms carriers are offering.

Once you're over that purchasing hurdle, your design here is inherently
going to be primary-secondary only (two hosts, with no shared storage),
and your interconnect speed will be limited by the available link
speeds and which particularly plays into software RAID-1 processing,
and you'll need to discuss whether that'll meet your customer's
clustering expectations. Primary-secondary also means manual
intervention to promote the secondary to primary status, whether the
primary is down or the link is down. And clustering usually means both
sites drop offline if somebody or something clobbers the shared
production data, whether that might have been an accidental or
incidental or malicious corruption.

I prefer LAN clustering and network bridging to the IPCI clustering /
NISCS_USE_UDP path as there are some other cluster-related options
available only with a bridged LAN, though both interconnects can and do
work.

Whichever you pick here, software RAID-1 needs beaucoup bandwidth or
you'll be enjoying cascading failures.

Also review the DT (delirium tremens, err, disaster tolerance)
alternatives. Splitting off volumes and a rotating schedule shipping
those copies to the warm site can be cost-acceptable for some
applications and environments.

The above assumes y'all are within maximum the clustering span; within
500 mi / 800 km, as the wires were drug; as the electrons fly.
--
Pure Personal Opinion | HoffmanLabs LLC
Marc Van Dyck
2021-06-10 15:22:53 UTC
Permalink
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two locations, with
Integrity boxes running VSI VMS. This is the very beginning of the research
on the possibility of clustering those two servers instead of just having
them networked. Probably have to be master/slave since only two nodes and no
shared storage.
After reviewing the various cluster docs, they seem to be focused on older
technologies like SoNET and DS3 using FDDI bridges (which would allow shared
storage). The prospect has a metropolitan area network but I do not have any
specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a
metro area network or fast/big enough VPN tunnel? Or is that just the
straight cluster over IP configuration in the docs (which we've never used)
that we need to concentrate on?
Thanks
Before going into the technical details, wouldn't it be interersting to
know what you want to achieve, and discuss whether clustering is the
best way to achieve it ? I, for one, would be interested to know why
you
believe that clustering without shared storage can be more beneficial
than simple networking.
--
Marc Van Dyck
Rich Jordan
2021-06-11 18:20:12 UTC
Permalink
Post by Marc Van Dyck
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two locations, with
Integrity boxes running VSI VMS. This is the very beginning of the research
on the possibility of clustering those two servers instead of just having
them networked. Probably have to be master/slave since only two nodes and no
shared storage.
After reviewing the various cluster docs, they seem to be focused on older
technologies like SoNET and DS3 using FDDI bridges (which would allow shared
storage). The prospect has a metropolitan area network but I do not have any
specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a
metro area network or fast/big enough VPN tunnel? Or is that just the
straight cluster over IP configuration in the docs (which we've never used)
that we need to concentrate on?
Thanks
Before going into the technical details, wouldn't it be interersting to
know what you want to achieve, and discuss whether clustering is the
best way to achieve it ? I, for one, would be interested to know why
you
believe that clustering without shared storage can be more beneficial
than simple networking.
--
Marc Van Dyck
VSI is involved. This is, as I mentioned, the very first part of the investigation into the possibility of clustering, and I have no experience with MAN's or wide area clusters to go on so have been reading up on the available docs. We _are_ going through the proper steps, just was hoping to pick up some info here from folks who might have already set up something similar. We have set up and maintained LAVC, FC, and SCSI clusters and long ago helped manage a CI cluster but a wide area cluster never came up.

The point: the system at the second location will be a backup site and/or disaster recovery box. There is no third location and I have no info on the likelihood of getting one.

This is all just investigation right now.

Thanks
Phillip Helbig (undress to reply)
2021-06-11 20:35:24 UTC
Permalink
Post by Rich Jordan
The point: the system at the second location will be a backup site and/or
disaster recovery box. There is no third location and I have no info on
the likelihood of getting one.
A two-node cluster either has each node with equal votes, and both must
be present, or one has more than the other, and the cluster needs it to
survive. In either case, if you lose the main site, some manual
intervention will be necessary to allow the cluster to continue. And
that might not be worth much without access to data. It is possible for
the two nodes to share storage, but unless that is at a third location,
with your setup above things won't work at all if you lose one of the
sites. HBVS, at least if we are talking about several kilometers
between the nodes, say, is probably the way to go. (The individual
members can be anything VMS sees as a disk, i.e. could come from a SAN
or whatever.)

Unless the office is at one of the locations, just put a third
node---the smallest you can find---in the office. (You could also put a
quorum disk there, and could thus keep going after losing one of the big
sites, but a node probably offers more for little or no additional
cost.)
Stephen Hoffman
2021-06-11 21:13:34 UTC
Permalink
A two-node cluster either has each node with equal votes...
Unless the office is at one of the locations, just put a third
node---the smallest you can find---in the office...
There's little reason to run a one and one vote two-host configuration,
nor particular reason to license a third cluster node co-resident with
either lobe unless there's also shared storage involved within that
lobe.
The former as a failure in either lobe stalls all, the latter as it's
little different from a one and none voting configuration and it costs
way more.
Same for using a quorum disk in any cluster without also having a
shared interconnect; assign those no-shared-interconnect quorum disk
votes to the host.
--
Pure Personal Opinion | HoffmanLabs LLC
Phillip Helbig (undress to reply)
2021-06-12 06:58:34 UTC
Permalink
Post by Stephen Hoffman
A two-node cluster either has each node with equal votes...
Unless the office is at one of the locations, just put a third
node---the smallest you can find---in the office...
There's little reason to run a one and one vote two-host configuration,
Right.
Post by Stephen Hoffman
nor particular reason to license a third cluster node co-resident with
either lobe unless there's also shared storage involved within that
lobe.
Note that I said in the office, not co-resident with any of the main
nodes. Yes, a license will cost something. On the other hand, one has
a node which is a full cluster member yet not responsible for the main
production stuff, so one can test things like reboots, backups, and so
on, which one can't do with a quorum disk.
Post by Stephen Hoffman
The former as a failure in either lobe stalls all, the latter as it's
little different from a one and none voting configuration and it costs
way more.
The minimum is three nodes (or two nodes and a quorum disk, but see
above), but in three separate locations.
Post by Stephen Hoffman
Same for using a quorum disk in any cluster without also having a
shared interconnect; assign those no-shared-interconnect quorum disk
votes to the host.
A quorum disk makes little sense without a shared interconnect.
Dave Froble
2021-06-11 23:04:22 UTC
Permalink
Post by Phillip Helbig (undress to reply)
Post by Rich Jordan
The point: the system at the second location will be a backup site and/or
disaster recovery box. There is no third location and I have no info on
the likelihood of getting one.
A two-node cluster either has each node with equal votes, and both must
be present, or one has more than the other, and the cluster needs it to
survive. In either case, if you lose the main site, some manual
intervention will be necessary to allow the cluster to continue. And
that might not be worth much without access to data. It is possible for
the two nodes to share storage, but unless that is at a third location,
with your setup above things won't work at all if you lose one of the
sites. HBVS, at least if we are talking about several kilometers
between the nodes, say, is probably the way to go. (The individual
members can be anything VMS sees as a disk, i.e. could come from a SAN
or whatever.)
Unless the office is at one of the locations, just put a third
node---the smallest you can find---in the office. (You could also put a
quorum disk there, and could thus keep going after losing one of the big
sites, but a node probably offers more for little or no additional
cost.)
What does a cluster license for that third node cost these days?

My worthless opinion is, if you are not going to have shared disks,
HBVS, and such, then why run a cluster?
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Phillip Helbig (undress to reply)
2021-06-12 07:03:17 UTC
Permalink
Post by Dave Froble
What does a cluster license for that third node cost these days?
I don't know, but at most half the cost for the two other licenses.
Post by Dave Froble
My worthless opinion is, if you are not going to have shared disks,
HBVS, and such, then why run a cluster?
Indeed.
Marc Van Dyck
2021-06-12 11:29:35 UTC
Permalink
Post by Phillip Helbig (undress to reply)
Post by Dave Froble
What does a cluster license for that third node cost these days?
I don't know, but at most half the cost for the two other licenses.
Post by Dave Froble
My worthless opinion is, if you are not going to have shared disks,
HBVS, and such, then why run a cluster?
Indeed.
I'm not even sure that you can purchase a clustering license
separately.
You either purchase a VMS basic license, or an enterprise license that
comes with clustering, shadowing, RMS journaling, and whatnot.
--
Marc Van Dyck
Arne Vajhøj
2021-06-15 13:24:26 UTC
Permalink
Post by Rich Jordan
Post by Marc Van Dyck
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two locations, with
Integrity boxes running VSI VMS. This is the very beginning of the research
on the possibility of clustering those two servers instead of just having
them networked. Probably have to be master/slave since only two nodes and no
shared storage.
After reviewing the various cluster docs, they seem to be focused on older
technologies like SoNET and DS3 using FDDI bridges (which would allow shared
storage). The prospect has a metropolitan area network but I do not have any
specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a
metro area network or fast/big enough VPN tunnel? Or is that just the
straight cluster over IP configuration in the docs (which we've never used)
that we need to concentrate on?
Before going into the technical details, wouldn't it be interersting to
know what you want to achieve, and discuss whether clustering is the
best way to achieve it ? I, for one, would be interested to know why
you
believe that clustering without shared storage can be more beneficial
than simple networking.
The point: the system at the second location will be a backup site
and/or disaster recovery box. There is no third location and I have
no info on the likelihood of getting one.
If the business problem is to ensure that the second location always
have a copy of all data, then a VMS cluster may not be the optimal
solution.

There are other VMS features/products than clustering that could
be relevant.

And depending on what the system is used for then there may also
be application specific solutions.

Arne
Michael Moroney
2021-06-15 17:55:08 UTC
Permalink
Post by Arne Vajhøj
Post by Rich Jordan
Post by Marc Van Dyck
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two locations, with
Integrity boxes running VSI VMS. This is the very beginning of the research
on the possibility of clustering those two servers instead of just having
them networked. Probably have to be master/slave since only two nodes and no
shared storage.
After reviewing the various cluster docs, they seem to be focused on older
technologies like SoNET and DS3 using FDDI bridges (which would allow shared
storage). The prospect has a metropolitan area network but I do not have any
specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a
metro area network or fast/big enough VPN tunnel? Or is that just the
straight cluster over IP configuration in the docs (which we've never used)
that we need to concentrate on?
Before going into the technical details, wouldn't it be interersting to
know what you want to achieve, and discuss whether clustering is the
best way to achieve it ? I, for one, would be interested to know why
you
believe that clustering without shared storage can be more beneficial
than simple networking.
The point: the system at the second location will be a backup site
and/or disaster recovery box.  There is no third location and I have
no info on the likelihood of getting one.
If the business problem is to ensure that the second location always
have a copy of all data, then a VMS cluster may not be the optimal
solution.
There are other VMS features/products than clustering that could
be relevant.
It appears to me one possibility is fiberchannel presenting RAID-1
units, with one member of each RAID-1 at the main site and one at the
backup site connected with dark fiber (FOIP?). VMS system (no cluster)
at main site and either no system or a cold backup at the backup site.
Rich Jordan
2021-06-15 22:19:38 UTC
Permalink
Post by Michael Moroney
Post by Arne Vajhøj
Post by Rich Jordan
Post by Marc Van Dyck
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two locations, with
Integrity boxes running VSI VMS. This is the very beginning of the research
on the possibility of clustering those two servers instead of just having
them networked. Probably have to be master/slave since only two nodes and no
shared storage.
After reviewing the various cluster docs, they seem to be focused on older
technologies like SoNET and DS3 using FDDI bridges (which would allow shared
storage). The prospect has a metropolitan area network but I do not have any
specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a
metro area network or fast/big enough VPN tunnel? Or is that just the
straight cluster over IP configuration in the docs (which we've never used)
that we need to concentrate on?
Before going into the technical details, wouldn't it be interersting to
know what you want to achieve, and discuss whether clustering is the
best way to achieve it ? I, for one, would be interested to know why
you
believe that clustering without shared storage can be more beneficial
than simple networking.
The point: the system at the second location will be a backup site
and/or disaster recovery box. There is no third location and I have
no info on the likelihood of getting one.
If the business problem is to ensure that the second location always
have a copy of all data, then a VMS cluster may not be the optimal
solution.
There are other VMS features/products than clustering that could
be relevant.
It appears to me one possibility is fiberchannel presenting RAID-1
units, with one member of each RAID-1 at the main site and one at the
backup site connected with dark fiber (FOIP?). VMS system (no cluster)
at main site and either no system or a cold backup at the backup site.
I love the discussions that questions like this bring up.

More info. The base requirement is to have a backup system at an existing remote office; this backup system will be kept 'up to date' and available for use if the primary site or system fails.

One option is nightly backup/restore operations, so (up to) one day latency. This is considered the low end of acceptable. The customer asked about options for keeping the backup closer to current.

The system cannot be quiesced during production hours (about 11 hours per day weekdays) so we cannot run periodic backups during those hours. Programs just were not written that way.

Since we can't do intraday backups, one option is shadowing, and so clustering. I didn't mention HBVS (my bad) but its the reason for the cluster option. I was mainly looking for info about running a cluster over a WAN connection/metro area network in case anyone had that experience, since much of the documentation available is pretty old and seems to concentrate on using FDDI bridges over what are currently modest speed links.

VSI _is_ involved and we are working with them on this possibility. At this point I think the additional license subscription costs are going to kill the HBVS/cluster option, especially if a third node was needed (and a third location and connection, and set of licenses). That means going with the one-day latency backup option and generating and testing the procedures for failing back to the main system when it is available again.

We have not determined availability of dark fiber; the intention is to use the metro area network, and so cluster over IP. We did some price checks on the equipment needed to bridge the sites and I am told it is not within the budget.

Thanks

Rich
abrsvc
2021-06-15 22:28:45 UTC
Permalink
Post by Rich Jordan
Post by Michael Moroney
Post by Arne Vajhøj
Post by Rich Jordan
Post by Marc Van Dyck
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two locations, with
Integrity boxes running VSI VMS. This is the very beginning of the research
on the possibility of clustering those two servers instead of just having
them networked. Probably have to be master/slave since only two nodes and no
shared storage.
After reviewing the various cluster docs, they seem to be focused on older
technologies like SoNET and DS3 using FDDI bridges (which would allow shared
storage). The prospect has a metropolitan area network but I do not have any
specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a
metro area network or fast/big enough VPN tunnel? Or is that just the
straight cluster over IP configuration in the docs (which we've never used)
that we need to concentrate on?
Before going into the technical details, wouldn't it be interersting to
know what you want to achieve, and discuss whether clustering is the
best way to achieve it ? I, for one, would be interested to know why
you
believe that clustering without shared storage can be more beneficial
than simple networking.
The point: the system at the second location will be a backup site
and/or disaster recovery box. There is no third location and I have
no info on the likelihood of getting one.
If the business problem is to ensure that the second location always
have a copy of all data, then a VMS cluster may not be the optimal
solution.
There are other VMS features/products than clustering that could
be relevant.
It appears to me one possibility is fiberchannel presenting RAID-1
units, with one member of each RAID-1 at the main site and one at the
backup site connected with dark fiber (FOIP?). VMS system (no cluster)
at main site and either no system or a cold backup at the backup site.
I love the discussions that questions like this bring up.
More info. The base requirement is to have a backup system at an existing remote office; this backup system will be kept 'up to date' and available for use if the primary site or system fails.
One option is nightly backup/restore operations, so (up to) one day latency. This is considered the low end of acceptable. The customer asked about options for keeping the backup closer to current.
The system cannot be quiesced during production hours (about 11 hours per day weekdays) so we cannot run periodic backups during those hours. Programs just were not written that way.
Since we can't do intraday backups, one option is shadowing, and so clustering. I didn't mention HBVS (my bad) but its the reason for the cluster option. I was mainly looking for info about running a cluster over a WAN connection/metro area network in case anyone had that experience, since much of the documentation available is pretty old and seems to concentrate on using FDDI bridges over what are currently modest speed links.
VSI _is_ involved and we are working with them on this possibility. At this point I think the additional license subscription costs are going to kill the HBVS/cluster option, especially if a third node was needed (and a third location and connection, and set of licenses). That means going with the one-day latency backup option and generating and testing the procedures for failing back to the main system when it is available again.
We have not determined availability of dark fiber; the intention is to use the metro area network, and so cluster over IP. We did some price checks on the equipment needed to bridge the sites and I am told it is not within the budget.
Thanks
Rich
Rich,

I sent you an Email describing what might work for you. There are client specifics that I'd rather not post publicly however. Contact me directly as I had a client with a similar requirement and a solution that was acceptable to them.

Dan
dthi...@gmail.com
2021-06-15 22:54:41 UTC
Permalink
Post by Rich Jordan
VSI _is_ involved and we are working with them on this possibility. At this point I think the additional license subscription costs are going to kill the HBVS/cluster option, especially if a third node was needed (and a third location and connection, and set of licenses). That means going with the one-day latency backup option and generating and testing the procedures for failing back to the main system when it is available again.
This is just a thinking-outside-the-box thought to bounce off of VSI in the design discussions. Since OpenVMS doesn't really have a good solution for stabilizing a wide area 2-node cluster without extensive hardware/software investment, you might ask them if they would authorize you to set up a FreeAXP OpenVMS VM at a third site using the Alpha Community License solely to provide a tie-breaking vote for your production two-node cluster, and to do no other work. It never hurts to ask if there's an acceptable lower cost alternative to an expensive situation. This tie-breaker VM could even be hosted at an offsite VM provider like AWS so that your company doesn't have to invest in a physical 3rd site and network presence.
Tad Winters
2021-06-16 04:10:53 UTC
Permalink
Post by ***@gmail.com
Post by Rich Jordan
VSI _is_ involved and we are working with them on this possibility. At this point I think the additional license subscription costs are going to kill the HBVS/cluster option, especially if a third node was needed (and a third location and connection, and set of licenses). That means going with the one-day latency backup option and generating and testing the procedures for failing back to the main system when it is available again.
This is just a thinking-outside-the-box thought to bounce off of VSI in the design discussions. Since OpenVMS doesn't really have a good solution for stabilizing a wide area 2-node cluster without extensive hardware/software investment, you might ask them if they would authorize you to set up a FreeAXP OpenVMS VM at a third site using the Alpha Community License solely to provide a tie-breaking vote for your production two-node cluster, and to do no other work. It never hurts to ask if there's an acceptable lower cost alternative to an expensive situation. This tie-breaker VM could even be hosted at an offsite VM provider like AWS so that your company doesn't have to invest in a physical 3rd site and network presence.
_______________________________________________
Info-vax mailing list
http://rbnsn.com/mailman/listinfo/info-vax_rbnsn.com
Hmm. Let's call the tie-breaking site, T. Site A is the site they want
to consider the main site and site B is the backup site.
Suppose a fire (or other spreading damage) occurs. Fiber is destroyed
such that site A no longer has connectivity. An hour or two passes,
where site B has continued to communicate with site T. Then the fire
(or whatever) terminates the connection. Now priorities as they are,
the company servicing the fiber, goes out and repairs/replaces the fiber
so that site A resumes communication with site T. It's going to be a
couple more days before they repair the connection so that site B can
communicate with site T.
The question: Is there enough information at site T for site A to
really resume operation, or did site B make sufficient changes that site
A cannot operate until site B is back in communication?

I think this solution is safer if you can be sure you have completely
redundant fiber links between site A and site B, where the fiber paths
enter and exit each building by two different paths and that they never
cross along the way. This reduces the chance of a communication issue
causing problems with the cluster.
Stephen Hoffman
2021-06-16 14:59:36 UTC
Permalink
Post by ***@gmail.com
Post by Rich Jordan
VSI _is_ involved and we are working with them on this possibility. At
this point I think the additional license subscription costs are going
to kill the HBVS/cluster option, especially if a third node was needed
(and a third location and connection, and set of licenses). That means
going with the one-day latency backup option and generating and testing
the procedures for failing back to the main system when it is available
again.
This is just a thinking-outside-the-box thought to bounce off of VSI in
the design discussions. Since OpenVMS doesn't really have a good
solution for stabilizing a wide area 2-node cluster without extensive
hardware/software investment, you might ask them if they would
authorize you to set up a FreeAXP OpenVMS VM at a third site using the
Alpha Community License solely to provide a tie-breaking vote for your
production two-node cluster, and to do no other work. It never hurts to
ask if there's an acceptable lower cost alternative to an expensive
situation. This tie-breaker VM could even be hosted at an offsite VM
provider like AWS so that your company doesn't have to invest in a
physical 3rd site and network presence.
Or more succinctly, ask VSI for "buy two get one free" cluster pricing.

Two cluster PAKs still won't be cheap, if VSI goes for that.

(VSI does not presently have a voting-only cluster member product
offering at present. The old cluster-client license can't vote.)
--
Pure Personal Opinion | HoffmanLabs LLC
Michael Moroney
2021-06-16 15:38:43 UTC
Permalink
Post by Stephen Hoffman
Post by ***@gmail.com
Post by Rich Jordan
VSI _is_ involved and we are working with them on this possibility.
At this point I think the additional license subscription costs are
going to kill the HBVS/cluster option, especially if a third node
was needed (and a third location and connection, and set of
licenses). That means going with the one-day latency backup option
and generating and testing the procedures for failing back to the
main system when it is available again.
This is just a thinking-outside-the-box thought to bounce off of VSI
in the design discussions.  Since OpenVMS doesn't really have a good
solution for stabilizing a wide area 2-node cluster without extensive
hardware/software investment, you might ask them if they would
authorize you to set up a FreeAXP OpenVMS VM at a third site using the
Alpha Community License solely to provide a tie-breaking vote for your
production two-node cluster, and to do no other work. It never hurts
to ask if there's an acceptable lower cost alternative to an expensive
situation. This tie-breaker VM could even be hosted at an offsite VM
provider like AWS so that your company doesn't have to invest in a
physical 3rd site and network presence.
Or more succinctly, ask VSI for "buy two get one free" cluster pricing.
Two cluster PAKs still won't be cheap, if VSI goes for that.
(VSI does not presently have a voting-only cluster member product
offering at present. The old cluster-client license can't vote.)
I was going to propose something similar. A very restricted cluster
license, (relatively) cheap, with the ability to provide one vote to a
cluster and nothing else. Perhaps even disable ability to do other
things. "Buy two, get this free" would be excellent other than the "buy
two" price, of course.

Once upon a time I even toyed with the idea of proposing creating a
"widget", a program which could speak just enough SCS to provide one
vote to a cluster, to be the third "node" in one, and absolutely nothing
more. The program (not necessarily a dedicated system) could be on a PC
or something and not a VMS implementation. Its only purpose would be to
join a cluster and provide 1 vote. Before thinking how absurd that idea
is, do remember that things like HSCs and DSSI disks spoke enough SCS to
provide MSCP disk access to a cluster over CI/DSSI buses and they showed
up on $ SHOW CLUSTER. Just not as full cluster members, and DSSI disks
did that with the firmware on a disk drive.

I didn't/don't want to get involved with the hornets' nest of licensing
such a thing.
Marc Van Dyck
2021-06-16 16:16:52 UTC
Permalink
Post by Rich Jordan
Post by Michael Moroney
Post by Arne Vajhøj
Post by Rich Jordan
Post by Marc Van Dyck
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two locations, with
Integrity boxes running VSI VMS. This is the very beginning of the research
on the possibility of clustering those two servers instead of just having
them networked. Probably have to be master/slave since only two nodes and no
shared storage.
After reviewing the various cluster docs, they seem to be focused on older
technologies like SoNET and DS3 using FDDI bridges (which would allow shared
storage). The prospect has a metropolitan area network but I do not have any
specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a
metro area network or fast/big enough VPN tunnel? Or is that just the
straight cluster over IP configuration in the docs (which we've never used)
that we need to concentrate on?
Before going into the technical details, wouldn't it be interersting to
know what you want to achieve, and discuss whether clustering is the
best way to achieve it ? I, for one, would be interested to know why
you
believe that clustering without shared storage can be more beneficial
than simple networking.
The point: the system at the second location will be a backup site
and/or disaster recovery box. There is no third location and I have
no info on the likelihood of getting one.
If the business problem is to ensure that the second location always
have a copy of all data, then a VMS cluster may not be the optimal
solution.
There are other VMS features/products than clustering that could
be relevant.
It appears to me one possibility is fiberchannel presenting RAID-1
units, with one member of each RAID-1 at the main site and one at the
backup site connected with dark fiber (FOIP?). VMS system (no cluster)
at main site and either no system or a cold backup at the backup site.
I love the discussions that questions like this bring up.
More info. The base requirement is to have a backup system at an existing
remote office; this backup system will be kept 'up to date' and available for
use if the primary site or system fails.
One option is nightly backup/restore operations, so (up to) one day latency.
This is considered the low end of acceptable. The customer asked about
options for keeping the backup closer to current.
The system cannot be quiesced during production hours (about 11 hours per day
weekdays) so we cannot run periodic backups during those hours. Programs
just were not written that way.
Since we can't do intraday backups, one option is shadowing, and so
clustering. I didn't mention HBVS (my bad) but its the reason for the
cluster option. I was mainly looking for info about running a cluster over a
WAN connection/metro area network in case anyone had that experience, since
much of the documentation available is pretty old and seems to concentrate on
using FDDI bridges over what are currently modest speed links.
VSI _is_ involved and we are working with them on this possibility. At this
point I think the additional license subscription costs are going to kill the
HBVS/cluster option, especially if a third node was needed (and a third
location and connection, and set of licenses). That means going with the
one-day latency backup option and generating and testing the procedures for
failing back to the main system when it is available again.
We have not determined availability of dark fiber; the intention is to use
the metro area network, and so cluster over IP. We did some price checks on
the equipment needed to bridge the sites and I am told it is not within the
budget.
Thanks
Rich
You don't really need clusters, HBVS, and dark fiber for that. You can
have controller-based storage replication between the two sites,
running
on fibre channel over IP. And just a cold system on the backup site
that
you can boot if the primary site fails. What I don't know is the price
aspect, i.e. whether there are reasonably cheap storage solutions for
controller-based replication and FC/IP.
--
Marc Van Dyck
Phillip Helbig (undress to reply)
2021-06-16 21:30:52 UTC
Permalink
Post by Rich Jordan
More info. The base requirement is to have a backup system at an existing
remote office; this backup system will be kept 'up to date' and available
for use if the primary site or system fails.
OK. If you have a cluster which is set up properly, then all of that
stuff just works.
Post by Rich Jordan
One option is nightly backup/restore operations, so (up to) one day latency
.. This is considered the low end of acceptable. The customer asked about
options for keeping the backup closer to current.
With HBVS, the latency is essentially zero. A WRITE won't complete
until all members have been written to. Split the shadow set across the
locations.
Post by Rich Jordan
The system cannot be quiesced during production hours (about 11 hours per
day weekdays) so we cannot run periodic backups during those hours. Programs
just were not written that way.
Another reason to go with a cluster and HBVS.
Post by Rich Jordan
Since we can't do intraday backups, one option is shadowing, and so
clustering. I didn't mention HBVS (my bad) but its the reason for the
cluster option.
OK.
Post by Rich Jordan
I was mainly looking for info about running a cluster over a WAN
connection/metro area network in case anyone had that experience, since
much of the documentation available is pretty old and seems to
concentrate on using FDDI bridges over what are currently modest speed
links.
These days, one can run a cluster over IP.
Post by Rich Jordan
VSI _is_ involved and we are working with them on this possibility. At
this point I think the additional license subscription costs are going
to kill the HBVS/cluster option, especially if a third node was needed
(and a third location and connection, and set of licenses). That means
going with the one-day latency backup option and generating and testing
the procedures for failing back to the main system when it is available
again.
I don't know what the license costs are. But with a cluster, someone
familiar with VMS could keep an eye on it in his spare time. With the
other option, someone will have to spend much more time on it. What are
salary costs?
Rich Jordan
2021-06-17 23:15:31 UTC
Permalink
Post by Rich Jordan
More info. The base requirement is to have a backup system at an existing
remote office; this backup system will be kept 'up to date' and available
for use if the primary site or system fails.
OK. If you have a cluster which is set up properly, then all of that
stuff just works.
Post by Rich Jordan
One option is nightly backup/restore operations, so (up to) one day latency
.. This is considered the low end of acceptable. The customer asked about
options for keeping the backup closer to current.
With HBVS, the latency is essentially zero. A WRITE won't complete
until all members have been written to. Split the shadow set across the
locations.
Post by Rich Jordan
The system cannot be quiesced during production hours (about 11 hours per
day weekdays) so we cannot run periodic backups during those hours. Programs
just were not written that way.
Another reason to go with a cluster and HBVS.
Post by Rich Jordan
Since we can't do intraday backups, one option is shadowing, and so
clustering. I didn't mention HBVS (my bad) but its the reason for the
cluster option.
OK.
Post by Rich Jordan
I was mainly looking for info about running a cluster over a WAN
connection/metro area network in case anyone had that experience, since
much of the documentation available is pretty old and seems to
concentrate on using FDDI bridges over what are currently modest speed
links.
These days, one can run a cluster over IP.
Post by Rich Jordan
VSI _is_ involved and we are working with them on this possibility. At
this point I think the additional license subscription costs are going
to kill the HBVS/cluster option, especially if a third node was needed
(and a third location and connection, and set of licenses). That means
going with the one-day latency backup option and generating and testing
the procedures for failing back to the main system when it is available
again.
I don't know what the license costs are. But with a cluster, someone
familiar with VMS could keep an eye on it in his spare time. With the
other option, someone will have to spend much more time on it. What are
salary costs?
Keeping shadowset on the main system and splitting off a member for backups periodically is possible but again given the nature of these programs and how they were written, NOT guaranteed to be 'transaction complete'. Sure individual writes will be done but a program updating records across multiple files can still see a shadow member dismounted when it has finished one file but not the other. We don't have transaction safe processing; this is old but very stable code running on RMS files.

I will ask about distributing the storage; that's outside my area of expertise. We did almost all small-med VMS business, and like wide area distributed clusters, that kind of configuration just never came up.
Mark Berryman
2021-06-12 01:10:09 UTC
Permalink
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two locations, with Integrity boxes running VSI VMS. This is the very beginning of the research on the possibility of clustering those two servers instead of just having them networked. Probably have to be master/slave since only two nodes and no shared storage.
After reviewing the various cluster docs, they seem to be focused on older technologies like SoNET and DS3 using FDDI bridges (which would allow shared storage). The prospect has a metropolitan area network but I do not have any specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a metro area network or fast/big enough VPN tunnel? Or is that just the straight cluster over IP configuration in the docs (which we've never used) that we need to concentrate on?
Thanks
First, I recommend you ignore the suggestions to add a 3rd node to your
cluster. In your situation, it is not really a viable answer.

There are configurations that will allow a member of a 2-node cluster to
automatically continue in the event that the other node fails. However,
if you lose the communication channel but both nodes stay up, the
cluster will partition and then you have to be really careful about how
you reform the cluster. Because of this, I tend not to recommend this
particular solution except in very specific circumstances.
(Circumstances where you can guarantee the correct node becomes the
shadow master when the cluster reforms and you haven't been writing
different data to each node).

As far as I can tell from your description, the only way clustering
would be a viable answer for you would be if you also did HBVS. In that
case, simply build a 2-node cluster with enough identical disks such
that all of the data you want present at the backup site can be placed
on a host-based volume set. HBVS will then keep the data at both sites
in sync.

Failure modes in this scenario.

1. Loss of the communication channel. In this case, both nodes will
hang (for the duration of the cluster timeout parameters). More
specifically, each will freeze any process that attempts a write to
disk. As long as the communication channel comes back up before the
cluster times out, everything will resume automatically. If it doesn't,
both nodes should take a CLUEXIT bugcheck. Once the communication
channel is back up, you then bring each node back up as appropriate.

2. Loss of one node. In this case the other node will hang. Manual
intervention is required to get it going again (specifically, a couple
of commands at the console to reset quorum). At that point, everything
simply resumes on that node.

The main reason for doing it this way is that it becomes a human
decision to decide what to do in the event of any failure. In the event
of any node or communication failure, any surviving cluster members will
simply stop until you tell them what to do. The main intent here is to
simply prevent the wrong node from becoming the shadow master when (or
if) the cluster is reformed.

Since you are in contact with VSI, I have no doubt they will cover this
type of scenario with you. This is presented merely as an idea to
generate questions as part of your discussions.

Mark Berryman
Tad Winters
2021-06-12 04:48:54 UTC
Permalink
Post by Mark Berryman
Post by Rich Jordan
We are looking at the possibility of putting VMS boxes in two
locations, with Integrity boxes running VSI VMS.  This is the very
beginning of the research on the possibility of clustering those two
servers instead of just having them networked.  Probably have to be
master/slave since only two nodes and no shared storage.
After reviewing the various cluster docs, they seem to be focused on
older technologies like SoNET and DS3 using FDDI bridges (which would
allow shared storage).  The prospect has a metropolitan area network
but I do not have any specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster
over a metro area network or fast/big enough VPN tunnel?  Or is that
just the straight cluster over IP configuration in the docs (which
we've never used) that we need to concentrate on?
Thanks
First, I recommend you ignore the suggestions to add a 3rd node to your
cluster.  In your situation, it is not really a viable answer.
There are configurations that will allow a member of a 2-node cluster to
automatically continue in the event that the other node fails.  However,
if you lose the communication channel but both nodes stay up, the
cluster will partition and then you have to be really careful about how
you reform the cluster.  Because of this, I tend not to recommend this
particular solution except in very specific circumstances.
(Circumstances where you can guarantee the correct node becomes the
shadow master when the cluster reforms and you haven't been writing
different data to each node).
As far as I can tell from your description, the only way clustering
would be a viable answer for you would be if you also did HBVS.  In that
case, simply build a 2-node cluster with enough identical disks such
that all of the data you want present at the backup site can be placed
on a host-based volume set.  HBVS will then keep the data at both sites
in sync.
Failure modes in this scenario.
1. Loss of the communication channel.  In this case, both nodes will
hang (for the duration of the cluster timeout parameters).  More
specifically, each will freeze any process that attempts a write to
disk.  As long as the communication channel comes back up before the
cluster times out, everything will resume automatically.  If it doesn't,
both nodes should take a CLUEXIT bugcheck.  Once the communication
channel is back up, you then bring each node back up as appropriate.
2. Loss of one node.  In this case the other node will hang.  Manual
intervention is required to get it going again (specifically, a couple
of commands at the console to reset quorum).  At that point, everything
simply resumes on that node.
The main reason for doing it this way is that it becomes a human
decision to decide what to do in the event of any failure.  In the event
of any node or communication failure, any surviving cluster members will
simply stop until you tell them what to do.  The main intent here is to
simply prevent the wrong node from becoming the shadow master when (or
if) the cluster is reformed.
Since you are in contact with VSI, I have no doubt they will cover this
type of scenario with you.  This is presented merely as an idea to
generate questions as part of your discussions.
Mark Berryman
This is all very interesting. If this was being done with virtual
machines, it would seem that the cheap solution would be to run the
"secondary" system as minimal, to keep down license costs, since the
only work it would be doing is handling HBVS. In the event of a
disaster at the primary site, like an earthquake leveling the building,
just configure new hardware at the secondary site and spin up a new
primary with the volumes from the secondary. Why pay for compute
resources and the extra power and cooling at the secondary site, which
you will likely never use?
Of course, this is hardware and had likely already been purchased, so
there's no real savings to be had.
Phillip Helbig (undress to reply)
2021-06-12 07:01:44 UTC
Permalink
Post by Mark Berryman
First, I recommend you ignore the suggestions to add a 3rd node to your
cluster. In your situation, it is not really a viable answer.
It would solve most of the problems you mention below, and also could
serve as a test node.
Post by Mark Berryman
There are configurations that will allow a member of a 2-node cluster to
automatically continue in the event that the other node fails.
How? If one has more votes, it is essential. If the votes are the
same, both are essential. Unless you mean a quorum disk. But it should
be at a third location.
Stephen Hoffman
2021-06-12 14:55:50 UTC
Permalink
and also could serve as a test node.
I'd recommend against having a test or development host integrated
within a production host or within a production cluster.
--
Pure Personal Opinion | HoffmanLabs LLC
Phillip Helbig (undress to reply)
2021-06-12 15:09:16 UTC
Permalink
Post by Stephen Hoffman
and also could serve as a test node.
I'd recommend against having a test or development host integrated
within a production host or within a production cluster.
Sure. But if costs are an issue, then one would need licenses for
development and test machines, so might want to combine them. Certainly
for testing the reboot of a production system it would make sense for it
to be in the same cluster.

As someone here pointed out, everyone has a test environment; the lucky
ones have a dedicated production environment. :-)
Bill Gunshannon
2021-06-12 15:11:46 UTC
Permalink
Post by Stephen Hoffman
and also could serve as a test node.
I'd recommend against having a test or development host integrated
within a production host or within a production cluster.
Really? Hmmmm... What could possibly go wrong? :-)

bill
Dave Froble
2021-06-12 16:29:46 UTC
Permalink
Post by Stephen Hoffman
and also could serve as a test node.
I'd recommend against having a test or development host integrated
within a production host or within a production cluster.
+1

A production system is for just that, production, and anything else
happening is a risk. (Cue Simon.)

Systems are now so inexpensive that separate systems for development is
the only reasonable choice.
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Phillip Helbig (undress to reply)
2021-06-12 17:23:41 UTC
Permalink
Post by Dave Froble
Systems are now so inexpensive that separate systems for development is
the only reasonable choice.
One of the objections to a third node here is the cost of the license.
TCO.
Dave Froble
2021-06-12 23:02:35 UTC
Permalink
Post by Phillip Helbig (undress to reply)
Post by Dave Froble
Systems are now so inexpensive that separate systems for development is
the only reasonable choice.
One of the objections to a third node here is the cost of the license.
TCO.
Not sure what you're trying to say ???

As for development, VSI has a development program.

With x86 VMS, and VMs, things will be even easier.
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Arne Vajhøj
2021-06-12 18:29:18 UTC
Permalink
Post by Dave Froble
Post by Stephen Hoffman
and also could serve as a test node.
I'd recommend against having a test or development host integrated
within a production host or within a production cluster.
+1
A production system is for just that, production, and anything else
happening is a risk.  (Cue Simon.)
A production system and one or more test systems and if development
need VMS then also a development system.

For native languages then a VMS development system will be needed - even
with VMS IDE. But many of the newer languages (Java, Python, PHP, Perl
etc.) then development may in many cases be done entirely on the
users PC running Windows/macOS/Linux and first go on VMS for test.
Post by Dave Froble
Systems are now so inexpensive that separate systems for development is
the only reasonable choice.
Amazing where some companies sometime want to "save" money.

Arne
Mark Berryman
2021-06-14 02:54:55 UTC
Permalink
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
First, I recommend you ignore the suggestions to add a 3rd node to your
cluster. In your situation, it is not really a viable answer.
It would solve most of the problems you mention below, and also could
serve as a test node.
Post by Mark Berryman
There are configurations that will allow a member of a 2-node cluster to
automatically continue in the event that the other node fails.
How? If one has more votes, it is essential. If the votes are the
same, both are essential. Unless you mean a quorum disk. But it should
be at a third location.
Situation: two separate nodes with no shared storage

Configure each node with one vote. Configure each node to use a local
disk as the quorum disk, also with one vote.

As the cluster is formed, the nodes will discover that they do not agree
on the quorum disk and will exclude it, resulting in quorum being
established with the 2 votes provided by the nodes.

One node goes down, the other pauses while it recomputes quorum. In
doing so it discovers there is no longer a conflict regarding the quorum
disk so it includes it, resulting in two votes which re-achieves quorum
and the node continues.

When the failed node comes back up, the quorum disks will be excluded
again and the cluster will return to its original state. The danger of
this configuration is that, if the communication channel between the two
nodes is lost but the nodes remain up, the cluster will partition. This
is addressed in my original posting.

Mark Berryman
Phillip Helbig (undress to reply)
2021-06-14 04:44:01 UTC
Permalink
Post by Mark Berryman
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
First, I recommend you ignore the suggestions to add a 3rd node to your
cluster. In your situation, it is not really a viable answer.
It would solve most of the problems you mention below, and also could
serve as a test node.
Post by Mark Berryman
There are configurations that will allow a member of a 2-node cluster to
automatically continue in the event that the other node fails.
How? If one has more votes, it is essential. If the votes are the
same, both are essential. Unless you mean a quorum disk. But it should
be at a third location.
Situation: two separate nodes with no shared storage
Configure each node with one vote. Configure each node to use a local
disk as the quorum disk, also with one vote.
As the cluster is formed, the nodes will discover that they do not agree
on the quorum disk and will exclude it, resulting in quorum being
established with the 2 votes provided by the nodes.
One node goes down, the other pauses while it recomputes quorum. In
doing so it discovers there is no longer a conflict regarding the quorum
disk so it includes it, resulting in two votes which re-achieves quorum
and the node continues.
When the failed node comes back up, the quorum disks will be excluded
again and the cluster will return to its original state. The danger of
this configuration is that, if the communication channel between the two
nodes is lost but the nodes remain up, the cluster will partition. This
is addressed in my original posting.
Is this scenario supported?
Michael Moroney
2021-06-14 19:56:00 UTC
Permalink
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
First, I recommend you ignore the suggestions to add a 3rd node to your
cluster. In your situation, it is not really a viable answer.
It would solve most of the problems you mention below, and also could
serve as a test node.
Post by Mark Berryman
There are configurations that will allow a member of a 2-node cluster to
automatically continue in the event that the other node fails.
How? If one has more votes, it is essential. If the votes are the
same, both are essential. Unless you mean a quorum disk. But it should
be at a third location.
Situation: two separate nodes with no shared storage
Configure each node with one vote. Configure each node to use a local
disk as the quorum disk, also with one vote.
As the cluster is formed, the nodes will discover that they do not agree
on the quorum disk and will exclude it, resulting in quorum being
established with the 2 votes provided by the nodes.
One node goes down, the other pauses while it recomputes quorum. In
doing so it discovers there is no longer a conflict regarding the quorum
disk so it includes it, resulting in two votes which re-achieves quorum
and the node continues.
When the failed node comes back up, the quorum disks will be excluded
again and the cluster will return to its original state. The danger of
this configuration is that, if the communication channel between the two
nodes is lost but the nodes remain up, the cluster will partition. This
is addressed in my original posting.
Is this scenario supported?
No!
Mark Berryman
2021-06-15 02:39:48 UTC
Permalink
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
First, I recommend you ignore the suggestions to add a 3rd node to your
cluster.  In your situation, it is not really a viable answer.
It would solve most of the problems you mention below, and also could
serve as a test node.
Post by Mark Berryman
There are configurations that will allow a member of a 2-node cluster to
automatically continue in the event that the other node fails.
How?  If one has more votes, it is essential.  If the votes are the
same, both are essential.  Unless you mean a quorum disk.  But it
should
be at a third location.
Situation: two separate nodes with no shared storage
Configure each node with one vote.  Configure each node to use a local
disk as the quorum disk, also with one vote.
As the cluster is formed, the nodes will discover that they do not agree
on the quorum disk and will exclude it, resulting in quorum being
established with the 2 votes provided by the nodes.
One node goes down, the other pauses while it recomputes quorum.  In
doing so it discovers there is no longer a conflict regarding the quorum
disk so it includes it, resulting in two votes which re-achieves quorum
and the node continues.
When the failed node comes back up, the quorum disks will be excluded
again and the cluster will return to its original state.  The danger of
this configuration is that, if the communication channel between the two
nodes is lost but the nodes remain up, the cluster will partition.  This
is addressed in my original posting.
Is this scenario supported?
No!
According to whom?

It was certainly supported by Digital when they gave me the set up back
in the 80s.

Mark Berryman
Phillip Helbig (undress to reply)
2021-06-15 05:05:20 UTC
Permalink
Post by Mark Berryman
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
When the failed node comes back up, the quorum disks will be excluded
again and the cluster will return to its original state.  The danger of
this configuration is that, if the communication channel between the two
nodes is lost but the nodes remain up, the cluster will partition.  This
is addressed in my original posting.
Is this scenario supported?
No!
According to whom?
It was certainly supported by Digital when they gave me the set up back
in the 80s.
I suppose that it is possible that something was supported in the 1980s
but not today, even if might still work (at least in some cases).
Michael Moroney
2021-06-15 06:51:00 UTC
Permalink
Post by Mark Berryman
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
Post by Phillip Helbig (undress to reply)
Post by Mark Berryman
First, I recommend you ignore the suggestions to add a 3rd node to your
cluster.  In your situation, it is not really a viable answer.
It would solve most of the problems you mention below, and also could
serve as a test node.
Post by Mark Berryman
There are configurations that will allow a member of a 2-node cluster to
automatically continue in the event that the other node fails.
How?  If one has more votes, it is essential.  If the votes are the
same, both are essential.  Unless you mean a quorum disk.  But it
should
be at a third location.
Situation: two separate nodes with no shared storage
Configure each node with one vote.  Configure each node to use a local
disk as the quorum disk, also with one vote.
As the cluster is formed, the nodes will discover that they do not agree
on the quorum disk and will exclude it, resulting in quorum being
established with the 2 votes provided by the nodes.
One node goes down, the other pauses while it recomputes quorum.  In
doing so it discovers there is no longer a conflict regarding the quorum
disk so it includes it, resulting in two votes which re-achieves quorum
and the node continues.
When the failed node comes back up, the quorum disks will be excluded
again and the cluster will return to its original state.  The danger of
this configuration is that, if the communication channel between the two
nodes is lost but the nodes remain up, the cluster will partition.
This
is addressed in my original posting.
Is this scenario supported?
No!
According to whom?
It was certainly supported by Digital when they gave me the set up back
in the 80s.
I worked for the VMS Cluster IO group for Digipaqard (and really still
do the equivalent at VSI but all my recent work is unrelated x86 stuff)
and there was no way our group would EVER approve that. The cluster
"quorum hang" exists for a reason, to explicitly avoid the "split brain"
situation if the channel goes down.

My guess is that you had a salesperson who knew of this "trick" to "get
around" the quorum hang "problem". I agree with Hoff that the 'two
quorum disk definitions' behavior is a bug and I'll try to reproduce it
and enter a problem report if it still exists.

The risks of a split brain cluster is VERY implementation dependent.
Anything from harmless to scrambling your data to destroying a chunk of
your chemical plant or something as the two clusters try to do two
incompatible things with valves or something.
Stephen Hoffman
2021-06-15 14:44:51 UTC
Permalink
Post by Michael Moroney
Post by Mark Berryman
It was certainly supported by Digital when they gave me the set up back
in the 80s.
AFAIK, no. That would not have been supported.

Quoth the current Cluster Systems tome:

"2.3.10. Rules for Specifying Quorum
For the quorum disk's votes to be counted in the total cluster votes,
the following conditions must be met:
On all computers capable of becoming [quorum disk] watchers, you must
specify the same physical device name as a value for the DISK_QUORUM
system parameter. The remaining computers (which must have a blank
value for DISK_QUORUM) recognize the name specified by the first quorum
disk watcher with which they communicate."
Either blank when not watching, or the same physical device when watching.
This whole cluster area needs UI work and preferably a UI overhaul as
the current cluster UI "design" is the manual assembly of accreted
kludges, but that's fodder for another time.
Post by Michael Moroney
I worked for the VMS Cluster IO group for Digipaqard (and really still
do the equivalent at VSI but all my recent work is unrelated x86 stuff)
and there was no way our group would EVER approve that. The cluster
"quorum hang" exists for a reason, to explicitly avoid the "split
brain" situation if the channel goes down.
OpenVMS development did some testing of partitioning with
deliberately-wrong configurations back ~Y2K, and—when we deliberately
misconfigured the parameters and configured the shared hardware for
effect—we could trash entire system disks and entire cluster storage
configurations during boot, before getting to the Username: prompt.
System and app startup alone was enough to seriously corrupt persistent
storage, when uncoordinated.
Post by Michael Moroney
My guess is that you had a salesperson who knew of this "trick" to "get
around" the quorum hang "problem". I agree with Hoff that the 'two
quorum disk definitions' behavior is a bug and I'll try to reproduce it
and enter a problem report if it still exists.
The risks of a split brain cluster is VERY implementation dependent.
Anything from harmless to scrambling your data to destroying a chunk of
your chemical plant or something as the two clusters try to do two
incompatible things with valves or something.
Best case, this particular configuration wedges (entirely) or crashes
(entirely) if (when?) that data link is re-established. Worst cast, it
doesn't, and corruptions ensue. A cluster member shouldn't even boot
and be allowed join the "club" with disparate, non-blank values for
DISK_QUORUM.

AFAIK, the official recommendation would have been more hosts and/or
more connections, and/or using the IPC handler or AMDS/AM/etc to select
a lobe. But not disparate quorum disks.
--
Pure Personal Opinion | HoffmanLabs LLC
Stephen Hoffman
2021-06-14 14:30:44 UTC
Permalink
Post by Mark Berryman
Configure each node with one vote. Configure each node to use a local
disk as the quorum disk, also with one vote.
As the cluster is formed, the nodes will discover that they do not
agree on the quorum disk and will exclude it, resulting in quorum being
established with the 2 votes provided by the nodes.
I'd consider that a bug in OpenVMS.

Any host with an inconsistent DISK_QUORUM quorum disk name
specification should trigger a bugcheck (probably SEPPUCLU) when trying
to join the cluster.

DISK_QUORUM non-blank and non-consistent should not be permitted to
join the cluster.
--
Pure Personal Opinion | HoffmanLabs LLC
Phillip Helbig (undress to reply)
2021-06-14 20:40:35 UTC
Permalink
Post by Stephen Hoffman
Post by Mark Berryman
Configure each node with one vote. Configure each node to use a local
disk as the quorum disk, also with one vote.
As the cluster is formed, the nodes will discover that they do not
agree on the quorum disk and will exclude it, resulting in quorum being
established with the 2 votes provided by the nodes.
I'd consider that a bug in OpenVMS.
That was what was in my mind when I was asking whether it was supported.
I could imagine that it is something which might work but is not
documented and might go away in the future.
Post by Stephen Hoffman
Any host with an inconsistent DISK_QUORUM quorum disk name
specification should trigger a bugcheck (probably SEPPUCLU) when trying
to join the cluster.
DISK_QUORUM non-blank and non-consistent should not be permitted to
join the cluster.
Makes sense, as the suggested setup is obviously an abuse of the
quorum-disk concept.
Bob Gezelter
2021-06-12 09:33:52 UTC
Permalink
We are looking at the possibility of putting VMS boxes in two locations, with Integrity boxes running VSI VMS. This is the very beginning of the research on the possibility of clustering those two servers instead of just having them networked. Probably have to be master/slave since only two nodes and no shared storage.
After reviewing the various cluster docs, they seem to be focused on older technologies like SoNET and DS3 using FDDI bridges (which would allow shared storage). The prospect has a metropolitan area network but I do not have any specs on that as yet.
Are there available docs relevant to running a distributed VMS cluster over a metro area network or fast/big enough VPN tunnel? Or is that just the straight cluster over IP configuration in the docs (which we've never used) that we need to concentrate on?
Thanks
Rich,

Another discussion I have had with clients in the past is where to put the tie-breaker (third node). If the organization has a third physical location, or can inexpensively obtain a small third location, e.g., a drawer in what was referred to as an "Internet Hotel", or for that matter a host in a shared hosting facility with command line access, one can even use an emulator-based instance, e.g., one of the Alpha emulators, as a tie breaker. It merely needs to be a cluster tie-breaker.

- Bob Gezelter, http://www.rlgsc.com
Loading...