Discussion:
Mailbox driver analysis regarding greater than 65535 messages in mailbox
(too old to reply)
Robert A. Brooks
2020-11-30 23:29:40 UTC
Permalink
For the most part, the mailbox driver does not base its behaviour on the
value in UCB$W_MSGCNT, except for three of checks that will return an error
upon a read request if the message count is zero and several other conditions are met.

While it *might* be possible to promote UCB$W_MSGCNT to a longword (but only for V9), that would not
be good enough to solve the complete problem because . . .

In the I/O User's Guide, chapter 4 (the mailbox driver chapter),

"DVI$_DEVDEPEND returns a longword field in which the two low-order bytes contain the number of messages in the mailbox.
(The two high-order bytes are not used and should be ignored.)
This information can also be obtained through the Get Mailbox Information function (see “Get Mailbox Information”)."

In the "Get Mailbox Information" area, it states that the upper word of the first longword in the IOSB returns the number
of messages in the mbx. (figure 4-11 in section 4.2)

For that reason above, there would be no reasonable way to return a 32-bit value containing the message count
without breaking a lot of stuff, given the existing size of the IOSB and the 16-bit region of the UCB$L_DEVDEPEND cell
that's used to hold the message count.

With minimal investigation (about an hour, which includes some email conversation), we suspect the proper way to deal with
this is to return an error to a writer if the mailbox has 65535 messages and to document this limitation.
--
-- Rob
Arne Vajhøj
2020-11-30 23:59:19 UTC
Permalink
Post by Robert A. Brooks
With minimal investigation (about an hour, which includes some email
conversation), we suspect the proper way to deal with
this is to return an error to a writer if the mailbox has 65535 messages
and to document this limitation.
I think it makes sense.

Supporting >=64K properly will break a lot.

And mailboxes should not be used to buffer that
many messages.

But OP had some unique requirements.

Arne

PS: So SS$_TOO_MUCH_IN_THIS_POOR_MAILBOX ? :-)
Marc Van Dyck
2020-12-01 17:44:52 UTC
Permalink
Post by Arne Vajhøj
Post by Robert A. Brooks
With minimal investigation (about an hour, which includes some email
conversation), we suspect the proper way to deal with
this is to return an error to a writer if the mailbox has 65535 messages
and to document this limitation.
I think it makes sense.
Supporting >=64K properly will break a lot.
And mailboxes should not be used to buffer that
many messages.
But OP had some unique requirements.
Which I can handle with 65535 messages. So for me Robert's proposal
is fine.
Post by Arne Vajhøj
Arne
PS: So SS$_TOO_MUCH_IN_THIS_POOR_MAILBOX ? :-)
--
Marc Van Dyck
Dave Froble
2020-12-01 00:49:07 UTC
Permalink
Post by Robert A. Brooks
For the most part, the mailbox driver does not base its behaviour on the
value in UCB$W_MSGCNT, except for three of checks that will return an error
upon a read request if the message count is zero and several other conditions are met.
While it *might* be possible to promote UCB$W_MSGCNT to a longword (but
only for V9), that would not
be good enough to solve the complete problem because . . .
In the I/O User's Guide, chapter 4 (the mailbox driver chapter),
"DVI$_DEVDEPEND returns a longword field in which the two low-order
bytes contain the number of messages in the mailbox.
(The two high-order bytes are not used and should be ignored.)
This information can also be obtained through the Get Mailbox
Information function (see “Get Mailbox Information”)."
In the "Get Mailbox Information" area, it states that the upper word of
the first longword in the IOSB returns the number
of messages in the mbx. (figure 4-11 in section 4.2)
For that reason above, there would be no reasonable way to return a
32-bit value containing the message count
without breaking a lot of stuff, given the existing size of the IOSB and
the 16-bit region of the UCB$L_DEVDEPEND cell
that's used to hold the message count.
With minimal investigation (about an hour, which includes some email
conversation), we suspect the proper way to deal with
this is to return an error to a writer if the mailbox has 65535 messages
and to document this limitation.
I'd tend to agree with that resolution.

I've never considered mailboxes to be used for an in memory (non-paged
pool even) database.

(Even though I've been known to call anything that stores data to be a
database.)

:-)

Thanks for looking into this.
--
David Froble Tel: 724-529-0450
Dave Froble Enterprises, Inc. E-Mail: ***@tsoft-inc.com
DFE Ultralights, Inc.
170 Grimplin Road
Vanderbilt, PA 15486
Hein RMS van den Heuvel
2020-12-01 03:34:25 UTC
Permalink
Post by Robert A. Brooks
For the most part, the mailbox driver does not base its behaviour on the
value in UCB$W_MSGCNT, except for three of checks that will return an error
upon a read request if the message count is zero and several other conditions are met.
With minimal investigation (about an hour, which includes some email conversation), we suspect the proper way to deal with
this is to return an error to a writer if the mailbox has 65535 messages and to document this limitation.
Hmm, as I read the descriptions thusfar applications actually causing there to be more than 64K messages in a mailbox can continue writing more and read them back in order, but cannot get a consistent message count - that part is obvious because of the 16 bit fields to hold and expose the count.
But still they can keep writing and reading peaking above 64K messages only getting a non-success status when the mailbox is empty. Correct?
If so, IMHO that behaviour should not be willingly broken.
Maybe an alternate success status, but even that could break applications testing for status == SS$_NORMAL.

Promoting the USB to LONG would seems to be a no-brainer.
But when exposing the count via SENSEMODE/IOSB perhaps that count value should just return 65535 when UCB$L_MSGCNT >= 65535 ???

I don't think there is a business case to do anything though.
Besides the recent C.O.V post, how many requests/concerns have there been thus far?
The recent C.O.V. post suggests a willingness to work-around to split into multiple mailboxes already.
Would multiple mailboxes have the potential for better performance allowing for more parallel activities or is all mailbox access behind a single spinlock anyway?

Now If someone cracks the seal on the mailbox driver anyway, then at least promote the USB counter to long (for V9 only) notably if it also gives better alignment?

Cheers,
Hein
Marc Van Dyck
2020-12-01 17:42:29 UTC
Permalink
Post by Hein RMS van den Heuvel
But still
they can keep writing and reading peaking above 64K messages only getting a
non-success status when the mailbox is empty. Correct? If so, IMHO that
behaviour should not be willingly broken. Maybe an alternate success status,
but even that could break applications testing for status == SS$_NORMAL.
No. If you look at the test I described in my original copy, you can
write any number of messages you want, but you can't read them all
back. The next time that the message count 16 bits field gets back to
zero, the read operation returns an "end of file" status and the
messages still remaining in the mailbox become inaccessible. In my
test,
I stored 1.000.000 messages, but could only read back some 16.000 of
them.

I would also agree with the proposal to refuse the creation of a
mailbox with more than 65535 positions.
--
Marc Van Dyck
Hein RMS van den Heuvel
2020-12-01 17:50:16 UTC
Permalink
Post by Marc Van Dyck
Post by Hein RMS van den Heuvel
But still
they can keep writing and reading peaking above 64K messages only getting a
No. If you look at the test I described in my original copy, you can
write any number of messages you want, but you can't read them all
back. The next time that the message count 16 bits field gets back to
zero, the read operation returns an "end of file" status and the
messages still remaining in the mailbox become inaccessible.
I just saw that in the original topic "Large mailboxes" - which got badly off-track as usual - thanks for testing!

So I have to revise my suggestion. too bad.
If it is broken as is then it is better to return an error when a send is attempted with UCB$W_MSGCNT being 65535.
I still maintain that it might be enough to promote the UCB count field to an aligned longword and let the reports be incorrect for the time being.
Of course it is better to fix it all, if fixed at all.

Cheers,
Hein.
Stephen Hoffman
2020-12-01 18:25:08 UTC
Permalink
Post by Marc Van Dyck
Post by Hein RMS van den Heuvel
But still they can keep writing and reading peaking above 64K messages
only getting a non-success status when the mailbox is empty. Correct?
If so, IMHO that behaviour should not be willingly broken. Maybe an
alternate success status, but even that could break applications
testing for status == SS$_NORMAL.
No. If you look at the test I described in my original copy, you can
write any number of messages you want, but you can't read them all
back. The next time that the message count 16 bits field gets back to
zero, the read operation returns an "end of file" status and the
messages still remaining in the mailbox become inaccessible. In my
test, I stored 1.000.000 messages, but could only read back some 16.000
of them.
I would also agree with the proposal to refuse the creation of a
mailbox with more than 65535 positions.
Seems there's a bug here, based on the reported behavior.

That count-messages-and-reject fix will work. ADAWI, etc.

So too will fixing the API and simply, well, lying API callers, err,
redefining and documenting that 65535 is now used as "many" in this API
as is common with some other OpenVMS API hackery

For status updates where the latest update is the one that matters,
I'll repeat my earlier reply that using mailboxes is ill-suited. For
status messages and performance counters and ilk, a lossy LIFO is
appropriate, and mailboxes are currently a somewhat-buggy FIFO. UDP and
DTLS are choices for that, SNMP messages, RedFish, etc. Hmmmm, haven't
seen SNMP on the roadmap, and SNMPv2 is somewhere between stale and
insecure, and OpenVMS SNMP is far too mixed up with the morass that is
SMH. But I digress.

Or it's possible here write a mailbox driver that follows the mailbox
model you want, and that implements LIFO. This is a pseudo driver and
lacks hardware and any need for an interrupt context, which reduces the
effort involved for the design and operation and debugging.

Thee's also the OpenVMS Zabbix client around, and a client for Nagios.
This if the remote display already works with SNMP or Zabbix or Nagios
or ilk.
--
Pure Personal Opinion | HoffmanLabs LLC
Andy Burns
2020-12-01 07:35:50 UTC
Permalink
Post by Robert A. Brooks
While it *might* be possible to promote UCB$W_MSGCNT to a longword (but
only for V9), that would not
be good enough to solve the complete problem because . . .
In the I/O User's Guide, chapter 4 (the mailbox driver chapter),
"DVI$_DEVDEPEND returns a longword field in which the two low-order
bytes contain the number of messages in the mailbox.
(The two high-order bytes are not used and should be ignored.)
If count exceeds MAXUINT16, return MAXUINT16 rather than the actual count?
Loading...