Discussion:
Do any disks still lie about writing data to permanent storage ?
(too old to reply)
Simon Clubley
2021-07-06 13:04:46 UTC
Permalink
Talking about disks has reminded me of something else.

In the not too distant past, some disks lied about having written data
to permanent storage instead of merely to some internal cache that
would be lost on a power failure.

Do any disks still lie about having committed data to permanent storage
or have we moved past that ?

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Jan-Erik Söderholm
2021-07-06 13:28:24 UTC
Permalink
Post by Simon Clubley
Talking about disks has reminded me of something else.
In the not too distant past, some disks lied about having written data
to permanent storage instead of merely to some internal cache that
would be lost on a power failure.
They did not lie, of course. They said that the data was in the cache,
which was perfectly true.
Post by Simon Clubley
Do any disks still lie about having committed data to permanent storage
or have we moved past that ?
I would expect any modern SAN system to have extensive caching
to reach good performance. Adn that the I/O finish when the data
is in the cache.

And why would the cache be lost at power failure?

I think that your way of putting your question is "questionable".
chris
2021-07-06 16:50:00 UTC
Permalink
Post by Jan-Erik Söderholm
Post by Simon Clubley
Talking about disks has reminded me of something else.
In the not too distant past, some disks lied about having written data
to permanent storage instead of merely to some internal cache that
would be lost on a power failure.
They did not lie, of course. They said that the data was in the cache,
which was perfectly true.
Post by Simon Clubley
Do any disks still lie about having committed data to permanent storage
or have we moved past that ?
I would expect any modern SAN system to have extensive caching
to reach good performance. Adn that the I/O finish when the data
is in the cache.
And why would the cache be lost at power failure?
Unless the cache has battery backup, all cache data will be lost on
power fail. Most have, but that only provides a limited time
to flush out to disk, typically minutes and that needs the drives
spunup and online to work.

Most modern drives have a large on board cache and that will be lost
on power fail, unless the system has a UPS. There are advantages for
cache, but it needs other system support to work properly...
Post by Jan-Erik Söderholm
I think that your way of putting your question is "questionable".
Simon Clubley
2021-07-06 18:11:25 UTC
Permalink
Post by Jan-Erik Söderholm
Post by Simon Clubley
Talking about disks has reminded me of something else.
In the not too distant past, some disks lied about having written data
to permanent storage instead of merely to some internal cache that
would be lost on a power failure.
They did not lie, of course. They said that the data was in the cache,
which was perfectly true.
We are not talking about SAN systems, we were talking about disk drives
such as Sata/SCSI/IDE disks. It's clear you have no experience working
with disk drives at the hardware level or you would have recognised the
issue I am raising.

Those disks have APIs built into them that tells the operating system
when the data in question has been committed to permanent storage by
the disk itself.

Unfortunately, in the past some disks have lied about this and have
told the operating system that the data has been committed to permanent
storage when in fact it is still in a cache on the disk.

That means if one of these disks suffers a power failure before the data
_is_ written to permanent storage, then that data is lost. This is after
the operating system has been told the data is safe and has potentially
done other things based on that. :-(
Post by Jan-Erik Söderholm
Post by Simon Clubley
Do any disks still lie about having committed data to permanent storage
or have we moved past that ?
I would expect any modern SAN system to have extensive caching
to reach good performance. Adn that the I/O finish when the data
is in the cache.
And why would the cache be lost at power failure?
I think that your way of putting your question is "questionable".
Next time Jan-Erik, before saying my question is "questionable", you
may want to ask yourself if in fact I am asking about something that
you are unaware of as is clearly the case here.

This was a well-known problem in the past for some cases for disk device
driver writers. I am just curious if it is still a problem these days
because that is something I don't know.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Jan-Erik Söderholm
2021-07-06 23:37:51 UTC
Permalink
Post by Simon Clubley
Post by Jan-Erik Söderholm
Post by Simon Clubley
Talking about disks has reminded me of something else.
In the not too distant past, some disks lied about having written data
to permanent storage instead of merely to some internal cache that
would be lost on a power failure.
They did not lie, of course. They said that the data was in the cache,
which was perfectly true.
We are not talking about SAN systems, we were talking about disk drives
such as Sata/SCSI/IDE disks. It's clear you have no experience working
with disk drives at the hardware level or you would have recognised the
issue I am raising.
BS! I have worked with PDP-11/VAX/Alpha system from the 80s and forward
incl MV-II, MicroVAX and up from that. And now including large IBM SAN
systems. Using all sorts of different physical disk drives.
Post by Simon Clubley
Those disks have APIs built into them that tells the operating system
when the data in question has been committed to permanent storage by
the disk itself.
Unfortunately, in the past some disks have lied about this and have
told the operating system that the data has been committed to permanent
storage when in fact it is still in a cache on the disk.
That was not a "lie", it did so by design. And in many cases (or at least
some) you could configure the drive to not do so, if you wanted.
Post by Simon Clubley
That means if one of these disks suffers a power failure before the data
_is_ written to permanent storage, then that data is lost. This is after
the operating system has been told the data is safe and has potentially
done other things based on that. :-(
*IF* this has been a real issue, well, it had been known as a real issue.
And if you concidered this to be a real issue, you selected your disk
hardware accordingly.
Post by Simon Clubley
Post by Jan-Erik Söderholm
Post by Simon Clubley
Do any disks still lie about having committed data to permanent storage
or have we moved past that ?
I would expect any modern SAN system to have extensive caching
to reach good performance. Adn that the I/O finish when the data
is in the cache.
And why would the cache be lost at power failure?
I think that your way of putting your question is "questionable".
Next time Jan-Erik, before saying my question is "questionable", you
may want to ask yourself if in fact I am asking about something that
you are unaware of as is clearly the case here.
BS! What I'm saying is that your are asking basied questions by
calling something a "lie", that might just as well be well known
design decisions. And yes, this was well know from 80s and forward.
Anyone working with disk drives knew this and made their own decissions.

I certenly know what you are asking about.

It is still not a "lie", which is what I'm pointing out. It was just
the way that (some) disk drives worked (and maybe works).

But you asked about today. I would not concider this an issue today,
since most storage is not based on single physical disks, at least
not in storage used from VMS systems of today.

Lying is something a human does, not some bit of hardware.
k***@gmail.com
2021-07-07 00:42:40 UTC
Permalink
-----Original Message-----
Söderholm via Info-vax
Sent: July-06-21 8:38 PM
Subject: Re: [Info-vax] Do any disks still lie about writing data to
permanent
storage ?
Post by Simon Clubley
Post by Jan-Erik Söderholm
Post by Simon Clubley
Talking about disks has reminded me of something else.
In the not too distant past, some disks lied about having written
data to permanent storage instead of merely to some internal cache
that would be lost on a power failure.
They did not lie, of course. They said that the data was in the
cache, which was perfectly true.
We are not talking about SAN systems, we were talking about disk
drives such as Sata/SCSI/IDE disks. It's clear you have no experience
working with disk drives at the hardware level or you would have
recognised the issue I am raising.
BS! I have worked with PDP-11/VAX/Alpha system from the 80s and forward
incl MV-II, MicroVAX and up from that. And now including large IBM SAN
systems. Using all sorts of different physical disk drives.
Post by Simon Clubley
Those disks have APIs built into them that tells the operating system
when the data in question has been committed to permanent storage by
the disk itself.
Unfortunately, in the past some disks have lied about this and have
told the operating system that the data has been committed to
permanent storage when in fact it is still in a cache on the disk.
That was not a "lie", it did so by design. And in many cases (or at least
some) you could configure the drive to not do so, if you wanted.
Post by Simon Clubley
That means if one of these disks suffers a power failure before the
data _is_ written to permanent storage, then that data is lost. This
is after the operating system has been told the data is safe and has
potentially done other things based on that. :-(
*IF* this has been a real issue, well, it had been known as a real issue.
And if you concidered this to be a real issue, you selected your disk
hardware
accordingly.
Post by Simon Clubley
Post by Jan-Erik Söderholm
Post by Simon Clubley
Do any disks still lie about having committed data to permanent
storage or have we moved past that ?
I would expect any modern SAN system to have extensive caching to
reach good performance. Adn that the I/O finish when the data is in
the cache.
And why would the cache be lost at power failure?
I think that your way of putting your question is "questionable".
Next time Jan-Erik, before saying my question is "questionable", you
may want to ask yourself if in fact I am asking about something that
you are unaware of as is clearly the case here.
BS! What I'm saying is that your are asking basied questions by calling
something a "lie", that might just as well be well known design decisions.
And
yes, this was well know from 80s and forward.
Anyone working with disk drives knew this and made their own decissions.
I certenly know what you are asking about.
It is still not a "lie", which is what I'm pointing out. It was just the
way that
(some) disk drives worked (and maybe works).
But you asked about today. I would not concider this an issue today, since
most storage is not based on single physical disks, at least not in storage
used
from VMS systems of today.
Lying is something a human does, not some bit of hardware.
As Jan-Erik correctly stated, there are no "lies".

The facts are that regardless of the physical server/storage device, there
are different write strategies that one can choose to implement based on
performance and data integrity requirements. This is an age-old design
decision that goes back decades.

"Write back" strategies are based on data being written to cache (HW or SW)
and then acknowledging the write is complete. Data is written to persistent
storage in the background. This results in higher performance, but lower
data integrity systems. This strategy can be somewhat mitigated with battery
backups on the cache systems. With new disk sizes in multiple TB's, this
write-back strategy can cause issues with the time to do file system
consistency checks after system crashes or power failures.

"Write through" strategies are based on data being written to persistent
storage before acknowledging the write is complete. These are slower
performing systems but higher data integrity file systems.

Most storage controllers today (and many from decades ago) offer design
options to implement one write strategy or the other.

By default, the OpenVMS file system is based on an overall write through
strategy.

In case anyone wants to understand the huge difference in latency between a
cache write vs. a disk IO write vs. a network IO, check out this link:
<https://gist.github.com/hellerbarde/2843375>

On a semi-related note, while it has not really picked up steam to much,
Intel and other vendors have released new low latency solid state memory
solutions that emulate storage i.e. in addition to high performance, low
latency, they also have data persistence. This means the in-memory data is
protected after power failures.

Reference:
<https://www.intel.ca/content/www/ca/en/support/articles/000024018/memory-an
d-storage/intel-optane-memory.html>

Write through vs. write-back and overall data persistence is just one of
many design decisions that one should take into consideration.


Regards,

Kerry Main
Kerry dot main at starkgaming dot com
--
This email has been checked for viruses by AVG.
https://www.avg.com
Simon Clubley
2021-07-07 12:24:06 UTC
Permalink
Post by k***@gmail.com
As Jan-Erik correctly stated, there are no "lies".
The facts are that regardless of the physical server/storage device, there
are different write strategies that one can choose to implement based on
performance and data integrity requirements. This is an age-old design
decision that goes back decades.
Since clearly neither you or Jan-Erik understand the issues being
discussed, here is some reading for you:

https://queue.acm.org/detail.cfm?id=2367378

There's no need to read the last bit about 4K sectors if you don't
want to as it's not applicable to this discussion.

In summary, yes, there has indeed been an historical issue with some
disks lying to the disk device driver about whether the data had made
it to permanent storage yet.

This has nothing to do with operating system level deferred write caching.
This is to do with some drives lying about the guarantee they are supposed
to give to the device driver controlling them.

What I don't know is if it's still a problem today.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Jan-Erik Söderholm
2021-07-07 13:40:24 UTC
Permalink
Post by Simon Clubley
Post by k***@gmail.com
As Jan-Erik correctly stated, there are no "lies".
The facts are that regardless of the physical server/storage device, there
are different write strategies that one can choose to implement based on
performance and data integrity requirements. This is an age-old design
decision that goes back decades.
Since clearly neither you or Jan-Erik understand the issues being
https://queue.acm.org/detail.cfm?id=2367378
There's no need to read the last bit about 4K sectors if you don't
want to as it's not applicable to this discussion.
In summary, yes, there has indeed been an historical issue with some
disks lying to the disk device driver about whether the data had made
it to permanent storage yet.
This has nothing to do with operating system level deferred write caching.
This is to do with some drives lying about the guarantee they are supposed
to give to the device driver controlling them.
What I don't know is if it's still a problem today.
Simon.
The disk was not lying. It said "I have the data in my cache".
You just had to understand that it was that it was saying.

It is you that used the concept of "lying" and that is not
applicable to hardware. It is humans that lies, not machines.

I'm not depating the base issue here. You are completely right
about that... :-)

Jan-Erik.
Simon Clubley
2021-07-07 17:36:00 UTC
Permalink
Post by Jan-Erik Söderholm
The disk was not lying. It said "I have the data in my cache".
You just had to understand that it was that it was saying.
Well, it's not allowed to do that. The drive APIs in question say
the drive can only return this status to the device driver after the
data has been written to permanent storage. That's the whole point
of those drive APIs.

If it returns that status and the data isn't yet in permanent storage,
then the drive is lying to the device driver about the status of
the data.
Post by Jan-Erik Söderholm
It is you that used the concept of "lying" and that is not
applicable to hardware. It is humans that lies, not machines.
A human wrote the firmware that is lying in this case.

I also note the article I pointed you to called it lying as well.
Post by Jan-Erik Söderholm
I'm not depating the base issue here. You are completely right
about that... :-)
Good to hear. :-) I don't know if it's actually a different culture
thing, but I do indeed regard the disk drive as lying if it returns
that "data committed to permanent storage" status while the data is
still in the drive's cache.

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
Jan-Erik Söderholm
2021-07-07 18:08:22 UTC
Permalink
Post by Simon Clubley
Post by Jan-Erik Söderholm
The disk was not lying. It said "I have the data in my cache".
You just had to understand that it was that it was saying.
Well, it's not allowed to do that. The drive APIs in question say
the drive can only return this status to the device driver after the
data has been written to permanent storage. That's the whole point
of those drive APIs.
If it returns that status and the data isn't yet in permanent storage,
then the drive is lying to the device driver about the status of
the data.
Post by Jan-Erik Söderholm
It is you that used the concept of "lying" and that is not
applicable to hardware. It is humans that lies, not machines.
A human wrote the firmware that is lying in this case.
I also note the article I pointed you to called it lying as well.
Post by Jan-Erik Söderholm
I'm not depating the base issue here. You are completely right
about that... :-)
Good to hear. :-) I don't know if it's actually a different culture
thing, but I do indeed regard the disk drive as lying if it returns
that "data committed to permanent storage" status while the data is
still in the drive's cache.
Simon.
The message returned might be wrong, but that doesn't make it into
a lie, a lie needs an intention from the one lying. Hardware does
not have any underlying intentions, that is a human thing...

Note that you could just have said "giving back the wrong message".
Or "giving back a missleading message".

I can tell you something that is wrong without lying, just becuse
I didn't knew better but I had no intention to lie to you. I was
simply wrong.
Dennis Boone
2021-07-06 18:42:01 UTC
Permalink
Post by Simon Clubley
Do any disks still lie about having committed data to permanent storage
or have we moved past that ?
Do any _not_ do so? :)

De
Simon Clubley
2021-07-07 12:06:35 UTC
Permalink
Post by Dennis Boone
Post by Simon Clubley
Do any disks still lie about having committed data to permanent storage
or have we moved past that ?
Do any _not_ do so? :)
:-)

Simon.
--
Simon Clubley, ***@remove_me.eisner.decus.org-Earth.UFP
Walking destinations on a map are further away than they appear.
gah4
2021-07-07 22:32:11 UTC
Permalink
Post by Simon Clubley
Talking about disks has reminded me of something else.
In the not too distant past, some disks lied about having written data
to permanent storage instead of merely to some internal cache that
would be lost on a power failure.
Do any disks still lie about having committed data to permanent storage
or have we moved past that ?
There was a big discussion of this some years ago related to NFS servers.

With synchronous writes (traditional default and still recommended)
the server is not supposed to acknowledge the write until (as you note)
it is on permanent storage.

For most computers, and most uses, this isn't really a problem.
If the computer powers off, along with the drive, the indication that it
was or wasn't written is likely lost at the same time.

In the case of an NFS server, though, the server could be powered down
after acknowledging the write, but before it was out of the drives cache.
The client might be somewhere else, and stay powered on.

Many systems, though, now use journaling file systems, which helps with
some of the problems. In theory, the drive has a small amount of time
before the filter capacitors discharge and the disk stops, to still get
data onto disk.

As noted above, in many cases it is the drive acknowledging the
transfer of data across the bus, and releasing the bus for the
next operation, and not that the data is on permanent storage.

NFS has read-ahead and write-behind, so that it can avoid the
large latency that would otherwise exist.

Loading...