USB related questions for Itanium2

[Neil Rieck]

Back in the day, almost every AlphaServer server hosted a 3.5 inch floppy drive and there were numerous OpenVMS tools to read/write DOS-formatted media. (PCDISK, MGPCX, WINFX, etc.)

On the flip side, every Itanium server hosts at least one USB port which can be manipulated through the UCM utility so here's my question: if we connected a DOS or Windows formatted drive to the USB port of our server, is there any OpenVMS software available which could read it?

ps-1: we use a Windows-7 server as an el-cheapo OpenVMS backup repository for our whole computer room. We create whole-disk save-set images which are FTP'd to a 1-TB drive on the Windows box each night over a private 1000-Gb/s ethernet. Since these drives only cost $100 each, we rotate through a set of 28. While we have never had to do an emergency restore in over a decade, I'm just looking for a fast way to get one of the save-sets off the windows median in an emergency without resorting to using a network (eg. just plug the windows drive into the USB port)

ps-2: I haven't been able to access http://www.decuslib.com/ for more than 24-hours now. Is it off the air?

Neil Rieck
Waterloo, Ontario, Canada.
http://neilrieck.net

[Simon Clubley]

Post by Neil Rieck
Back in the day, almost every AlphaServer server hosted a 3.5 inch floppy

drive and there were numerous OpenVMS tools to read/write DOS-formatted media.
(PCDISK, MGPCX, WINFX, etc.)

Post by Neil Rieck
On the flip side, every Itanium server hosts at least one USB port which
can be manipulated through the UCM utility so here's my question: if we
connected a DOS or Windows formatted drive to the USB port of our server,
is there any OpenVMS software available which could read it?

Before anyone can answer that question, you need to specify the file system
in use on the drives. Is it FAT-32, NTFS, or something else ?

Also, does the backup drive use partitions (even just a single partition)
and if so, how do the Itanium drivers handle drives with partitions ?

Post by Neil Rieck
ps-1: we use a Windows-7 server as an el-cheapo OpenVMS backup repository
for our whole computer room. We create whole-disk save-set images which are
FTP'd to a 1-TB drive on the Windows box each night over a private
1000-Gb/s ethernet. Since these drives only cost $100 each, we rotate
through a set of 28. While we have never had to do an emergency restore in
over a decade, I'm just looking for a fast way to get one of the save-sets
off the windows median in an emergency without resorting to using a network
(eg. just plug the windows drive into the USB port)

I hope you at least do a test restore or compare every so often to verify
that what is being written to the drives is what you think is being written.

Also, how do you handle/detect bad blocks or a transient error on the
destination drive when you writing your backups to it ?

Post by Neil Rieck
ps-2: I haven't been able to access http://www.decuslib.com/ for more
than 24-hours now. Is it off the air?

I hope not. :-) I'm planning on sending you all there next month to
download the Macro-64 assembler for Alpha...

Simon.

[Stephen Hoffman]

Post by Neil Rieck
Back in the day, almost every AlphaServer server hosted a 3.5 inch
floppy drive and there were numerous OpenVMS tools to read/write
DOS-formatted media. (PCDISK, MGPCX, WINFX, etc.)
On the flip side, every Itanium server hosts at least one USB port
if we connected a DOS or Windows formatted drive to the USB port of our
server, is there any OpenVMS software available which could read it?

Wouldn't be my choice. Some of the older-era FAT-reading tools are
listed in section 7.2 of the old OpenVMS FAQ.
http://www.hoffmanlabs.com/vmsfaq/vmsfaq.pdf and there are also some
ODS readers for Windows NT around. Which is clearly the same list that
you're already familiar with.

Post by Neil Rieck
ps-1: we use a Windows-7 server as an el-cheapo OpenVMS backup
repository for our whole computer room. We create whole-disk save-set
images which are FTP'd to a 1-TB drive on the Windows box each night
over a private 1000-Gb/s ethernet.

I'm going to assume that's gigabit Ethernet network; GbE. Fastest
OpenVMS NICs are 10 GbE models and GbE NICs are common on Itanium and
latter-era Alpha, and Windows has support for yet faster NICs.

Post by Neil Rieck
Since these drives only cost $100 each, we rotate through a set of 28.
While we have never had to do an emergency restore in over a decade,
I'm just looking for a fast way to get one of the save-sets off the
windows median in an emergency without resorting to using a network
(eg. just plug the windows drive into the USB port)

It'll need to be FAT-format volumes and not the more common NTFS-format
volumes as there are no NTFS readers available for OpenVMS AFAIK.

I'd probably set up an emergency network anew, as a couple of cables
and a Windows box will get you were you need, without having to try to
read and process FAT-format files directly, via USB-device swapping.
But if you do go the FAT-format disk swapping, test it. OpenVMS USB
support was pretty slow for large disks, too. Officially, "USB 2.0
High Speed" allowed 480 Mbps, but even with the USB connection oriented
to allow the bits to flow downhill with a bit-tailwind, I'd expect to
get maybe ~200 Mbps at most and likely less than that.

Given the choice and particularly if you need go-fast recovery, I'd try
scrounging some 10 GbE NICs for the associated boxes, if you don't
already have those. If you need to perform a cold restart and reload
for OpenVMS itself, create a bootable OpenVMS disk with everything you
need to recover the base box (hard disk or optical or USB flash), a GbE
or 10 GbE switch, test it a few times, and then keep that with your
backups.

The OpenVMS boot disk won't change all that often in most environments,
so keeping a copy for three or six months or for a year is often
entirely feasible.

As an alternative to that old Windows Server box and TB USB disk
configuration, 24, 32, 48 TB and larger RAID arrays are easily
available, not all that expensive, and don't involve re-cabling and
avoid tripping over a bazillion separate USB disks and cables.
https://www.promise.com/Products/Pegasus/Pegasus2
https://www.apple.com/mac-mini/
etc. There are other vendors of servers and arrays.

I'd also be running or upgrading Windows and Windows Server to Windows
10 and Windows Server 2016 at this point. 2012 R2 is getting old, and
Windows 7 is ancient.

It wouldn't surprise me to learn that the "official" recommendation
from HPE may well involve a FC HBA and a shared FC tape library or some
such, as a completely different approach.

Use /zip "-V"/ to protect the RMS metadata and the BACKUP saveset
attributes (and with zip 3.0 and unzip 6.0, minimally), and expect to
have to fix any RMS files that have been stored directly on the FAT or
NTFS-format volumes; BACKUP savesets or otherwise.

Or I'd flip this around, and would use the USB disks directly (and
slowly) from OpenVMS, recording the data using the native ODS-5 format,
and storing those disks as your backup. Yeah, you're probably trying
to "split the difference" here between swapping those disks around and
the relative ease of network automation via the network-based copies
dropping all that stuff onto the Windows Server box, most likely.

FWIW, there've been other discussions of reading FAT28-, err,
FAT32-format volumes and other FAT-format volumes here in comp.os.vms,
too. OpenVMS doesn't have any integrated tools for reading or writing
FAT28-, err, FAT32-format volumes AFAIK, and that or a related format
will almost certainly be what you're using here if not NTFS, and what
is around on OpenVMS wasn't (still isn't?) supported for this sort of
usage.

Whatever approach you decide to use here, test it and document it.

One of the other issues with OpenVMS is getting hot backups of live
files, absent a database with that support. But of course we can't
have that database support integrated in some future version, because
reasons.

Post by Neil Rieck
ps-2: I haven't been able to access http://www.decuslib.com/ for more
than 24-hours now. Is it off the air?

http://downforeveryoneorjustme.com
http://www.digiater.nl

[Neil Rieck]

Oops! Thanks Steve for pointing out that FUBAR which probably posted during a low caffeine level. I meant to type 1000-Mb/s.

On a related note. I was looking at your FAQ yesterday and found most of the reference links dead. I guess that is to be expected since it was dated "September 2006". Do you have any plans to update the FAQ?

And more importantly, do you have any plans to bring back the rest of your very valuable web site?

Neil

[Chris]

Post by Neil Rieck
Back in the day, almost every AlphaServer server hosted a 3.5 inch
floppy drive and there were numerous OpenVMS tools to read/write
DOS-formatted media. (PCDISK, MGPCX, WINFX, etc.)
On the flip side, every Itanium server hosts at least one USB port
which can be manipulated through the UCM utility so here's my
question: if we connected a DOS or Windows formatted drive to the USB
port of our server, is there any OpenVMS software available which
could read it?

Wouldn't be my choice. Some of the older-era FAT-reading tools are
listed in section 7.2 of the old OpenVMS FAQ.
http://www.hoffmanlabs.com/vmsfaq/vmsfaq.pdf and there are also some ODS
readers for Windows NT around. Which is clearly the same list that
you're already familiar with.

Post by Neil Rieck
ps-1: we use a Windows-7 server as an el-cheapo OpenVMS backup
repository for our whole computer room. We create whole-disk save-set
images which are FTP'd to a 1-TB drive on the Windows box each night
over a private 1000-Gb/s ethernet.

I'm going to assume that's gigabit Ethernet network; GbE. Fastest
OpenVMS NICs are 10 GbE models and GbE NICs are common on Itanium and
latter-era Alpha, and Windows has support for yet faster NICs.

Post by Neil Rieck
Since these drives only cost $100 each, we rotate through a set of 28.
While we have never had to do an emergency restore in over a decade,
I'm just looking for a fast way to get one of the save-sets off the
windows median in an emergency without resorting to using a network
(eg. just plug the windows drive into the USB port)

It'll need to be FAT-format volumes and not the more common NTFS-format
volumes as there are no NTFS readers available for OpenVMS AFAIK.
I'd probably set up an emergency network anew, as a couple of cables and
a Windows box will get you were you need, without having to try to read
and process FAT-format files directly, via USB-device swapping. But if
you do go the FAT-format disk swapping, test it. OpenVMS USB support was
pretty slow for large disks, too. Officially, "USB 2.0 High Speed"
allowed 480 Mbps, but even with the USB connection oriented to allow the
bits to flow downhill with a bit-tailwind, I'd expect to get maybe ~200
Mbps at most and likely less than that.
Given the choice and particularly if you need go-fast recovery, I'd try
scrounging some 10 GbE NICs for the associated boxes, if you don't
already have those. If you need to perform a cold restart and reload for
OpenVMS itself, create a bootable OpenVMS disk with everything you need
to recover the base box (hard disk or optical or USB flash), a GbE or 10
GbE switch, test it a few times, and then keep that with your backups.
The OpenVMS boot disk won't change all that often in most environments,
so keeping a copy for three or six months or for a year is often
entirely feasible.
As an alternative to that old Windows Server box and TB USB disk
configuration, 24, 32, 48 TB and larger RAID arrays are easily
available, not all that expensive, and don't involve re-cabling and
avoid tripping over a bazillion separate USB disks and cables.
https://www.promise.com/Products/Pegasus/Pegasus2
https://www.apple.com/mac-mini/
etc. There are other vendors of servers and arrays.
I'd also be running or upgrading Windows and Windows Server to Windows
10 and Windows Server 2016 at this point. 2012 R2 is getting old, and
Windows 7 is ancient.
It wouldn't surprise me to learn that the "official" recommendation from
HPE may well involve a FC HBA and a shared FC tape library or some such,
as a completely different approach.
Use /zip "-V"/ to protect the RMS metadata and the BACKUP saveset
attributes (and with zip 3.0 and unzip 6.0, minimally), and expect to
have to fix any RMS files that have been stored directly on the FAT or
NTFS-format volumes; BACKUP savesets or otherwise.
Or I'd flip this around, and would use the USB disks directly (and
slowly) from OpenVMS, recording the data using the native ODS-5 format,
and storing those disks as your backup. Yeah, you're probably trying to
"split the difference" here between swapping those disks around and the
relative ease of network automation via the network-based copies
dropping all that stuff onto the Windows Server box, most likely.
FWIW, there've been other discussions of reading FAT28-, err,
FAT32-format volumes and other FAT-format volumes here in comp.os.vms,
too. OpenVMS doesn't have any integrated tools for reading or writing
FAT28-, err, FAT32-format volumes AFAIK, and that or a related format
will almost certainly be what you're using here if not NTFS, and what is
around on OpenVMS wasn't (still isn't?) supported for this sort of usage.
Whatever approach you decide to use here, test it and document it.
One of the other issues with OpenVMS is getting hot backups of live
files, absent a database with that support. But of course we can't have
that database support integrated in some future version, because reasons.

Post by Neil Rieck
ps-2: I haven't been able to access http://www.decuslib.com/ for more
than 24-hours now. Is it off the air?

http://downforeveryoneorjustme.com
http://www.digiater.nl

I wouldn't use usb based drives for any serious work, as they tend
to be consumer grade designs with minimal parts count to minimise cost.
Not only that, but the usb stacks and drivers can be a bit marginal
on some systems as well, with speeds are often only a fraction of rated
interface speed.

Nor would I use usb sticks, as they are notoriously unreliable. Fine for
backing up family snaps etc or copying between systems, but not much
else...

Regards,

Chris

[David Froble]

Post by Chris
Nor would I use usb sticks, as they are notoriously unreliable. Fine for
backing up family snaps etc or copying between systems, but not much
else...

Ya know, there are some who might feel those family snaps are rather important.

Around here, multiple copies are maintained, on multiple systems, plus backups.
No USB sticks.

[Bill Gunshannon]

Post by David Froble

Post by Chris
Nor would I use usb sticks, as they are notoriously unreliable. Fine for
backing up family snaps etc or copying between systems, but not much
else...

Ya know, there are some who might feel those family snaps are rather important.
Around here, multiple copies are maintained, on multiple systems, plus
backups.  No USB sticks.

Unreliable? I have USB sticks that are over a decade old. I have never
had one go bad before I was long done using it.

bill

[Scott Dorsey]

Post by Bill Gunshannon

Post by David Froble

Post by Chris
Nor would I use usb sticks, as they are notoriously unreliable. Fine for
backing up family snaps etc or copying between systems, but not much
else...

Ya know, there are some who might feel those family snaps are rather important.
Around here, multiple copies are maintained, on multiple systems, plus
backups.  No USB sticks.

Unreliable? I have USB sticks that are over a decade old. I have never
had one go bad before I was long done using it.

They go bad from writing, not from reading. They are a good choice for a
read-only application like transferring boot files to numerous machines,
but they wear out pretty quickly with regular writing.

They will also go bad from general charge leakage sitting on the shelf,
but a lot of that depends on manufacturing factors that you can't really
identify easily. You can make the assumption that smaller ones are likely
to last longer than larger ones although that is not necessarily true.
The smaller the cells and the more bits they try to store per cell, the
more of an issue that will be.
--scott

[Dennis Boone]

Post by Scott Dorsey
They will also go bad from general charge leakage sitting on the shelf,
but a lot of that depends on manufacturing factors that you can't really
identify easily. You can make the assumption that smaller ones are likely
to last longer than larger ones although that is not necessarily true.
The smaller the cells and the more bits they try to store per cell, the
more of an issue that will be.

I'm given to understand that 8-level mylar punched tape might survive
thousands of years of burial. ;)

De

[Scott Dorsey]

Post by Dennis Boone

Post by Scott Dorsey
They will also go bad from general charge leakage sitting on the shelf,
but a lot of that depends on manufacturing factors that you can't really
identify easily. You can make the assumption that smaller ones are likely
to last longer than larger ones although that is not necessarily true.
The smaller the cells and the more bits they try to store per cell, the
more of an issue that will be.

I'm given to understand that 8-level mylar punched tape might survive
thousands of years of burial. ;)

Yes.
Same basic issue: whether you are storing bits mechanically, electrostatically,
or magnetically, the more bits you jam into a space, the easier it is for the
media to be damaged.

The problem is that people have an awful lot of bits now. If you made that
1" mylar tape 8000 levels wide with microscopic holes in it, the failure rate
would be much higher.
--scott

[David Froble]

Post by Dennis Boone

Post by Scott Dorsey
They will also go bad from general charge leakage sitting on the shelf,
but a lot of that depends on manufacturing factors that you can't really
identify easily. You can make the assumption that smaller ones are likely
to last longer than larger ones although that is not necessarily true.
The smaller the cells and the more bits they try to store per cell, the
more of an issue that will be.

I'm given to understand that 8-level mylar punched tape might survive
thousands of years of burial. ;)
De

And that does what good, if there is not a reader for it?

:-)

[Simon Clubley]

Post by Scott Dorsey
They go bad from writing, not from reading. They are a good choice for a
read-only application like transferring boot files to numerous machines,
but they wear out pretty quickly with regular writing.
They will also go bad from general charge leakage sitting on the shelf,
but a lot of that depends on manufacturing factors that you can't really
identify easily.

I have had data on USB sticks go bad on me long after it was written.

Whenever I backup data to a USB stick, I do a _full_ verify pass on
the whole directory tree and this has sometimes caught data blocks
failing which have not have been written to for a long time.

Simon.

[Michael Moroney]

Post by Bill Gunshannon

Post by David Froble

Post by Chris
Nor would I use usb sticks, as they are notoriously unreliable. Fine for
backing up family snaps etc or copying between systems, but not much
else...

Ya know, there are some who might feel those family snaps are rather important.
Around here, multiple copies are maintained, on multiple systems, plus
backups.  No USB sticks.

Unreliable? I have USB sticks that are over a decade old. I have never
had one go bad before I was long done using it.

I have had one go bad, in a slightly unusual way. It was in a metal case,
shaped like a key. A freebie (from a VMS boot camp). When it failed, it
showed up as a USB drive with a size of 0. It also got HOT when left
in the socket, with the metal making the whole "key" hot.

In a way, too bad, I did want to leave it on a key ring so if I had a
random need to grab or transfer a file I'd always have a USB drive with
me.

Some USB drives are interesting when used on VMS Itaniums. When being
written, the writing process shows up as in RWAST state, not the usual
LEF. The process makes progress however and the writes complete OK.
Probably a combo of a slow USB stick and some weird bug in the USB
driver.

[Snowshoe]

Post by Neil Rieck
Back in the day, almost every AlphaServer server hosted a 3.5 inch floppy drive and there were numerous OpenVMS tools to read/write DOS-formatted media. (PCDISK, MGPCX, WINFX, etc.)
On the flip side, every Itanium server hosts at least one USB port which can be manipulated through the UCM utility so here's my question: if we connected a DOS or Windows formatted drive to the USB port of our server, is there any OpenVMS software available which could read it?
ps-1: we use a Windows-7 server as an el-cheapo OpenVMS backup repository for our whole computer room. We create whole-disk save-set images which are FTP'd to a 1-TB drive on the Windows box each night over a private 1000-Gb/s ethernet. Since these drives only cost $100 each, we rotate through a set of 28. While we have never had to do an emergency restore in over a decade, I'm just looking for a fast way to get one of the save-sets off the windows median in an emergency without resorting to using a network (eg. just plug the windows drive into the USB port)

Is there a decent quality freeware or open source version of the NTFS
file system, ones that are easily adaptable? How about FAT-32 & exfat?

[IanD]

Linux can read NTFS and can write as well, although last time I looked at NTFS support under Linux was they said write was unsupported, that was a few years ago now

The only time I have run into issues with zip is with multiple files with the same name and version buried in different directories

I can't remember if it was the -w switch I used to get around this or not

[Hans Vlems]

Run simh or another emulator on the Windows box and let it Access the save sets.
Hans

[Steven Schweda]

Post by IanD
The only time I have run into issues with zip is with
multiple files with the same name and version buried in
different directories

I assume that it was all your fault, but actual details
could be interesting.

Post by IanD
I can't remember if it was the -w switch I used to get
around this or not

As the "-h" help says:

-V save VMS file attributes (-VV also save allocated blocks past EOF)
-w store file version numbers -ww store file version numbers as ".nnn"

If you discard the directory information, then you might
get some conflict when extracting. Otherwise, what could go
wrong?

Post by IanD
Linux can read NTFS and can write as well, [...]

In principle, many things are possible. For a good time,
browse the user forums for Netgear routers (which are
GNU/Linux-based), and look for problems with ReadySHARE.
That's the pile of freeware which is supposed to deal with
file sharing using USB-connected disks with any disk format,
file system, or protocol known to man. It's a grand idea,
but the execution seems to lack something. Likely, Netgear
doesn't keep their stuff current, and their testing makes
mine look good.

[Neil Rieck]

Post by IanD
Linux can read NTFS and can write as well, although last time I looked at NTFS support under Linux was they said write was unsupported, that was a few years ago now
The only time I have run into issues with zip is with multiple files with the same name and version buried in different directories
I can't remember if it was the -w switch I used to get around this or not

A few people told me to try MTOOLS but I could only find binaries for VAX and Alpha until I stumbled across Steve Schweda's offering:

http://antinode.info/dec/sw/mtools.html

Thanks Steve!

p.s. Yesterday was a day from hell so I wasn't able to test it.

Neil Rieck
Waterloo, Ontario, Canada.
http://neilrieck.net

[Colin Butcher]

VMS on IA64 has a tool to read/write a FAT partition, because it needs
to manipulate the FAT partition visible from EFI shell.

You could try using EFI$CP and see how well it manages. I've used it for
the occasional small file and small firmware update image successfully -
moving them in and out of the FAT partition from VMS. I've also been
able to move files from the FAT partition to/from a USB flash drive from
EFI shell and vice versa.

Whether EFI$CP is good enough to handle a bigger file or anything over
2GB I don't know. Nor have I yet tried using EFI$CP to directly
manipulate a FAT formatted USB flash drive.

Might be worth a try.

Cheers, Colin.

[Stephen Hoffman]

Post by Colin Butcher
VMS on IA64 has a tool to read/write a FAT partition, because it needs
to manipulate the FAT partition visible from EFI shell.
You could try using EFI$CP and see how well it manages. I've used it
for the occasional small file and small firmware update image
successfully - moving them in and out of the FAT partition from VMS.
I've also been able to move files from the FAT partition to/from a USB
flash drive from EFI shell and vice versa.
Whether EFI$CP is good enough to handle a bigger file or anything over
2GB I don't know. Nor have I yet tried using EFI$CP to directly
manipulate a FAT formatted USB flash drive.

FWIW...

EFI$CP is not supported for that usage. It's intended for managing
the EFI boot partition.

EFI$CP implements FAT12 and FAT16, as that was what was minimally
necessary for the EFI-related bootstrap work. The boot partitions
just don't (yet?) need to be that big. Work on FAT32 was certainly
desirable, but was also beyond what was necessary back then. I'd
doubt that anybody has finished or replaced what I'd implemented,
either; the remaining bits of FAT32 implementation and the requisite
FAT32 testing.

TL;DR: EFI$CP wouldn't be the centerpiece of my recovery strategy, and
certainly not for FAT32.

[David Turner]

Why not just buy a 250GB SSD in a USB credit card sized container.
They are fast as hell, you can map them then do what you need to do...
Work really well with HP Integrity rx2800s

Post by Neil Rieck
Back in the day, almost every AlphaServer server hosted a 3.5 inch floppy drive and there were numerous OpenVMS tools to read/write DOS-formatted media. (PCDISK, MGPCX, WINFX, etc.)
On the flip side, every Itanium server hosts at least one USB port which can be manipulated through the UCM utility so here's my question: if we connected a DOS or Windows formatted drive to the USB port of our server, is there any OpenVMS software available which could read it?
ps-1: we use a Windows-7 server as an el-cheapo OpenVMS backup repository for our whole computer room. We create whole-disk save-set images which are FTP'd to a 1-TB drive on the Windows box each night over a private 1000-Gb/s ethernet. Since these drives only cost $100 each, we rotate through a set of 28. While we have never had to do an emergency restore in over a decade, I'm just looking for a fast way to get one of the save-sets off the windows median in an emergency without resorting to using a network (eg. just plug the windows drive into the USB port)
ps-2: I haven't been able to access http://www.decuslib.com/ for more than 24-hours now. Is it off the air?
Neil Rieck
Waterloo, Ontario, Canada.
http://neilrieck.net

[Simon Clubley]

Post by David Turner
Why not just buy a 250GB SSD in a USB credit card sized container.
They are fast as hell, you can map them then do what you need to do...
Work really well with HP Integrity rx2800s

I hope the VMS USB stack has less flaws in it than the Linux one does: :-)

https://www.theregister.co.uk/2017/11/07/linux_usb_security_bugs/

BTW, note how they were found by using fuzzing techniques.

[Sorry if this is a duplicate. Eternal September has been playing up again.]

Simon.