By Andreas Schmidt

Mirrored Disk/iX is an optional subsystem for HP 3000 mission critical systems, and it’s vital to guaranteeing high availability of your company’s data. This article explains the fundamentals of HP’s Mirrored Disk/iX: how to set up volume sets, how to deal with disk errors and how to establish a split-volume backup.

The complete resource on this subject is, of course, the HP manual for Mirrored Disk/iX (User’s Guide, HP Part No. 30349-90003.) Some of what follows is based on this. But we all know that a summary sometimes will help better than reading through a complete HP manual — especially if you are under pressure in a delicate situation.

What are mirrored disks?

Mirrored Disk/iX is a subsystem for HP 3000s which needs to be ordered separately. The installation follows the normal subsystem Installation Process as documented in the Installation Manual. Mirrored Disk/iX is designed to work only with non-system volumes. To make it very clear: Mirrored Disk/iX does not support mirroring the HP 3000’s system volumes.

It supports disk drives that use HP-FL cards or NIO SCSI cards. But mirrored partners must be the same model of fiber-link drive or NIO SCSI drive, and mirrored partners must be connected to different HP-FL cards or NIO SCSI. Otherwise a single point of failure would still exist.

Mirrored disks are designed to provide high data availability by automatically maintaining identical information on two partner disks. When an application writes to a disk, disk mirroring causes the information to be written to both drive partners. When an application reads from a disk, there are two places to access the requested data. This may give performance benefits on large systems which do a lot of reads for queries but only a few writes to the same data. Applications running on the system are unaware that disk mirroring is present.

Once disk mirrors have been established using the VOLUTIL utility, a mirrored disk acts just like any other disk connected to the system, until a disk failure occurs. If either disk of any pair fails, normal system operation continues. When the partner is ready to resume operation, the system copies data from the good disk, bringing the pair to a consistent state, and normal mirroring resumes.

Once mirrored disks have been installed, you can use them like any other disks connected to the system. Additionally, you can perform split-volume backup of mirrored disk data while still accessing the data.

So, Mirrored Disk/iX supports the following features:

High data availability: System automatically maintains identical information on two partner disks. Users continue to access data if either disk of any pair is disabled or under repair.

Reduced downtime: Users continue to access data while system performs file backup.

Disk failure recovery: System detects failed drive, continues to run application, and discontinues mirroring until drive is repaired.

Resume mirroring: System allows for the removal of the failed drive from pair, the mounting of another drive in its place while the system is running, then copies data to the new drive, and resumes disk mirroring.

Data consistency: System writes data to both partners of a mirrored pair, so data is always consistent, even during the repair process.

The installation of Mirrored Disk/iX is easy:

• Use the SYSGEN utility to configure the disks into the system.

• Install the disk hardware.

• Boot the system with the new configuration.

• Use the AUTOINST utility to install the mirrored disk software.

• Use the VOLUTIL utility to create a mirrored volume set.

• Move files, if necessary.

• Set up accounts and groups.

The subsystem installation of Mirrored Disk/iX enhances the HP 3000’s VOLUTIL commands. HP provides both commands VOLUTIL and MIRVUTIL to make life easier for the System Manager. The functionality is the same when Mirrored Disk/iX has been installed.

Please be careful: The Create Volumes (CV) capability is required to use VOLUTIL to initialize mirrored volumes. You also need it to input system commands from the system console to perform split-volume backups.

How to set up volume sets

Assuming that the subsystem has been installed and the hardware as been plugged in and is configured, the new volumes will be in state SCRATCH or UNKNOWN. Verify this via :DTSTAT ALL

LDEV-TYPE          STATUS         VOLUME (VOLUME SET - GEN)
30-079370          SCRATCH
31-079370          SCRATCH
32-079370          SCRATCH
33-079370          SCRATCH

Now invoke :VOLUTIL or :MIRVUTIL and create the new set’s master disk as mirrored pair:

volutil: NEWMIRRSET PROD_SET:MEMBER1 (30,31)

and verify via DSTAT:

volutil: :DSTAT ALL

LDEV-TYPE          STATUS         VOLUME (VOLUME SET - GEN)
30-079370          MASTER-MD      MEMBER1 (PROD_SET-0)
31-079370          MASTER-MD      MEMBER1 (PROD_SET-0)
32-079370          SCRATCH
33-079370          SCRATCH

Now add volumes in this mirrored set. These volumes must be in state SCRATCH or UNKNOWN.

volutil: NEWMIRRVOL PROD_SET:MEMBER2 (32,33)

and check via DSTAT again:

volutil: :DSTAT ALL

LDEV-TYPE          STATUS         VOLUME (VOLUME SET - GEN)
30-079370          MASTER-MD      MEMBER1 (PROD_SET-0)
31-079370          MASTER-MD      MEMBER1 (PROD_SET-0)
32-079370          MEMBER-MD      MEMBER2 (PROD_SET-0)
33-079370          MEMBER-MD      MEMBER2 (PROD_SET-0)

In VOLUTIL, the command SHOWSET with the MIRROR option (this option has been installed via the added subsystem) will show the state of the mirrored set:

volutil: SHOWSET PROD_SET MIRROR

Volume Name   Vol Status     Mirr Status          Ldev      Mirr Ldev
MEMBER1       MASTER        NORMAL                  30       31
MEMBER1       MASTER        NORMAL                  31       30
MEMBER2       MEMBER        NORMAL                  32       33
MEMBER2       MEMBER        NORMAL                  33       32

There are two types of disk errors: Disk errors after mount (in normal operation) and a disk cannot be mounted (already defective in booting the box)

Disk Error after Mount: If a disk has a problem after the mount, the system will continue to work automatically with only one disk of the affected pair.

Possibilities for Disk Errors are:

Disk reports ERRORS — Disk will immediately become DISABLED and the System will continue to work without a mirroring for the affected volume set without any interruption.

Disk does not answer any longer — The system waits about two minutes for an answer by the disk. During this period, all I/O processes are suspended. If the disk will answer in this interval, the system will continue to work with mirroring for this pair. If the disk will not answer the disk will become DISABLED. The system will continue without mirroring for the affected volume.

Here’s an example: During normal operation LDEV 32 fails. The following message will appear on the Console:

?09:09/12/MIRRORED VOLUME DISABLED ON LDEV#32
?09:09/22/ACKNOWLEDGE MIRRORED VOLUME DISABLED ON LDEV#32 (Y/N)?

:REPLY 22 ,Y

The system will continue to run. The problem may be that this message and the reply will be overlooked on big systems. It’s recommended to have a monitor in place. HP’s OpenView Operations Center (a.k.a. IT/O OpC) may be one option.

A check using DSTAT and VOLUTIL will show the following:

:DSTAT ALL

LDEV-TYPE    STATUS              VOLUME (VOLUME SET - GEN)
30-079370     MASTER-MD          MEMBER1 (PROD_SET-0)
31-079370     MASTER-MD          MEMBER1 (PROD_SET-0)
32-079370     *DISABLED-MD       MEMBER2 (PROD_SET-0)
33-079370     MEMBER-MD          MEMBER2 (PROD_SET-0)

volutil: SHOWSET PROD_SET MIRROR

Volume Name     Vol Status           Mirr Status       Ldev      Mirr Ldev
MEMBER1            MASTER             NORMAL           30           31
MEMBER1            MASTER             NORMAL           31           30
MEMBER2            MEMBER             DISABLED         32           33
MEMBER2            MEMBER             NON-MIRROR       33           32

Having repaired or exchanged the disk, you must continue by issuing the command: volutil: REPLACEMIRRVOL PROD_SET:MEMBER2 32 and re-establish the mirroring. Check it via

volutil: SHOWSET PROD_SET MIRROR

Volume Name Vol Status    Mirr Status              Ldev  Mirr Ldev
MEMBER1         MASTER     NORMAL                  30      31
MEMBER1         MASTER     NORMAL                  31      30
MEMBER2         MEMBER     REPAIR-DEST             32      33
MEMBER2         MEMBER     REPAIR-SRCE             33      32

The repair will happen automatically. No further intervention will be needed. This process is fully transparent for applications and users of the data on the affected volume set.

Disk Error before Mount: Here’s another example: LDEV 33 cannot be mounted during the system’s startup. Following message will appear on the console:

?09:09/12/MIRRORED PARTNER MISSING FOR LDEV# 32

The system sets the “good” mirrored disk LDEV 32 on PENDING and waits for SUSPENDMIRRVOL to allow LDEV 32 to work without the mirrored partner:

?09:09/22/ACKNOWLEDGE MIRRORRED PARTNER MISSING FOR LDEV# 32(Y/N)?

:REPLY 22 ,Y

This volume (here: MEMBER2) will stay unavailable until either the mirrored pair (here: LDEV 33) will become available again, or the mirror will become suspended via SUSPENDMIRRVOL command. DSTAT will show the following:

:DSTAT ALL

LDEV-TYPE STATUS VOLUME (VOLUME SET - GEN)
30-079370 MASTER-MD MEMBER1 (PROD_SET-0)
31-079370 MASTER-MD MEMBER1 (PROD_SET-0)
32-079370 *PENDING-MD MEMBER2 (PROD_SET-0)

By using SUSPENDMIRRVOL the mirror will become suspended, and the disk will work without a mirror for it. Now, this disk should not become faulty — it’s a single point of failure now! SUSPENDMIRRVOL will work only for disks in state PENDING!

volutil: SUSPENDMIRRVOL PROD_SET:MEMBER2 32

MEMBER2 is now available again, but without a mirror:

volutil: SHOWSET PROD_SET MIRROR

Volume Name Vol Status Mirr Status Ldev Mirr Ldev
MEMBER1 MASTER NORMAL 30 31
MEMBER1 MASTER NORMAL 31 30
MEMBER2 MEMBER SUSPEND-MIRR 32 33

Having repaired the disk (LDEV 33), the mirror must become active again:
:DSTAT ALL

LDEV-TYPE STATUS VOLUME (VOLUME SET - GEN)
30-079370 MASTER-MD MEMBER1 (PROD_SET-0)
31-079370 MASTER-MD MEMBER1 (PROD_SET-0)
32-079370 *PENDING-MD MEMBER2 (PROD_SET-0)
33-079370 SCRATCH

By using REPLACEMIRRVOL, the repaired LDEV 33 will be initialized as mirror for LDEV 32 :

volutil: REPLACEMIRRVOL PROD_SET:MEMBER2 33

You can verify this with SHOWSET

volutil: SHOWSET PROD_SET MIRROR

Volume Name Vol Status Mirr Status Ldev Mirr Ldev
MEMBER1 MASTER NORMAL 30 31
MEMBER1 MASTER NORMAL 31 30
MEMBER2 MEMBER REPAIR-SRCE 32 33
MEMBER2 MEMBER REPAIR-DEST 33 32

Split-Volume Backup

Another feature of Mirrored Disk/iX is the way it can make backups. It is only necessary to take (physically spoken) one disk’s content out of a mirrored pair. This is possible via the split-volume backup. To split a volume set, no user is allowed to stay logged on for this volume set: You can use the command :TELL @ LOGOFF FOR BACKUP. The volume set will become unavailable now for one half:

:VSCLOSE PROD_SET; SPLIT
:DSTAT ALL

LDEV-TYPE STATUS VOLUME (VOLUME SET - GEN)
30-079370 LONER-SU MEMBER1 (PROD_SET-0)
31-079370 LONER-SB MEMBER1 (PROD_SET-0)
32-079370 LONER-SU MEMBER2 (PROD_SET-0)
33-079370 LONER-SB MEMBER2 (PROD_SET-0)

Now make via VSOPEN both SETS (USER and BACKUP) available: :VSOPEN PROD_SET

PROD_SET SPLITT USER VOLUME MOUNTED ON LDEV 32 (AVR23)
PROD_SET SPLITT BACKUP VOLUME MOUNTED ON LDEV 33 (AVR24)

Now the PROD_SET is available for production but without the mirroring:

:TELL @ SYSTEM IS AVAILABLE NOW
:DSTAT ALL

LDEV-TYPE STATUS VOLUME (VOLUME SET - GEN)
30-079370 MASTER-SU MEMBER1 (PROD_SET-0)
31-079370 MASTER-SB MEMBER1 (PROD_SET-0)
32-079370 MEMBER-SU MEMBER2 (PROD_SET-0)
33-079370 MEMBER-SB MEMBER2 (PROD_SET-0)

Start the backup with the command: :FILE T;DEV=TAPE :STORE /; *T; SPLITVS=PROD_SET; SHOW. This backup is compatible to the “normal” STORE.

Having finished the backup, the split can be canceled:

volutil: JOINMIRRSET PROD_SET SOURCE=USER

SOURCE=USER tells the system that during the JOIN and the following REPAIR to synchronize the BACKUP with the USER split, so the on-line users are allowed to continue to work. The status in VOLUTIL during the synchronization will look like:

volutil: SHOWSET PROD_SET MIRROR

Volume Name Vol Status Mirr Status Ldev Mirr Ldev
MEMBER1 MASTER REPAIR-SRCE 30 31
MEMBER1 MASTER REPAIR-DEST 31 30
MEMBER2 MEMBER REPAIR-SRCE 32 33
MEMBER2 MEMBER REPAIR-DEST 33 32

At most, six mirrored pairs will become synchronized in the same period for performance reasons. That means as soon as one of the six pairs is finished the next waiting pair will be processed.

This is an optional way to make a backup — as I said earlier, Mirrored Disk/iX is fully transparent to all applications. We’re still using TurboStore/iX online to back up our mirrored volume sets, and didn’t encounter any problem because of Mirrored Disk/iX.

Summary

Mirrored Disk/iX is a MUST for mission critical systems to guarantee high availability of the data. In case of problems because of the physical disks, the data stays available. The automatic repair processes in Mirrored Disk/iX are transparent to the users. The procedures are quite easy — but you must know them! This article will help all HP 3000 System Managers to have this handy. Comments and questions are welcomed; just send me e-mail at aschmid4@csc.com.

Andreas Schmidt is a computer technology specialist working for CSC Ploenzke AG, Germany.