Patch/iX: Maintenance Simplified

By Guy Smith





If I want to incite fear in the system administration team here at The Dungeon, I simply tell them that someone has to install a patch on one of our MPE systems, then begin with the eny-meany-miney-moe routine. As I watch them scramble, I find it hard to believe those donut hounds can move that fast.

HP has performed miracles in the design of MPE/iX, constantly enhancing the system while maintaining strict backwards compatibility. But that laboratory marvel didn't improve the real world need for applying improvements or correcting the occasional bug. The uninitiated might even believe that each programming group within HP designed their own patch installation routine, stopping just short of documenting it for PICS. This might be an optimistic analysis.

Though there has been a consolidation of late, with AUTOINST replacing many segregated tools, there is still no single coherent system software for tracking or reconciling patches. This is not only inconvenient for the customer, but it has likely cost HP many engineer hours in manually cross-referencing patches, versions, and hand-holding users through the use of the latest tool.

This contrasts with many other operating systems, most notably HP's own Unix variant, HP-UX. The jaded among us contend that Unix systems need patching more often and thus engineers have had more opportunity for refining the process. Cynicism aside, Unix has several distinct advantages in managing code replacement:

Directories: The hierarchical Unix directory structure, and common routines for copying complete directories, simplify saving existing code, overlaying old code and recovering from both software errors and defective patches.

File overlays: Most Unix systems, with some exceptions, can overwrite program files that are currently in use. This makes it possible to overlay critical pieces of code without interrupting services until more convenient times (after the next system crash or even a planned outage). Many MPE components in program files cannot be overwritten while loaded. This is true of most of MPE's networking software. Thus even the preliminary step of patching MPE, loading files into target locations, may entail knocking users off the system.

Kernel overlays: Patching the MPE kernel or the equally important system shared libraries (XL and NL.PUB.SYS) requires the patches to be made to a copy of the object and then for the object to be unloaded to tape, then copied back in during an outage. Unix systems are able to patch the primary kernel, keeping the new version on disc, which meansŠ

Multiple boot environments: Users can boot from any number of resident kernels, which makes backing out from a bad patch as simple as booting the system (this does not apply to patches that replace files in hierarchical directories, though some Unix variants have streamlined this process as well through directory cloning). For MPE/iX there is an added half-hour tape operation (more if the older OS tapes are stored off site).

HP is staging a new tool which it contends will address the three issues of system up-take, patch management and uniform installation. Patch/iX will be released in stages with release 5.5 (no release date for 5.5 has been announced, but public speculation is that it will appear in the first quarter of 1996). Not surprisingly, many of the capabilities of Patch/iX were made possible by the introduction of POSIX features. Patch/iX is divided among several tasks, outlined below.

On-line patch retrieval
HP is putting MPE patches on-line, eliminating for most of us the delays, tedium and expense associated with shipping tapes. Patches will be available via dial-up (HP Support Link) and across the Internet in either e-mail packages or by direct download from a World Wide Web (WWW) server. Due to the limitations of many e-mail programs, patches will be broken up into uuencoded pieces of less than 256 kilobytes.

To simplify patch reconstruction (gluing several e-mail messages, uudecode-ing them) HP will provide three utilities. PATCHMKR will combine the individual messages or downloads into a single file, a truck file for those familiar with the older UHAUL and RYDER utilities. UNPACKP will extract files from the truck file. A final utility, PREPATCH, is actually a script (shades of AUTOPAT!) that will execute both PATCHMKR and UNPACKP to simplify the process.

The result of this confusion will be at least two files. One (REFxxxxx) contains ASCII text describing the patch, while the other (GENxxxxx) contains binary information on the patch, patch dependencies, successions and checksums. Both will be used in the next step of the patching process.

Knowing your patch
Patch/iX is an on-line tool that allows you to review the specifics about a patch, and which will validate that a patch is appropriate for your system. Some of the information that will be available to you includes:

A description of the patch and the symptoms it addresses
Known conflicts the patch may have
Dependent patches (other patches you need for this one to work)
Environmental changes (new prompts, messages, etc.)
Hardware, software and application dependencies
Notification of special installation steps that may be required
A list of replaced files

Qualifying a patch
Qualifying is the act of assuring that it would be appropriate for a specific patch to be installed on your system, a stunt I've forced many PICS engineers to perform manually. AUTOINST and HPINSTALL could qualify patches on a system, but they only worked with Power Patch tapes. Patch/iX extends this to individual patches.

Patch/iX takes a number of steps to determine if your system should have the patch installed. It validates the patch dependency list to see if dependent patches have been installed. It also checks to see if the patch you want to install or one of its successors has already been installed, thus saving you some time if receive cross dependent patches. Patch/iX will only try to qualify the latest version of a patch.

Patch/iX provides a meaningful report on why a patch might be disqualified, which will quicken the patch cycle and reduce my stress level. Before continuing, the system administrator has the option of staging the patch even if it does not qualify. This backdoor is provide for those emergency situations where a patch needs to be installed despite potential side effects of missing dependent patches or other dangers.

Staging a patch
Backing out a patch can rival the agony of abdominal surgeryŠ without the benefit of anesthesia. Often the last system backup must be retrieved, and files in the SYS account restored all but blindly to back-out program files overlaid by a patch. If there was a patch to NL.PUB.SYS, or CL.PUB.SYS, then an update must be performed as well.

Well, welcome to the 21st century. MPE will go one better than most Unix systems with StageMan/iX. This tool places the files in a patch into a special staging area in the POSIX directory space. You can have multiple staging areas under the root staging directory /SYS/hpstage (for example /SYS/hpstage/new_patch could be created for a newly received patch - and as with all things POSIX, each staging area name is case sensitive, so NEW_PATCH and new_patch are different patch sets). These directories will hold the files (presumably including patched versions of NL and XL) that comprise the patch or patches you have installed.

To implement a staged patch, you run one more utility called STAGEMAN. This utility (and a pared down version that runs from ISL) identify what patch area should be used during the next reboot. At boot time the files in the stage area are renamed to effectively replace their targets (i.e., /SYS/hpstage/new_patch/DSDAD.NET.SYS is renamed to DSDAD.NET.SYS). The original files are preserved into a special directory of the staging area (/SYS/hpstage/base_archive). This allows a patch to be backed out with a mere reboot, except for rare circumstance when an SLT operation may be needed.

On this note, I plan on date encoding the patch staging area names (i.e., SYS/hpstage/95_09_10) to eliminate confusion on what staging areas I should roll back to, or in what order patches were added. STAGEMAN provides a COMMIT operation that will purge the base_archive and the target patch directory. You will want to perform this operation periodically since the patch directories will consume disc space on your system volume set.

This staging methodology eliminates one pet peeve of mine, the need to bring down subsystems to prepare a patch. Many files in the group NET.SYS comprise the networking subsystem. These files are loaded and cannot be overwritten while the network is up. This means I must come in on my days off, or stay up late on work nights just to stage a patch (i.e., execute an AUTOPAT job which would abort while replacing the files in NET.SYS). With STAGEMAN I can execute the predatory phase while at the office, stereo blasting and hot coffee in one hand, and leave simple reboot instructions for the night shift operators.

HP's take on all this
I swapped e-mail with Patrick Goddi, the lead for the Patch/iX project (soon to achieve sainthood in the Church of the Frustrated SysAdmin). I was surprised to learn that HP has been quietly working on Patch/iX for over a year. Many roadblocks needed to be cleared before the product could be introduced. Most of the design for Patch/iX came from end user suggestions. Because of the user-driven nature of the project, Goddi did not have estimates on how this will effect HPs support organization, or how much money it might save HP in chasing down patch requests.

"The Patch Solution in CSY was created strictly to address customer issues with the patching process, not as a cost savings measure," he said. "Of course we took input from the Hewlett-Packard support community since they have extensive contact with customers, but the main drivers in defining solution components and functionality have been our customer partners."

I hope Patch/iX does save HP some time and money, and that the savings will be reflected in my next software support bill.