Roc Software Sponsor Message

Backing Up is Still Hard to Do

By Scott Hirsh

There are some system management functions that are so fundamental to the well being of your HP 3000 that it’s hard to believe that we system managers haven’t nailed it by now, weeks before Y2K. And yet, even to this day, many shops still have difficulty ensuring consistent reliable backups. (I’ve always maintained that one of the holiest laws of system management is “Thou shall not lose data.”) About 10 years ago I wrote a monthly HP 3000 system management column, one installment of which was “Backing Up Is Hard to Do,” my (marginally) humorous look at system backup. Because I have seen much backup difficulty in my travels, an updated look at this critical task is in order.

Backup Then

Ten years ago, and in most HP 3000 shops today, system backup was host-based. I haven’t taken a survey, but my guess is that most host-based backups today, as then, are performed using good old STORE, part of FOS. True, there were hints of client-server backup to come, with third party products that would backup PCs to your HP 3000, but they didn’t catch on. Then as PC networking (mostly NetWare) became popular, PC network-based backup products eliminated the business case for backing up to the HP 3000. Back then, Unix systems still hadn’t become the ubiquitous presence they are today in most IT shops, and the thought of making a Unix system the “master” in a client-server backup architecture was unthinkable. Perhaps it still is today to many HP 3000 system managers.

Backup Now

Today, there are few pure HP 3000 shops. Everyone has at least PCs, and the rise of the Internet almost guarantees a mix of operating system platforms. Increasingly, systems (or servers) are special purpose, or part of a distributed, load-balanced environment. No matter how you approach it, backup is heterogeneous.

While it isn’t required, the larger shops I have visited personally tend to segregate the administration of their HP 3000s. Smaller shops often can’t afford many specialized admins, so HP 3000 system managers in small environments usually administer the other platforms as well. The degree to which the staff of a shop has cross-platform responsibilities, determines the method by which backup is performed.

Appropriately, with the heterogeneous data center comes lots of backup options; certainly many more than we had in the mid- to late 1980s. Now, even if we want to back up our HP 3000s separately from our other systems, there are third-party products like Backup/iX from ORBiT that meet or exceed the capabilities of HP’s STORE/TURBOSTORE.

But even more interesting to me are the products — Legato Networker, HI-COMP’s Hi-Back/iX and ROC Software’s ADSM client — that incorporate your HP 3000s as clients into an enterprise backup framework. The “HP 3000 as backup client” has some definite advantages. For example, media management and general administration are incorporated into your overall (typically, Unix) backup, offering a centrally administered model provides economies of scale and should prevent backup misfires in many shops.

However, there are also disadvantages, among them cost, complexity and possible performance problems tied to network bandwidth requirements. Still, the client-server backup model has taken the data center by storm, and no HP 3000 shop can overlook this option.

What Can Go Wrong?

Still, with all the options available today, the typical HP 3000 shop is still performing host-based backups using STORE, TURBOSTORE or one of the popular third party host-based products. Host-based backup should be easy: create a job that performs the appropriate backup — full, incremental, differential, archive, SLT — and make sure it runs every night or every weekday, whichever works for your shop. You would think this would be the easiest task in the world to perform, but you would be wrong. See if you recognize any of these backup breakdowns:

Not Knowing the Environment: You can’t develop and implement a backup strategy if you don’t know what your backup requirements are. Is the data dynamic or static? Is speed of backup or restore a priority?

No Media Management: Your basic STORE program has no facility for media management — the tracking of your backup media for usage, retention, vaulting, etc. For a large tape library, manual media management is heck. If the lack of media management is compounded by not using labeled tapes, there’s a good chance someone will eventually use the wrong tape and/or tapes will be lost.

Not Labeling Media: Putting ANSI labels on tapes helps prevent inadvertent use of an unexpired media. Putting numbers and descriptive labels on the outside of the media helps keep you sane.

Not Maintaining Sufficient Media Inventory: Gee, I thought we had some scratch tapes left. Oh well, let’s just scratch that full backup a little early — guaranteeing you’ll need a file from that backup the instant you overwrite it.

Not Verifying Media: In the “good old days” we validated tapes because a bad 19th of 20 tapes would force us to start over again. These days, verifying media takes on a different twist. Now, we don’t have to start over, and our backups generally don’t take that many tapes anyway, with storage media being ever more dense. Instead, our media verification issue has to do with our old backups or archives. Will that 5, 10 or 15 year old backup gathering dust in the vault still be readable if and when we need it? I wouldn’t count on it.

Not Using Online Backup: Holding up application batch processing or user application access until your backup completes is increasingly difficult, with users expecting 24x7 access to everything. Even so, somebody must pay the cost of online backup, and management isn’t always willing to shell out for the online backup option.

Lingering Operator Prompts: A perennial problem if you don’t have a product that reads console messages and notifies you. No fun if your applications wait for successful completion of your backup.

Not Checking Backup Results: Everyone is guilty of this one at least once in their system management careers. But it only takes one experience where a file is needed for recovery but it’s not on the backup to correct the error of your ways. If you want to be sure, use one of the utilities that reads your backup or write your own script/command file to check SYSLIST.

Not Sending Media Offsite: There are so many scenarios where you won’t have access to your data center that make offsite storage is essential. And yet, plenty of otherwise sensible system managers don’t do it. When I started at my last shop, the operator was putting the tapes in the trunk of his car every night after completion of the backup. That is not what is meant by offsite storage. If you’re lucky enough to have your own offsite vault, use it. If not, use one of the standard offsite storage services. And while you’re at it, don’t forget to send a hardcopy tape library listing offsite, too.

Other Backup Issues

These are but a few ways you can take a routine function and make it exciting. (Remember: anything exciting is bad, and that goes double for backup and recovery.) I haven’t discussed SLTs, which also offer opportunities to hose yourself. [Ed. Note: John Burke summarizes SLT advice in this month’s net.digest column]. And I haven’t discussed recovery either (ever forget the OLDDATE option?). The main point here is that none of these issues is new, but we still trip over them periodically because the procedures are often informal, lost in staff turnover. The good news, though, is it’s not hard to do a great job if you pay attention to the details and learn from your mistakes.

Backup Horror Stories Wanted

If you have a particularly juicy backup mishap you want to share [anonymously] with your fellow system managers, please e-mail me. I promise to include it in a future column.

Scott Hirsh, former chairman of the SIG-SYSMAN Special Interest Group, is a partner at Precision Systems Group, an authorized HP Channel Partner which consults on HP OpenView, Maestro, Sys*Admiral and other general HP 3000 and HP 9000 automation and administration practices.