CSI Sponsor Message

It's All in Your Head

By Scott Hirsh

Imagine the scene: a group arranged in a circle, the flames of the campfire reflecting off the faces of those assembled to hear the stories told in hushed, reverent tones. Hard lessons passed on to those who will someday repeat the ritual, enhanced by their own experiences.

What could this be? Story time on a Scout camp trip? A heart-to heart with Cookie at the ol’ chuck wagon? Perhaps just another group of aspiring millionaires killing time on a deserted island.

Maybe a few years ago I would have guessed one of the above, but those groups have all moved on, adopting new technology to better communicate their respective collective knowledge. No, the scene I describe is the training of System Managers and their support staff at your average HP 3000 shop.

Ok, so maybe I’m exaggerating … a little. But let’s be honest here: The documentation and communication of policies and procedures in the typical HP 3000 shop is mired in the 1970s: yellowing and dog-eared paper supplementing the sacred oral tradition of system management knowledge transfer.

Today we have inexpensive (or free) word processors that can create and maintain hyperlinked electronic documents that can be accessed via an intranet or stored on palmtop computers. We have HTML, PDF and other paperless formats. We have databases that can mix and match text to provide customized views of information. So how do we bring new System Managers up to speed? “Sit down and let me tell you about our startup and shutdown procedures…”

There is a paradox today in the HP 3000 community: While the installed base is static or declining slightly, there is a shortage of qualified, experienced System Managers. So the need for people-less knowledge transfer and easy-to-run operations has never been greater.

For those who can barely find the time to read this column, perhaps these worst documentation and knowledge transfer practices will strike a chord.

Just because you’re paranoid

With paternalistic employers all but extinct, some paranoia is now probably justified. But even in the days when lifers hoarded information “just in case,” keeping critical system management information upstairs was misguided. These days, people resist documenting critical system information for some valid reasons, such as general lack of time, need to multi-task, increased responsibility over other platforms (because the HP 3000 is so stable), or the old standby, “the HP 3000 will be going away in six months” (usually uttered two years ago, and still untrue). Regardless, even the indispensable get the heave-ho in today’s “new economy,” so you might as well open the vault.

A chain is only as strong … When we were forced to use a PC to manage DTCs (remember those?), there was an outcry over making the reliable HP 3000 platform dependent on the notoriously unreliable Windows platform. Many old timers are still apprehensive about storing system documentation anywhere but right on the good old HP 3000. Sure, you could fire up Apache and start serving HTML pages from old reliable, but who’s doing that? (Feel free to tell me if you are, as the CSY Jazz Web site is, but I’ll bet the percentage is minuscule.)

There are many legitimate reasons to store your documentation elsewhere. So how is it that we follow the startup procedures that are stored on our downed system? Oh, we printed those out? How 1960s! (And don’t call me to complain about your electronic documentation you can’t access due to rolling blackouts.)

Window dressing: Some shops have an unbelievable amount of documentation. Run books, policy and procedures manuals, As-Built documents, User Guides — you name it. Alas, these incredible walls of binders often serve to overwhelm, are unlikely to stay current — who has time for that? — and, believe it or not, are really just for show. Trot them out for the auditors, but when the chips are down it’s “page Charlie” and not “get me that binder!”

Once burned: Does this sound familiar: “We once tried documenting our operation, but it turned out to be a waste of time.” Especially in shops with long-term staff, the need to document is not an obvious priority. And if management once got a wild hair concerning documentation (hmm, who will know how everything works when Mary’s out on maternity leave?) only to lose interest when the effort took longer and cost more than anticipated… that makes for a reliable ongoing excuse for avoiding putting digit to keyboard.

Isn’t it obvious? Who needs to document? Our shop is so easy to run we don’t need to document anything. (Maybe in the Jetsons’ datacenter …)

There must be some way out of here: There are two tests of your documentation and knowledge transfer: First, how do things run without you, the shop guru? Let’s say someday you actually get a day off, and then for whatever reason you don’t go back to work for two weeks (I think that’s called a vacation).

Furthermore, let’s say during that time you can’t be reached via e-mail, telephone, pager or astral projection. And to make things even more interesting, let’s say this all occurs with no notice (I believe this is known as an industrial accident). What happens to your shop then? (If you answered, “we wouldn’t miss a beat,” you’re either in denial or have participated in too many election year polls.)

Second, how quickly can you augment or replace staff? If it takes months instead of weeks or days to make a new staff person functional, you have some work to do.

In the spirit of system management camaraderie, I offer some food for thought in the documentation and knowledge department.

1. The Best Documentation is No Documentation

Automation, with in-line comments, is a major step in the right direction. Rather than using UDCs and command files for everything — requiring a person to type the commands — use jobs to perform the tasks. And the more you can execute the jobs automatically, the better. Added bonus: the $STDLIST provides an audit trail.

Why is it the same processes that just start up on their own on a Unix system (e.g., FTP background process) are usually started manually on an HP 3000? What I always hear is “we like to do a controlled startup.” Yes, but every time? Consider using a job that automatically streams out of SYSSTART — under normal conditions — to bring your standard background jobs up. And use jobs in conjunction with scheduling to perform mundane tasks like switching welcome messages, changing limits and fences, logging off users, purging old files. Whenever you put hands on the keyboard, ask “Is there a way I can automate this?”

Remember: You should be designing your operation for the normal, not the exception. Then you document only the exceptions.

2. Be Consistent

Use templates as the basis for critical operations knowledge, whether it be management jobs or UDCs or operations procedures or training materials. The fewer variations on whatever it is you must manage, the easier it will be to maintain and explain.

3. Be Clear

Most of us still live in an 8-character world. But that’s no excuse for cryptic command and file names. As discussed in a previous column, you can use the MPE group name as an extension to the filename; that is, FORECAST.JOB rather than JFORECST. The less obvious the names of critical components, the greater the need for documentation and one-on-one time.

4. Be Generous

Disk space is cheap these days, so you no longer need to worry about saving a few bytes in the comments you put into your jobs, UDCs, command files, procedures, etc. If information is pertinent to the object at hand, put it in there. I can’t believe how many jobs, command files and UDCs I encounter without a single in-line comment. Having associated references to your cryptic jobs in some run book or other is no excuse for leaving your jobs mysterious. People read $STDLISTs when there is a problem. If these $STDLISTs contain useful information, your life will be that much easier.

5. Be Innovative

There are valid arguments for wanting to keep all system documentation on the HP 3000 itself, but they’re pretty threadbare by now. To overlook obvious breakthroughs like browser-based intranet documents is to justify all the worst criticisms of the HP 3000 and its supporters as “legacy.” Yes, you can run Apache on your e3000 and serve up documentation that way.

There is an overlooked HP 3000-based option: the MPE help system. Personally, I like MPE-style help and encourage others to create their own help catalogues. Being able to type something like HELP MYBACKUP at the colon prompt can be very helpful. Likewise, the third-party help subsystems from Robelle and others are viable adjuncts to MPE’s.

6. Be Open

How do you know if you’re doing a good job with knowledge transfer and documentation? Ask those who are on the receiving end. What don’t you understand and how can I communicate critical information better? Make yourself scarce and see how things run.

7. Be Realistic

Don’t expect to end world hunger all by yourself. Start small, with specific problem areas and work from there. If you set unrealistic goals you will be back to, “We tried ending world hunger and it turned out to be a waste of time.”

8. Be Persistent

Creating a guru-independent and easy-to-learn operation is a continuous process and there is always room for improvement. Your metrics will allow you to gauge progress. As you see progress you (hopefully) will be encouraged to keep plugging away.

9. Be Optimistic

If you act as your own best friend — yes, I want to get a full night’s sleep; yes, I want to take a vacation; yes, I want to reduce the learning curve for new staff — nobody will conclude that you are no longer needed. Worst case, anyone who creates a well-run shop will be in demand. So don’t fear sharing your secrets of how things run. Concentrate on the positive aspects of not being interrupted during your much-deserved personal life.

Scott Hirsh (scott@acellc.com) former chairman of the SIG-SYSMAN Special Interest Group, is a partner at Precision Systems Group, an authorized HP Channel Partner which consults on HP OpenView, Maestro, Sys*Admiral and other general HP 3000 and HP 9000 automation and administration practices.