Pursuing an Education About Migration

Richard Gambrell’s HP 3000 transition will be a journey crowded with lessons. The director of computing systems for the University of Tennessee at Chattanooga (UTC) might prefer to skip some of the classes HP has mandated with its departure from the 3000 market. But he and his staff bear their course-load with good humor. That might be because Gambrell has already been through a transition toward HP’s Unix, and knows how educational a migration can be.
UTC operates two K-Class HP systems, one HP 3000 Series 959 and an HP 9000 K570. Gambrell’s experience with Unix goes back into the 1970s, so he’s no neophyte on HP-UX’s commodity computing culture and extensive maintenance needs. In other ways Gambrell’s customer site represents a typical HP 3000 installation: plenty of custom-written code still working on a 3000, running alongside packaged software on a Unix system.

Gambrell’s Unix experience predates his 3000 work. His MPE exposure started in 1983. HP 3000 use at UTC goes back to 1976, and four years later the university deployed one of the first programs anywhere which showed the requirements its students had satisfied and what courses they needed to be take to graduate. But other campuses in the UT system moved to Unix, and UTC went along with the other schools to migrate its finances to the SCT Banner application. UTC stopped its migration in 2000, staying flexible enough to integrate its MPE systems with the parts of Banner it managed to get implemented.

While seasoned IT veterans say there are no good migrations — just some that are better than others — we thought Gambrell’s UTC perspective with both MPE and Unix experience could deliver fresh information about the existing wisdom on migrating from custom code to packaged solutions on Unix. We spoke with the manager who controls a $1 million IT budget within days of the HP’s multi-billion merger vote.

How extensively will UTC be using HP’s solutions in the future? Any new 3000s in your future?
We lease our systems, so we’ve a revenue stream that lets us go on leasing something else. We will probably replace our current 3000 before Oct. 2003, to have as long of a life on the 3000 as we can afford. This gives us the maximum time for the migration, but it also provides the strongest safety net. We are not sure exactly what we will buy. It may be a new N-Class or it may be a 989.

I see our migration path as a long thing, over five to 10 years. I’m enthusiastic about the improvements a rewrite can provide, although I’m apprehensive about the work. I’m not one to say let’s keep using the same stuff, because the same stuff is pretty aged. It’s been patched to death, and there’s design flaws. The good thing about the announcement is that it provides a lever I can use to solve that problem. Before, I couldn’t find the right lever to free up the resources. It wasn’t a big problem before, they were just little problems.

The best way I’ve thought about this is like turning the clock back. Linux is the opportunity we saw in 1975. And the equivalent of the cheap minicomputers we bought then from DEC and HP are the commodity servers today.

Is commodity computing working for your organization? What chance do you see of HP having a viable play in commodity computing for your enterprise against IBM, Dell and Linux?
HP has got it right in terms of its forecast about commodity computing. How to get there is the real challenge. I don’t believe they know how to do it, and it’s one of the reasons they want to buy Compaq. But Compaq doesn’t know how to do it, either. HP probably knows how to do enterprise marketing better than Compaq, though they’re not very good at that either, compared to IBM.

It’s definite that commodity servers are here. In the modest-sized computing environments I’m familiar with, it’s all about commodity servers. But I don’t care about a 1 percent better performance for a price premium. Dell is our hardware vendor of choice. Our internal goal is to move to Linux or Netware, both on Intel servers, for our standard set of enterprise computing services — that’s e-mail, file sharing, Web — except for using MPE for our custom database applications and Windows 2000 where necessary for a specific application.

I see no reason to buy HP-UX, Solaris, or IBM’s AIX except if required for a specific application and we have to include the cost of these in the decision to choose an application.

We already use Dell servers like Legos to build servers and buy servers as we need them. We don’t worry about hardware support at all, except on unique equipment that we try hard not to use. We will maintain a standby server that we can slap the right operating system on from a CD then restore the data. Or we will maintain a “cold” standby server fully configured and ready to replace a dozen other systems in a few minutes. On these, we will automate copying data over to it regularly to maintain readiness to take over. For example, we could log database changes to it to be applied to the last full database copy that was made last night.

Inexpensive servers that are powerful and reliable change the way we run the datacenter. For us, a half-hour down time isn’t terrible and that allows us great flexibility to take advantage of commodity servers.

Do you believe a commodity aspect can be introduced to MPE solutions, provided the hardware can become a commodity item?
That’s been my vision of the future for MPE: to hitch the MPE wagon to the Linux parade. One of the killers for supporting operating systems is device drivers, the latest hardware. A lot of people are unsettled about going to Unix for the same reasons the MPE users are unsettled: MPE offers complete solutions.

Instead of tying MPE to IA-64 in the traditional fashion, CSY should have ported it to Linux, in effect. Provide an emulation environment so you can bring your applications over easily, and a business-oriented shell and file system and database — the same things MPE provides, maybe with a little more change than MPE customers are used to.

Would you use an emulator for your MPE applications?
If sufficient performance and reliability can be obtained.

So will you be migrating your 3000 enterprise server’s in-house code to another platform?
Before November of 2001, we were actively pursuing a five-year plan to consolidate on preferred system solutions using Linux and Netware, with MPE for custom programming. Windows 2000 would be used only where necessary for the solution.

After HP’s announcement, for our largest and most important applications we are researching, scoping, and budgeting two alternative projects, one a major rewrite and the other moving to a commercial package with customization. We will present these project alternatives to upper management for a decision before the end of the year. A rewrite would likely be a rewrite “in place” on the 3000, but with a portable approach to the programming.

Eventually we must also plan alternatives for a half-dozen smaller applications, or leave them on MPE, if OpenMPE is a success. Some of these may be moved to commercial systems easily, but the more custom ones can stay.

HP believes there are applications on other platforms which its 3000 customers can use to replace custom apps. What’s been your experience with that strategy?
For our largest application, we were unable to move to a commercial package a couple of years ago, because of its limitations and its quality problems, without extensive customization. It didn’t have the features that our existing custom code had. We knew there were some unique policies we had which wouldn’t be covered. But there were larger chunks that couldn’t be covered. There were also software quality issues and performance issues.

It’s not easy to move to packaged applications, partly because it’s a real culture change, for the IT staff as well as the users you’re supporting. In many cases, the commercial applications expect a great deal more out of the end users. System-wide, the university has migrated its accounting to SAP. To enter a requisition people now have to go to about six different screens, and ignore about half the fields on the screens. It’s actually quicker to sit at a typewriter and type in the requisition on a paper form than to enter the stuff in the computer. Not to mention the startup costs. That curve is pretty high, but it does go away.

This changes the way you do things. Right now we’re going through a big period of inefficiencies, and everything takes longer. Fixing any mistake takes longer, and [outside vendors] don’t get paid on time anymore. A little pain for a little while is okay. But we’ve been running the SAP financial stuff for a year.

UTC has a history of doing its own enhancements on its MPE applications. What was it like becoming a user of packaged software?
Our vendor has thousands of enhancement requests. In many cases they don’t get fulfilled. They act on tens of them. The backlog just grows and grows. Surely some of those are not worth doing. But I can guarantee a big bunch of them would have beneficial effects on existing customers.

We’ve been a customer for four years; people who’ve been customers for 10 years complain the same way we do. This seems to me kind of typical. They’re not really interested in existing customers. They’re interested in new customers, and adding new products that we have to buy.

The other campuses picked this product first. We joined the club. Gartner Group said that in our metropolitan university, 10,000-student niche, there’s nobody else out there offering a solution.

How much support do you want from HP for OpenMPE’s efforts?
One hundred percent. I’d really like to see HP provide some funding for OpenMPE Inc., to be able to say they’ll be able to support its efforts. I don’t know how to twist it so HP benefits — but that’s the same problem HP has with all of its Linux solutions, too.

But it’s like a paradox. I can understand the whole decision why HP canceled the 3000 was the 3000 was no longer a good bet. So how can HP support OpenMPE without talking out of both sides of its mouth?
HP should come out with the terms and conditions and pricing of licensing MPE in the future, however they want to do it. They should have done that November 14, and let the third parties have at it. I’m a market-driven person. I would have said, let the market decide. If the third parties have to merge together, somebody will be clever enough.
Instead it comes across like they’re trying to play Big Brother. I just don’t think the CSY folks see it. I understand that — because if they did see it, they wouldn’t have cancelled the 3000.

How would your organization like OpenMPE to calculate support payments?
You might have a low-cost entry model, something kind of flat, a per-machine cost. I’d look at what some of the Open Source people are doing. We’re willing to pay Red Hat a couple thousand dollars a year for a better version of Linux they’d tested better. I’d pay a few thousand a year for something that makes it easier to do better business. Those are the kinds of dollars that are going to be available.

We pay HP about $5,000 a month now to support two boxes. This is a whole different cost structure, but I don’t really think we’re getting the support we’re paying for, and haven’t for years. We’re paying for an elaborate level of support that’s barely needed.

Can a Linux-style model of support, where everyone is responsible for contracting their own gurus, work for the 3000 environment you’re maintaining?
Sure. We have to work a little to have contracts and have resources identified that can help us if we need it. It’s a matter of learning the territory and keeping in touch. I expect we’ll spend some dollars here and there on time for expertise to help us when we need it, and maybe a few dollars on Red Hat’s Enterprise Linux. We’ll have lots of budget dollars available by not paying the traditional hardware and software support costs.

For computing environments I’m familiar with, Linux is already better than HP-UX and Solaris. My Unix administrators can’t wait to migrate to Linux.

HP talks up its patches as a reason to stick with HP support. What’s been your experience with patched systems in other HP environments?
In the MPE world you can live without them very nicely. In the HP-UX world, if you’re clever in not going to the latest thing, you can patch a system and be in pretty good shape. I’ll take MPE patches over HP-UX patches times 100.

HP has a very long way to go to get good quality HP-UX patches. It’s one of the most disappointing things. The other is fancy hardware that causes trouble and causes trouble, and lots of patches never fix it.

What service does a user group have to perform to be useful for all of the 3000 customers of 2002 and 2003?
Provide forums for learning migration technical topics as well as strong arm-twisting of HP to support OpenMPE. By working in clear support of a future for MPE, some form of user group will be needed in the future, but it will likely be scaled down some.

What about Oracle as a substitute for IMAGE? What’s been your experience with Oracle on HP-UX?
Oracle and Unix can be reliable and powerful, if configured well and with applications designed well by experts. The complexity and deep skills needed to do this well are much more demanding than in the 3000 environment. The 3000 mostly assumes it knows what you want and in Unix and Oracle you have to pick and choose the “right choices” to get the “right results.” Thus it is easier to go wrong with Unix and Oracle, but it is possible to have very good results, too.
It definitely takes more processor and disk IO power for Oracle to match IMAGE performance, but these days for moderate to small size databases that hardware power is available.

Is the PostgreSQL database going to play a part in your application future? Are you after independent computing, where no vendor can force this kind of transition on you again?
If we rewrite, we definitely are after “vendor free” applications, but we may have to compromise some on tools and utilities or libraries of code to obtain the productivity we need. I’d like our new code to be largely aimed on Posix, but able to run well in MPE or Linux.

We would like to look at PostgreSQL closely. However, the University of Tennessee has a excellent contract with Oracle, so we may use that instead. However, we expect initially to use the SQL part of IMAGE/SQL on MPE, then migrate if and when we need to as a platform decision not driven by the code.

It doesn’t really matter much, as long as you design carefully for “generic” SQL and stay away from unique features of each database.

What do you want HP to do with the resources that were CSY?
I wanted CSY to be sold, but reorganized into a lean and mean company. That’s not going to happen, so I’ll push for as much support HP can find to move as much as possible to OpenMPE. Staff is the hardest. The entire enterprise will have to be scaled down to be the size of the market.

What’s your experience with the supportability of MPE versus your Unix systems?
I could write a book on Unix war stories that would frighten all MPE users. But the bottom line is that Unix has become good enough and pretty darn good at that, particularly Linux. The trick is knowing what you’re doing, as with anything. But where Unix and RDBMS go, there is a bunch more to know than with MPE — and these technologies will let you take a deep fall, whereas MPE usually won’t let you go too far wrong.

Would you rather pay a fee for a port of MPE to IA-64 than a paying for a rewrite of your applications?
Well, I’d love to have a MPE on IA-64 that I could afford, but our applications will really benefit from a rewrite, too.
I think the best thing would have been for CSY to have taken Linux for IA-64 and developed a part emulator, part integrated module for MPE applications to be ported to. This would have gotten them largely out of the device driver business and they could instead concentrate on a superior business shell and database for Linux, with emulation support for backward compatibility and to provide a transition for MPE users.
Even now, I think this is the best strategy for success. Try to gain new converts from the mass of Linux users. There’s going to be a lot of Digital VMS and IBM AS/400 users to recruit one day, too. Now is the time to create a “business friendly” application environment with a powerful and simple-to-use database, one that runs on the Linux kernel.

The 3000 division has a reputation as being the most customer-focused in HP. What should that reputation lead CSY to do about supporting its customers through the Transition, whether they migrate or Homestead?
It should have provided an out-of-the-box, canned HP solution for continuing to run MPE applications. I know they wanted to announce the death of the 3000 to the world, and I respect them for that, but I think they should have committed to offering a simple, mostly straight path to something, even HP-UX, if that was the HP choice.

They didn’t do that, and they still seem stuck in a “we’ll study the choices and we will make the decision” mode that comes across as almost arrogant. If they don’t have a direction to send customers that is a real solution — unlike their “free” conversion to HP-UX if you have a new N or A box — then they should just name the stakes and the limits, and the costs and the conditions, and just get the heck out of the way.