Robelle Tech: Porting Qedit from MPE to HP-UX

Acucorp Sponsor Message

May 2002

Boosting your e3000 productivity

Porting Qedit from MPE to HP-UX

By Bob Green

Our Qedit full-screen text editor was first written in 1977 and has been continually enhanced since then. About ten years ago we ported Qedit (and then Suprtool) from MPE to HP-UX. We thought those of you who are now migrating your own code would be interested in the technical details of how we converted 100,000 lines of code that had been written over a period of 15 years.

From first inception to delivering a production product took about 20 months. The work effort is estimated as 19 person-months. We could never have achieved this short porting time if we had rewritten all of our source code into a new language as well as for a new operating system.

Most of our development, including Qedit, is done in SPL and SPLash! SPL was the original Systems Programming Language for the HP 3000. The SPL compiler only produces object-code for Classic HP 3000s (you can also run this code in compatibility-mode on MPE/iX machines). SPLash! is a compiler from Allegro Consultants Inc. (www.allegro.com) that translates SPL source code into HPPA RISC object-code.

Subroutine libraries

Because SPL does not rely on a language-specific run-time library, we thought it might be possible to port the SPLash! object-code to HP-UX running on the HPPA RISC architecture. The only thing left would be to replace all of the subroutine libraries on which Qedit depends. Each set of libraries depends on other sets of libraries. The libraries look something like this: Qedit, Qedit Library, Calculator, Qhelp, and Other Common Libraries, Low-level Robelle Library, and finally the MPE and Compiler Libraries. Libraries at the upper levels can call libraries at any lower level. For example, Qedit makes many calls directly to the MPE library (e.g., readx, print, fopen, fread, ...). Similarly, the Qedit calculator makes calls to both the Robelle low-level library and the MPE library.

Object-code format

We had a theory that object-code files were the same on MPE/iX and HP-UX, but we needed to verify that. To show this was the case we compiled a small SPLash! program on MPE, copied the object-code to HP-UX, linked the object-code, and ran the resulting program. Our first test program just called the HP-UX “puts” routine to print a string.

We compiled the program using SPLash! and copied the resulting object-code to HP-UX with these commands:

:splash testsrc,testobj

:dscopy testobj to /users/david/test.o:hpuxdev[user:pass]

The HP-UX command for linking is called ld. To link our example program, we used the command $ ld /lib/crt0.o test.o /lib/libc.a

This linked the three files /lib/crt0.o, test.o, and /lib/libc.a. The first file is /lib/crt zero dot oh (don’t mix up the zero and the oh), which is the C run-time startup library. The test.o file is the SPLash! object-code generated on MPE. The /lib/libc.a file is one of the HP-UX run-time libraries. Like most HP-UX command names and commands, everything must be in lower case

By default, the ld command creates a program file called a.out. To run this program, we only had to type its name (like using implied run on MPE/iX). We tried running the resulting program and got:

$ a.out {run our sample program}

Hello, World! {result of running a.out}

This was exactly what we expected. The buffer that we initialized with “Hello, World!” was printed out on stdlist and the program stopped. We had now answered one of our most basic questions — we could compile programs on MPE, copy the object-code to HP-UX, link the object-code into a program and run it on HP-UX.

Floating-Point Numbers

In MPE, we use the compiler-library routines hpinext and hpextin to convert floating-point numbers from human-readable form to the internal IEEE format and vice versa. These routines have many options and handle different sizes of floating-point numbers. Rewriting these routines would be a lot of work.

Qedit does not use floating-point numbers very often, except in the calculator. In order to implement the calculator, we would need the functionality of the hpinext and hpextin routines. A search of the man pages revealed no documentation for either routine, although there were other routines to handle floating-point conversions. All of our code assumes that hpinext and hpextin are available. So we searched for these routines using the nm command:

$ cd /usr/lib

$ nm -r * | grep hpinext | more

The result of this search was as follows:

libcl.a:hpinext | 13836|extern|entry |$CODE$

libcl.sl:hpinext | 421568|extern|code |$CODE$

libcl.sl:hpinext | 421524|extern|entry |

libf.a:hpinext | 13836|extern|entry |$CODE$

libf.sl:hpinext | 421568|extern|code |$CODE$

libf.sl:hpinext | 421524|extern|entry |

Success! We had found the hpinext routine. We also found the hpextin routine in the same libraries (using the same technique). These routines were located in either the libcl.a or libf.a library.

This was one example of how familiar HP-UX is to MPE programmers. It is highly unlikely that any other version of Unix has the hpinext or the hpextin routines. Even if they did, they are unlikely to have the same parameters and work the same way as the HP version.

Implementing MPE Routines on HP-UX

To get ourselves started, we wrote HP-UX replacements for a number of MPE routines: binary, dbinary, ascii, dascii, print, readx, ccode, and hpsetccode. With this small set, we could start porting our most fundamental library source modules. Several modules started working on HP-UX. However, we would often try to port a new Robelle library only to find that we needed to implement additional MPE routines. So the process went back-and-forth between porting Robelle source modules and writing new MPE replacement routines. In total, we implemented about twenty MPE routines.

MPE File System Routines

Things went well until we had to port our first source module that needed to access files. Using sample code written by Stan Sieler of Allegro Consultants, Inc. we investigated how difficult it would be to implement the MPE fopen, fread, fwrite, and fclose routines.

The HP-UX file system is simple compared to the MPE file system. Files are collections of bytes. There is no implied structure to any file (although you may deduce the structure by examining the filename extension or by looking at some of the data in the file). In MPE, we have fixed-length, variable-length, ascii versus binary, record-length, blocking-factor, and many other file attributes.

Our software depends on a number of assertions that are provided by MPE’s file system routines. For example, if you open a new file with a filecode of 111 and a record size of 256 words, write some records to it, close it, open the file again, then ask about its structure, you expect to get a filecode of 111 and a record size of 256 words. In HP-UX, there is no place to store this information (without using an auxiliary file).

It was at this point that we made a pivotal decision about how we would port our software. We would not emulate the MPE file system routines. Instead, we would reengineer our products to invoke an interface layer that would provide file system functionality. We would isolate the interface layer for each module that needed file system access into a separate source file (in object-oriented terminology this is called encapsulation).

Recognizing Qedit Files

On MPE, Qedit files are easily recognized as those with a filecode of 111 and a specific record length. But HP-UX doesn’t have filecodes or record lengths. We designed a routine to examine the first 1,024 bytes of an HP-UX file. By looking at various relationships in this data, we hoped to determine if the file was a Qedit file or not.

To test effectiveness of our routine, we used dscopy to copy one Qedit file from MPE to HP-UX. We wrote a test program that opened every file on our HP-UX workstation and checked to see if it was a Qedit file. Our first implementations found many HP-UX files that the routine thought were Qedit files when they were not. But our final implementation found the one and only Qedit file that we had copied from MPE.

The 32-Bit Alignment Problem

The HP-UX C compiler prefers to have 32-bit variables aligned on 32-bit boundaries. In SPL and SPLash!, it is common to have 32-bit variables aligned on 16-bit boundaries. The interface layer for Qedit was much more complicated than any of the other interfaces that we had written. Because the interface was more sophisticated, large data structures had to be passed between SPLash! and C routines. We had to carefully implement these structures so that all 32-bit variables were aligned on 32-bit boundaries. We also had to insure that these structures themselves started on a 32-bit boundary.

You can force SPLash! to allocate arrays on 32-bit boundaries by making the arrays virtual. This often has unexpected side-effects (e.g., you cannot pass a virtual integer array to an option splash procedure that is only expecting an integer array). In Qedit, we ended up doing the following. We would allocate one more word of storage than was necessary and then would have code

that looked like this:

if @ext’record mod 2 <> 0 then

@ext’record := @ext’record(1); ! assumes integer array

This code fragment adjusts the start of the ext’record structure if it does not start on a 32-bit boundary

“New” Files

The Qedit Redo module requires a “new” file which, on MPE, you cannot see with Listf or Listftemp. HP-UX does not have new files, but there is a solution. For the invisible scratch files needed by a program, you use the following algorithm on HP-UX:

get a temporary filename (use tmpnam or tempnam)

open the temporary filename

unlink the filename from the file system

If your program stops for any reason, the scratch file will disappear and the space the file occupied will be returned to the HP-UX free space list.

Temporary Files

On MPE, Qedit uses temporary files for scratch files and for “hold” files. On HP-UX there really are no temporary files. HP-UX programs just create permanent files in a directory considered to be temporary. The different-looking filename is a result of the HP-UX routine tempnam which creates a file with a unique name in /usr/tmp (HP-UX 9.x or earlier). So we modified Qedit/UX to create the qeditscr, hold, and hold0 files in the /usr/tmp directory.

Summary

This is a sample of the more interesting technical problems that occurred in porting Qedit from MPE to HP-UX. For more complete details on this project, read this paper on our Web site: www.robelle.com/ftp/papers/qxport.txt