Analysis by John Burke
"Some restrictions may apply"
Usually you hear this phrase at the end of a commercial for something that
seems too good to be true -- and probably is.
Old timers will recall that one of a system manager's more delicate, and interesting, tasks on the Classic MPE systems was configuring table sizes and managing the restrictions on the size of various data structures. For example, on MPE V, directories are stored in fixed sized b-trees. Following are the estimated values allowed for MPE V (assuming that the accounts, groups, users, and files were added to the system in alphabetical order). Note that the actual maximums may be higher or lower depending on the order in which you set up and modify the directory.
Total number of accounts in the system 561
Total number of groups per account 311
Total number of users per account 623
Total number of files per group 1322
[From a Response Center Q&A submitted 12/07/89, which points out that the System Operation and Resource Management Reference Manual gives values that are incorrect (too large).]
The truth of the last statement was born out for me in the late 1980s. I inherited an MPE/V system that had an account with considerably fewer than 300 groups defined but which was frozen because of the way the groups were entered -- there was no way could any more groups be added.
When MPE/XL came along I was both dismayed and thrilled. Thrilled because most of the arbitrary, and low, limits on things such as number of groups per account had been removed (phrases like "no practical limits" were being thrown around). Dismayed because much of the system tuning available to system managers of MPE V systems was removed (system table/buffer size is generally not configurable, but rather good values are established in the lab). This turned out not to be a problem for me until MPE/iX 5.0. With this release the number of terminal buffers became an issue for us. The number of pooled terminal buffers is not configurable, so we had to perform a Rube Goldberg workaround to prevent sessions from hanging.
Now many users are finding that "no practical limits" doesn't have the meaning it once had. Someone about to build the "mother of all accounts" said
"I know there must be a limit to the number of groups that can be in any one account. Does anyone know what that limit is? Or where I can find it? Also, the maximum number of files that can be in a group or account? Also, the maximum number of users in an account, or that can be 'homed' to a given group?"There is a brief article, "Introducing the New DIRLIMIT Utility", in the MPE/iX 5.0 Communicator (Page 3-79) that discusses this issue. More detailed explanations were posted to the list by several HP engineers. I am going to try to present a consolidated version of their responses. For information on other system limits, see the MPE/iX 5.0 Communicator (pages 10-42 to 10-47) and/or, as Craig Vespe noted for "thrill seekers," the Web site version of the paper ]
From HP:
On MPE V, directories are stored as b-tree structures. However, MPE XL (and MPE/iX) have always stored the root, account and group directories as special hidden, sorted files. Since these directories are files, they have a theoretical limit of 4Gb of file space to store directory entries. However, other system limits will form practical limits long before you reach 4Gb.Determining a practical limit (like most system limits) is an art as much as a science. The current major gating factor for directories is that directory contents are attached to transaction management, and there is a limit on the size of the transaction management system buffer. The more data that you try to log, the more you increase the chance of having a stalled transaction that will result in a system abort. Therefore, the smaller your directories, the safer you are.
The thing to keep in mind is that all activity in directory nodes (insert, delete, rename) causes transaction manager (XM) activity (i.e. transactions), since all directory nodes' integrity is protected by XM. The larger the directory nodes, the larger the transactions. If your directory nodes grow too large, the performance of your system may suffer due to the large disc IOs that XM has to perform to protect your system from directory corruptions (in case of a system abort).
As a general rule of thumb, directory sizes of 1Mb or less should be highly unlikely to cause any type of trouble. Now, to translate this into the number of entries per directory, you have to know the entry size. To keep things interesting, the root, accounts, groups, and HFS directories all have different record sizes. This is because they each have to store different types of information such as accounting information, security information, in addition to the name and address of a file. Also, note that the directory entry size for a given directory type is constant, regardless of the type of object that the entry points to. So, an account directory entry will be 180 bytes large, whether it points to a group, an HFS directory, or a file.
The following table gives the record sizes for each directory type:
root 132 bytes ~7,943 entries/Mb accounts 180 bytes ~5,825 entries/Mb groups 40 bytes ~26,214 entries/Mb hfsdirs 96 bytes average ~10,922 entries/Mb
If you have a directory that exceeds these sizes, it's not the end of the world. The way you use the files, your system load, the phase of the moon and many other attributes will conspire to determine how likely large directory sizes are to cause your system to abort. I would suggest, however, that your limit your directories to 3Mb.
Another engineer recommended that you not to keep more than 50,000 files in a group. Since group entries in the account directory nodes are larger than file entries, a reasonable soft limit should be 20,000 groups per account.
Please note also that the above mentioned XM transactions have a limited physical size, and it is possible that you will encounter a system abort 2200, if you have too many files in a group or groups in an account. As of 5.0, you risk encountering this system abort if a group contains more than about 155,000 files (in a worst case scenario). In MPE/iX 4.0, the risk starts with more than 78,000 files.
A user reported that one client ran into this limit several times because the application frequently exceeded 10,000 files in a GROUP. He was advised by the Response Center that no more than 10,000 files should be placed in a GROUP.
And the response:
The RC is correct. You can have multiple XM transactions for different large directories on the same volumeset happening simultaneously, which means you can hit the XM limit with smaller directory sizes as well."MPE/iX Spoolers R' Us" noted:CSY lab is also investigating the design limitation of the transaction manager to address the above problem with a better long-term solution.
This (the XM limitation) was the reason we limited the spool file expansion to 50,000 (after discussion here in the lab). All those files go in OUT.HPSPOOL, so we wanted to set a reasonable limit with respect to the XM buffer. There are also performance hits as the number of spool files increase, but the overriding concern was preventing system aborts from XM.Leaving the world of files and addressing the world of users -- prior to MPE/iX 5.0 users were stored in a hidden directory associated with each account. However, since 5.0 all users have been stored in a system-wide user database called HPUID.PUB.SYS. This file has a limit of 10,000 entries, implying that 10,000 different users can be defined on a given system. There are no per account limits, and no limit on how many of these users can be "homed" to a single group.
I'll bet he wishes he did not ask
It seemed innocent enough. Gopi (M. Gopalakrishnan of CSY R&D, India) said:
"We are currently in the process of adding a new CENTURY CI variable for MPE/iX. The proposed HPCENTURY CI variable would return the century component (first two digit of the current year) corresponding to the HPYEAR CI variable. For example,There followed no fewer than 40 posts discussing the various aesthetics of naming conventions!In 1997, HPYEAR = 97 and HPCENTURY = 19
2000, HPYEAR = 0 and HPCENTURY = 20
2001, HPYEAR = 1 and HPCENTURY = 20 and so on."Hence, the HPCENTURY and HPYEAR variables could be used to get the 4-digit year.
"Question: Do you need, or how important is it to have a new CI variable for four-digit year, having 1997 for Year 1997, 2000 for Year 2000, and so on? If you need it, what would be the best name for the variable? (one suggestion is : HPYEAR4)."
It was pointed out that currently, HPYEAR returns 100 for the year 2000, but Gopi noted that would be fixed in the "next release" of MPE/iX.
The final result, I believe, is Gopi will add a variable called HPYYYY and, probably, a variable called HPYYYYMMDD. Their content is obvious.
Gopi's question to the group was part of a much larger project to add a number of new date intrinsics to MPE/iX. And this date intrinsic project is another example of the fantastic lines of communication that now exist between HP and its customers. This communication is facilitated by the Internet and HP 3000-L, but it is quite a tribute to HP CSY management that this dialogue is not only allowed, but encouraged.
In addition to the date intrinsic proposals, a system CI variables proposal has been thoroughly hashed out recently. Each discussion is too long to publish anywhere (even the "final" proposals are too long for this column). And, while the whole process undoubtedly takes longer than if HP just imposed a design, the result is clearly superior because the customers are getting what they want and need. Thanks, HP.