March 2005

HP labors over repair for IMAGE

Database’s LargeFile datasets cause corruption

HP preached the message of investment protection while it sold the HP 3000. The sermon apparently stuck at the vendor’s labs. Engineers there have been toiling for months on a repair that could make an IMAGE enhancement reliable at last — more than a year after HP stopped selling 3000s.

IMAGE’s LargeFile Datasets (LFDS) have been in repair mode for more than nine months, as HP’s engineers work to prevent data corruption which can occur while using LFDS. In some cases an access to one of these datasets will corrupt the data inside, an absolute no-no to the stickler customers who rely on IMAGE.

HP started the repairs after database utility vendor Adager reported the problem. The company first detected the problem during its internal tests of LFDS. Adager’s customers then began encountering data corruption when they used LFDS, according to CEO Rene Woc. The problem has been so severe that Adager has advised customers not to use LFDS and rely on Jumbo datasets instead. Jumbo datasets are multi-file datasets that consist of several “chunks,” each limited to a maximum of 4 GB.

HP provided LFDS with the best of intentions, hoping to overcome a limitation of Jumbos: the Jumbo datasets did not support dynamic dataset capacity expansion (called DDX for detail datasets, MDX for masters). Starting with version C.10.01 of IMAGE, HP 3000 TurboIMAGE databases with big datasets could use LargeFiles, which rely on the relatively new MPE/iX ability to create files larger than 4 GB. Adager offers a complimentary utility to convert a database with Jumbo datasets to one which uses LFDS.

An LFDS is a single MPE/iX file that can be as large as 128 GB. Such LargeFile datasets support dynamic dataset capacity expansion, which makes capacity management less critical for HP 3000 customers, according to Woc. But he said customers need to take notice of the risks of using LFDS before HP finishes its fix.

“The main thing is to be aware of the issues,” he said. “We’re in the bug avoiding business, so we need to identify them, so we don’t step on the land mines.”

A late February Internet message from Ken Sletten, the last chairman of the SIG-IMAGE special interest group, alerted customers that LFDS “is seriously broke.” Sletten noted that HP’s repair efforts were already overdue on a deadline that had been extended several times. The patch is still in process while HP has missed its own estimates of November and December of 2004.

“Not only did it not make the end of 2004, HP is now having to regroup at least to some extent on the planned fix,” Sletten said in his message. “The patch delivery date is now unknown.”

The data corruption can occur for any customer who uses LFDS, Sletten explained. “Any TurboIMAGE entry than spans the 4GB point of an LFDS will be corrupted by DBGET, and will corrupt data during DBPUT,” he said. These new datasets always have one block that spans the 4GB point, “unless the dataset BlockLength results in an MPE record size of 128 half-words, a rare event these days.”

After getting notified by Adager of the problem in April of 2004, HP has been working on the problem since June. Now the HP’s 3000 lifecycle is getting close to having less than 18 months of technical resources left for IMAGE repairs. Sletten and others are concerned that even if a patch for LFDS surfaces in beta test, a 3000 community now leery of making changes will avoid implementing it. A growing number of HP patches for the 3000 are struggling to get beta test results.

The problem is becoming more widespread when customers use DBSCHEMA, according to Woc. The new DBSCHEMA default creates datasets as LFDS; only issuing a $CONTROL JUMBO command in DBSCHEMA avoids the broken LFDS format when the resulting datasets are greater than 4 GB.

HP senior support engineer Cathlene Mc Rae updated the HP 3000 community on the LFDS problem the day after Sletten posted his message. “If you were considering this LFDS option, it is not working at this time,” she said. “The lab is working on the problem and we hope to have something in the near future.” Customers who want to test the patch should open an HP Response Center case, she added.

Some customers believe HP would do better to remove LFDS altogether, since a significant pool of beta-testers for the patch seems unlikely to surface. Meanwhile, the LFDS format has been spreading. HP’s designs don’t allow Jumbo datasets to coexist with LargeFile datasets in a single database. Customers who have needed the DDX and MDX dynamic expansion capability on their Jumbo datasets need to convert to LFDS, either with a DBUNLOAD/DBLOAD or with the complimentary utility from Adager.

HP’s repair efforts show that it is protecting its investment in LFDS development. The vendor considered an option to complete its design of Jumbo datasets several years ago, a path that would have give Jumbos DDX and MDX capabilities.

“In retrospect it seems it would have been better to complete the original design idea on Jumbo Datasets,” said Sletten, “and permanently forget about using LFDS in TurboIMAGE.” HP has indicated that extending the capabilities of Jumbos is no longer an option.

MPE/iX versions 6.5, 7.0 and 7.5 all have TurboIMAGE releases that permit LFDS. HP’s Mc Rae said the vendor is keeping Adager updated on the progress of the HP fix, and offered to notify customers through the 3000-L mailing list when a patch becomes available. “The new code is being tested now,” Mc Rae reported at the end of February. “A [release] schedule has not yet been set.”