Working with Files With Unprintable Characters

May 2005

Get the most out of your Robelle Tools

Working with Files With Unprintable Characters

By Robelle Staff

While working on a file that was originally produced by a PC program (PPTIA, the PowerPoint Internet Assistant from Microsoft), we transferred the file to our HP-UX system and needed to clean it up before putting it up on our Web server. The file was not editable in Visual mode because many lines contained unprintable (non-printing ASCII) characters. These were shown as dots/periods on the Visual mode screen, and Qedit showed a question mark at the start of each line to indicate that there were questionable contents.

1. What were these characters?

2. How could I get rid of them?

Answer 1: We used the Char and Decimal options of Qedit’s List command to see the numeric values of the offending characters. For example:

qux/list $h $c 35
35
0000: 0D3C 4832 3E52 6F62 656C 6C65 2043 6F6E 7375 6C74 .<H2>Robelle Consult
000A: 696E 6720 4C74 642E 3C2F 4832 3E20
ing Ltd.</H2>

We knew the line should have started with <H2>, but it had an extra mystery character at the start, which was shown as a dot on the right side of the List output. In the example, on the left side we can see that the character has a hex value of 0D.

Answer 2: The Change command can replace the offending character with a normal printing character, so that it can be edited in Visual mode, or we can use Change to remove the character by changing it to nothing. You can specify strings in the Change command by their numeric values. The numeric values must be specified in decimal, from 0 through 255. We know the hex value, 0D. Using the handy calculator built into Qedit, we see that hex 0D is decimal 13:

qux/=$0d
Result=13.0

Now we can change the character whose value is 13 to nothing. We’ll do it in all lines of the file. First we put Qedit into “decimal mode”, then we do the Change command:

qux/set decimal on
qux/change ‘13 “” all
35 <H2>Robelle Consulting Ltd.</H2>
36 </P>
37 <P> —>
3 lines changed

There were a number of other mystery characters besides 0D/13. These we changed to other printing characters. When we had figured it all out, we put together a Use file with the change commands, so that we could easily make the same change to all the files created with PPTIA. The Use file had these lines in it:

qux/l ufix
1 set dec on
2 cq ‘9 “ “ @ {HT tab}
3 cq ‘11 “<BR>” @ {VT vertical tab, new line}
4 cq ‘13 “” @ {CR}
5 cq ‘145 “’” @ {opening single quote}
6 cq ‘146 “’” @ {closing single quote}
7 cq ‘147 \”\ @ {opening double quote}
8 cq ‘148 \”\ @ {closing double quote}
9 set dec off

Another Way to Identify Nonprinting Characters

The following question came to us from the support mailbox:

“I have a QEDIT file that is displaying a “?” at the beginning of some lines. I understand that this indicates that the line contains nonprintable characters. How can I get QEDIT to display the data so that I can determine what the data is?”

One way to display the data is by using QZMODIFY as follows:

The Visual screen with an unknown character in the line will resemble the following:

===>
Start of file 1 QED21739.PAUL.ACCOUNT
*?sadlkfjslkdj.flkfjkl
^ unknown character
// .....+....1....+....2....+....3....+....4....

Set modify to QZMODIFY: /set mod qz. Now modify the line in question: /m1

Qedit will display the line, but the cursor will not be underneath the line, it will be on the line.

Press the spacebar until the cursor is directly on top of the line with the mystery period, then press Control-W followed by a question mark (?).

1 sadlkfjslkdj.flkfjkl

Qedit will then display the data:

ASCII character code = #25 , %000031, $0019

In this case, the character was a decimal 25 = octal 31 = hex 19. Press the Return key to get out of modify mode.

1 sadlkfjslkdj.flkfjkl

If you need to change this character, then use Set Decimal On and single quotation marks to isolate the decimal 25. For example,

/set dec on
/c ‘25 “#” 1
1 sadlkfjslkdj#flkfjkl
1 line changed