I’m Old, Part VII: This Code Changes Itself

On modern CPUs, there is a lot we take for granted. Memory abounds and we’ve more or less settled on a set of addressing modes that make it reasonably straightforward to implement high level language features.

I started coding on an Apple II+ that my dad bought in 1979. We hooked it up to a TV up until the point where my dad picked up a black and white monitor from a computer swap meet. Initially, I started with BASIC, but grew frustrated with its (lack of) speed.

One day, when I was home sick, I sat down at the computer with a copy of the Apple II Reference manual and started looking at the routines that were in ROM. The one I started with was called COUT and lived at address $FDED. The manual said that the character to print would be in the A register. OK – easy enough. There was an instruction to put a value in A, called LDA and the variant I wanted was 0xA9, which I found out by looking it up in Programming the 6502. I decided to print HI, so I started it at address $300, which had, more or less, 256 bytes of space to work from. I dropped into the monitor and entered my code:

300:A9 C8 A9 C9 20 ED FD 4C 00 03

which turned into:

300: LDA #$C8 ; Put ‘H’ in A

302: JSR $FDED ; call COUT

305: LDA #$C9 ; Put ‘I’ in A

307: JSR $FDED ; call COUT

30A: JMP $0300 ; Jump to 300, the start

I ran it an the screen filled before I knew it. It was amazing. I was hooked on the speed.

Unfortunately, the 6502 is a fairly broken CPU in a number of ways. It’s got registers you can use as indexes, but they’re 8 bit, so you can’t address anywhere near the whole address range. It’s a serious hassle. Also the stack on most 6502 machines maxes out at 256 bytes, which means that things most programmers take for granted like using the stack for passing parameters or for locals is pretty much off the table. You also can’t nest functions particularly deeply and no way are you going to do recursion. Heaven forbid.

Since I started writing assembly by typing in raw bytes, I had in intimate familiarity with the encoding of the instructions. I found that I could make up for weaknesses in the hardware architecture by rewriting code on the fly. Don’t have a 16 bit address register? No problem – just rewrite the high byte of the target address of the instruction you want to change and ta-da: 16 bit indexing.

I also found out that this is pretty much the fastest way to get things to work, especially when I was writing graphics routines where the 1MHz CPU was just not quite fast enough.

When I got to college and took CS courses, I took an assembly language course and the professor spent a small amount of time talking about self-modifying code and said with reverence that John Von Neumann often wrote self-modifying code. Wait? Was it supposed to be hard? Seriously? I had been writing self-modifying code since I was 13.

A few years back, there was a thread on Metafilter about the Apple II, and I fired up an emulator and knocked together this little chunk of code for grins, which naturally, contains self-modifying code – in this case to get around the limitation of stack space for parameters, a dearth of registers, and to avoid the cost of doing arithmetic in the middle of a loop by precalculating the invariant values and stuffing them into the right places in the code.

I’m Old, Part VI: I Can’t Believe My Past Sometimes

When I was in high school, my French teacher, a truly wonderful woman, was hired by Max Mathews to teach him French. She started an extracurricular club call The Erasmus Club, which was a nice haven for the geek set in my High School. Max did a demonstration for the club of one of his research projects, an electronic violin that was designed to sound as close as possible to an Amati violin.

Through this set of connections, my French teacher arranged (or at least I think so), to get me a job working at Bell Labs with Max Mathews. My job was violin maker. I measured the parts in his prototype, made drawings, and then made many copies. I think I made something like 14 or the instruments.

4651368848_351711ed8f_o

I loved the job. I got to work in an illegal machine shop within Murray Hill Bell Labs, work with wood, aluminium, and nuts and bolts.

But wait – there’s more! After working for a little while, I asked for an account on their UNIX systems, and it was granted. I had accounts on two of the Vaxes: Alice and Rabbit. I played a lot of Rogue, learned the ins and outs of UNIX, had email, and a compiler.

Oh and did I mention what was down the hall? The UNIX room. This is where I met Brian Kernighan, Dennis Ritchie, Rob Pike, Tom Duff, Andrew Hume, Ken Arnold, Joe Condon, and Robert Morris (just to name a few). I poked my head into the UNIX room frequently to see what they were working on.

It was truly a formative experience.

I also would walk the halls of Bell Labs and see what else there was to see. Once I found the chemical stock room and for grins, I paged through the catalog of things you could get. It might as well have been the periodic table of the elements. Hmmm…I paged to the P’s. Yup. You could, in fact, order Plutonium, but there were a number of annotations indicating that you pretty much needed military clearance and a signature from God in order to actually get it. Fair enough.

I know that I wasted a lot of time there, but in the process I learned so very much and was very lucky to have the opportunity to do so.

I will also mention that there is one job that I refuse to remove from my resume, and it’s Violin Maker.

I’m Old, Part V: Messing with Managers

While I was in college, I took a year off and worked as a contractor at Bell Communications research. During that time, I found out about the merit review process at Bellcore. The first thing each employee does is fill out a form that was pejoratively called an “I Am Great Form”. In this form, you have to sing the praises of everything you accomplished in the previous year. Next your manager collects the forms and then all the managers cloister themselves in an office and order the forms, sorting by greatness. The greater your ranking, the greater the raise you would get.

It’s a horrible process and I bet that the managers hated it even more than the employees.

My friend Dan and I had a brilliant idea. The office that the managers were in had a speaker which was hooked into an experimental phone system that could pipe you notifications of phone calls read by a DECtalk speech synthesizer (samples, here). As a user of the system, you could either take the default, which was “call for <username>”, but a lot of people changed theirs and put in the magic escape codes to change the output voice. One user had a high female voice say, “bridge to Captain Kirk”. Another had a gravelly voice saying, “Yo! Talk to me baby!”.

This is neither here nor there. What was important to us was not that there a speaker, but that there was a wire going to a speaker that we could try to use as a microphone. So we found the terminus of that line in the machine room and hooked up an amplifier and tried to get sound out of it. No good. We could get hints of voices, but nothing clear.

Since there had been a big computer recently installed (I think it was a Pyramid Technologies machine), there was a correspondingly big cardboard box. Dan and I cut one side panel off the box, got a big marker and wrote “SPEAK UP! WE CAN’T HEAR YOU!” on it. Then we got a microphone on a long cord and some long lengths of wire to tie to the sign. Dan and I climbed out onto the roof and went to the corner above the office full of managers, then dangle the sign and the microphone by the window. Then, since it was a corner office, we went to the other side of the corner and dangled there for good measure.

We heard later that they liked that quite a bit – broke up the torture.

It turns out that managers have to go through the same process with their bosses, so in this case shit rolls uphill.

At the time, I had written a raster driver for an early HP ink jet printer so you could print graphics from windows in the in-house windowing system, MGR. I did a lot of hacking on MGR and one of the early fonts looked like kid’s handwriting. I used that for many of my windows just for grins.

My manager saw it and that I could print with it and asked me to print his “I Am Great” form in that font. You dance with them what brung ya, so I obliged. He also asked me to change the title from “Performance Review, 1985 – T. K. Landauer” to “WHAT I DID LAST YEAR BY TOMMY LANDAUER.” For good measure I popped a couple bitmapped fingerprints onto the page underneath the text. He was thrilled.

As a side note, most of the department had standardized on Sun 3 computers, which were OK when they ran, but were prone to failure. I personally witnessed my machine shit the bed 3 times. Each time, the Sun maintenance tech who was in the building would come in and swap the motherboard, but would put the Ethernet MAC address ROM from the bad board into the new one. After the third time, he told me that when a motherboard failed, it was sent back for testing and if it tested green, it went back out for use as a replacement. They wouldn’t scrap it unless it had failed a certain number of times in the field. At least that’s what he told me. He could have been yanking my chain.

Still, a manager on my floor, Lynn Streeter, had a conversation one weekend with a non tech person. I’ll write it the way that person had heard it.

Non-tech person: “How are things going?”

Lynn: “Well, my son died last week.”

NTP: “I’m so sorry!”

Lynn: “It’s OK – I’m getting another one; I’m going to name it Bullwinkle.”

Lynn picked up the look of abject horror on the person’s face and explained the Sun/son issue.

Of course you then have to explain the whole computer naming thing to people as well. This department used a bunch of different conventions. For example, file servers were words that ended in ‘day’. The days of the week were the first 7, there was holiday, one-a-day, Farraday, etc. One set of machines was named after Dwarfs that never were: Filthy, Grubby, Sleazy. When 3 NeXT stations appeared in the lab, I insisted that they be named Dacron, Orlon, and Rayon since they looked and acted so differently from everything else.

My Sun was named Peabody.

I’m Old, Part IV: Weird Printers

Printer manufacturers had some interesting challenges in front of them in the 90’s. For the most part, Canon was one of the big print engine manufacturers and if you were a making a printer with PostScript using a Canon engine, you were competing against HP, DEC, Canon, Apple and probably a few other big names.

You need to look for ways to differentiate your product from other manufacturers. If you bought into Adobe PostScript, that was one thing, but that raised the cost of your printer. In for a penny, in for a pound, so lets make the PostScript implementation more complicated by having a front panel UI and a whole bunch of custom PostScript operators that were “value add” (and for which no driver except maybe the one in the manufacturer’s proprietary print server software would use.

Still, we saw some wacky printers that should never have seen the light of day. One was worked on by my co-worker, Kathie. This printer had the code name ‘Jolt’ and was a designed by Rube Goldberg.

Our boss, John, had a technique for estimating the time that it would take to complete a PostScript printer implementation. It was, if I recall correctly, based on a log scale on the number of complications in the project. Jolt had so very many.

First off, let’s talk about the controller. A printer controller is an embedded system with hardware for reading input for print jobs, controlling the print mechanism, and reading user input. Many manufacturers used very common controllers and Adobe very often would have reference designs that printer manufacturers could use. If they chose a reference design, the time to market was usually quicker. Not this company. Oh no. They chose the Weitek 8220, which was an early RISC processor with a lot of high ideals which interfered with practical engineering. The first was that it ran on a Harvard Architecture model which meant that there were two separate address and data buses. One of code and one for data and never the twain shall meet. Unfortunately, this meant that it was next to impossible to have any kind of reasonable debugger for it, since debuggers often modify existing code to replace existing instructions with break instructions.

Next, the printer was color (there were not many at that point), and instead of a toner or dye sublimation printer, this one used crayons. It had four hoppers that were similar to Pez dispensers that were loaded with color pellets. When you turned on the printer, the print head had a heater that kept 4 separate molten pools of wax that were shot out of the print head onto the paper.

Paper was sheet or tractor fed, but the hybrid design was clumsy. In order to keep the paper flat against the platen, the plated had a row of holes in it that were connected to a modified pump that acted as a vacuum, rather than a pump. The actual device in the printer was an actual aquarium pump.

When the printer sensed that it was low on crayons, it would send the print head to the hopper to feed itself. If you made the mistake of turning the printer off, it would require a much longer start up time in order to melt the wax and very often the jets would get clogged. Therefore the printer required special cleaning paper which was a piece of paper with a puffy diaper in the middle and some markings on the edge that the printer would recognize when it was fed in that would send the printer into “cleaning mode”. At the point, the printer would attempt a technicolor Montezuma’s revenge onto the diaper complete with a back and forth wipe at the end.

It was a miracle to see it work.

Meaning, it rarely worked.

I felt very sorry for Kathie, and if I recall correctly, this was the printer that made her decide that maybe printer development was not her cup of tea. I was sad to see her go.

I’m Old, Part III: More Unusual Resources at Hand

At the time that I was at Adobe, they had some pretty fantastic Holiday parties. I really liked going to these because it was an excuse for me to pull out formal wear. Having a musical bent, I make sure that I always have several sets of formal wear ready to go. I’ve played trumpet at a number of weddings (mostly co-workers) and there are expectations that male musicians in the church should wear a monkey suit. I’m OK with that as I’ve always held that I look good in formal wear.

Honestly, I don’t have a great deal of vanity, except when it comes to formal wear. Last year, I got the opportunity to play the part of the butler in a local production The Nutcracker. When there was a mix-up in the costume for me at the dress rehearsal and the costume wasn’t fitting me, so it was easy enough to run home and pick up a set of tails.

IMG_20151212_130715882 (1)

But that’s neither here nor there.

For one party, I though it would be cool to have an Adobe branded cummerbund and tie. I had ready access to the PostScript logo artwork and was able to knock together a PostScript program that rendered it as if it were a fabric pattern.

postscript-logo

I wanted to be able to silkscreen that onto some, well, silk, but I wanted to remove as much friction from the process as possible. I talked to Jeff Sherwood, who worked on printers that printer to film at very high resolutions and got him to inject my job into one of his test runs so I could get a nice high resolution print of the logo, ready for making a silk screen. I found out from Andy Shore, a long time Adobe employee that Adobe green was pantone 321. Then I contacted a single person t-shirt printing shop and offered the owner the job. She wasn’t sure that she could do it, but was willing to try if I would help. Together, we printer 3 yards of PostScript fabric that I then turned over to a friend of mine who was willing to make a cummerbund and tie in exchange for homemade jams and jellies.

20160616_193411

I was quite happy to wear this to the party and show it off to John Warnock and Chuck Geschke, who were both similarly happy to see it.

20160616_193422Details matter to me and I’m very glad that I was able to put time and effort into this project and that so many people were able to help me with it.

I’m Old, Part II: Resources At Hand

Working at Adobe was an astounding experience. When I was stuck on a problem, I made a point of walking around and seeing what other people were working on. It was a good way to meet people and to see what was going on in the company beyond my cube. At this point, I’m working at home and miss the ability to do this.

At one point, I stumbled across Joe Holt, who was a developer on Illustrator for the Mac. I was always grateful for his friendship as he and I got along so easily. It was a commonality I’ve found in just a handful of people. At that point, I owned an Asteroids machine which was living in my dad’s house and I had the discovered that an Arcade in Oakland was closing, so I called Joe and on a Saturday, I got some cash, rented a truck and appliance dolly and Joe and I went up to scout out what they had.

We settled on a Robotron machine, paid for it, and hauled it back to Adobe where we set it up in a storage closet. We opened it up and set it up for free play and played a bunch of games. I returned the truck before it got too late and headed home.

Joe stayed late and pulled the ROMs from the game and put them into a ROM burner/reader and pulled the code. This is what I mean by resources. Since Adobe did a lot of embedded development, there were a number of machines at our disposal for reading/writing ROMS and they could handle the obsolete ROMs from the Robotron. Joe found a 6809 reference manual and wrote a 6809 disassembler in C and ran it on the ROMs, producing an ASCII listing which he put in a shared directory on his machine. Over the next few months, we went through the code and started commenting it and we figured out how the game worked at a fairly high level.

We had gotten stuck on some of the hardware addresses that were being used in the game, so I found the mailing address of Eugene Jarvis and asked him for some hints. He sent me a listing printed with an impact printer.

Scan-130712-0001

I found out the Robotron’s predecessor, Stargate, would run on a Robotron, so we bought some vintage ROMs from a local surplus house, downloaded Stargate, and burned some ROMs. Another engineer, an avid wood worker, built us a control panel that we wired up to play Stargate. We also set up Joust.

Since we had access to a hardware lab, we built a little panel of switches and buttons that we could wire up to a spare sound board and play with all the sounds in isolation of the game. Eventually, I ended up writing an emulator for the sound board and was able to get the sounds in realtime on a Macintosh Quadra.

It was so fantastic to be able to work and play in the same place. The reason it worked out so well was that Adobe had a very open and flexible approach to the hardware lab and so many of the engineers were willing and/or excited to talk about what they were working on or playing with.

I’m Old, Part I: Working on PostScript Printers

I was very lucky on a number of fronts in my career. In many things, it was a “right place, right time” confluence that worked out in my favor. For example, I knew 6502 assembly language as my second programming language and as a result had a fairly deep understanding of how software works at the lowest levels. I got involved in a program at Bell Labs in high school where I learned C one night a week. By the time I started college, I was already familiar with many of the concepts before I encountered them. And for no good reason, I did a private study in college on the language FORTH and did a reference implementation for the Macintosh.

Similarly, while I was in college, I worked at Bell Communications Research, doing work on Macintosh machines and Sun workstations before I decided to see what other opportunities lay before me. I managed to wrangle interviews at Apple and Adobe and thanks to some pull I had with an employee at Adobe (hey Curtis!), I landed a job there where, to my great relief, I discovered that development was being done on Sun workstations. Of course, I had no idea what PostScript was or how to do embedded software development, but I grabbed copies of the all the PostScript manuals on my first day and read up over the weekend. Oh hey – FORTH and PostScript share a lot in common. I can deal with this.

I took over on a project that was owned by DEC to work on their DECLaser 2200 series of printers. I was taking over the project from a woman who was going on maternity leave (hi Kathie!). On the day I started, she was in the middle of an early release and had no time to do much information transfer.

I took over, bouncing between my cube and her cube, trying to learn the ins and outs of not only PostScript, but also the sets of printers (there were two models that were going to be built from one code base), and their controllers. The underlying hardware was running on a National Semiconductor NS32CG16 processor which was sold to DEC as being a much better choice than the competing Motorola 68K series. National was lying through their teeth on a number of fronts, not the least of which was because the processor, which had an optional floating point unit, was designed in such a way that when the FP unit was installed, half of the machine registers (IIRC) magically went away, being dedicated to the FP unit. This meant that if you had to compile code to run with and without the FP unit, you had to assume that you had half as many registers as you might have. It made the code nearly impossible for a compiler to optimize, so the end result ran OK. Just OK.

In the middle of the project, I was given an awkward assignment to work with engineers from National to try and address the problems DEC was seeing. It involved a lot of benchmarks and we were stuck using GCC for the main compiler, but the National engineers were telling us that if we used their compiler, we’d get better performance, but the output file format was incompatible with the rest of out tool chain for making code that would embed on this controller. I was required to die on that hill.

Still, in the process, I learned a lot about development and the benefit of having a solid QA department who was on your side. To keep organized, I kept a binder where I put copies of the bug reports. I had them in sections for done, in progress, and needs to be done.

One of the problem with printer bugs is that sometimes it takes a long time just to reliably reproduce a bug. For example, there was a bug that was reported that would occur once in a while with only a certain set of serial settings when the printer was printing in duplex (both sides of the paper). I think I ended up using close to an entire case of paper finding and fixing that bug. It turned out to be a bug in the low-level code that handled serial input through a ring buffer. The stars had to align just right in terms of rate of input and interrupts for that one to happen. I found the bug and made sure that it made it into the main code base so that future printers wouldn’t have the problem. This was the “right thing”, but it turned out that it didn’t really matter because most of development at that point was focused on the next version of PostScript, which had none of the old I/O code.

Working on that printer was really entertaining because it was my first exposure to an ICE (In Circuit Emulator), which was a chunk of hardware that plugged into the CPU socket and acted like the main CPU, but let me set instruction level breakpoints, look at registers and memory, and generally figure out what was going on.

In the process of working on this printer, I learned a lot about corporate/political processes. There was a bug that opened up on the printer wherein the PostScript code would draw a solid 50% gray box on the page then render an image of a teeny-tiny helicopter scaled up to fit the box. When rendered on the page, the helicopter was supposed to cover the entire gray box, but on my printer (and in fact on all PostScript printers), there was a gray hair line on two edges – either the top or bottom and the left or right. Math said it shouldn’t happen, but it did.

I spent a week on this bug. After a lot of hacking and fiddling, I was able to determine that it was due to floating point error and that the location of the hairlines was predictable based on the floating point implementation and the margins settings of the print engine. DEC accepted my analysis and closed the bug. My boss was shocked – DEC opened this bug on every printer they worked on with Adobe. It was a tradition. Oh, we’re working with Adobe on a printer? Let’s just open this bug up on it. Suck it DEC, I have the answer!

At one point, as this project progressed, I had gotten a little behind and DEC was getting unhappy. They had a number of high priority bugs that they wanted fixed and were worried about the schedule. In total, there were around 40 bugs open on the printer. I managed to sort through their high priority bugs fairly quickly and then pounded through the rest. We had a conference call with them where they were assuming that I hadn’t fixed the high priority bugs and were ready to do some intense horse trading on them. When they asked about the status, I said, “I fixed all the bugs.” There was a long pause. “All the high priority bugs?” “No,” I replied, “all the bugs.” There was another long pause. “Is that OK? I mean, I’m pretty sure I can put some of them back if you’d like…” Peals of laughter. They faxed me a thank you card later in the day.

Unfortunately, there was one solid fuck-up in the printer and it was all on me. The printer had a tiny amount of non-volatile RAM that was used to store settings like the number of pages printed, the serial settings and so on. Through many changes and revisions, I had managed to subtly break the NVRAM code such that it worked great as long as the NVRAM had been written to once. If it had never been written to, the printer ended up going to a dreaded procedure named “CantHappen”, which would make the printer reboot. So when a brand new printer had the PostScript cartridge installed, it would boot, then try to read NVRAM, fail and reboot. Lather, rinse, repeat. DEC was in a panic about this because the printer was in production and the ROMs that went into it were already “masked”, which means that instead of burning each ROM individually, they were making the chips directly with my bug in them and they had already paid for thousands of them. I came up with a solution which was a chunk of code that would initialize the NVRAM and print a happy message on the front panel. Adobe footed the bill to have this code put onto a one-time-programmable cartridge and shipped with all the version 1.0 printers. You put in the OTP cartridge, it formatted the NVRAM and made it so that the version 1.0 printer would work. I wasn’t proud that that bug had made it into final release, but DEC and Adobe were amicable about the solution.