I’m Old, Part II: Resources At Hand

Working at Adobe was an astounding experience. When I was stuck on a problem, I made a point of walking around and seeing what other people were working on. It was a good way to meet people and to see what was going on in the company beyond my cube. At this point, I’m working at home and miss the ability to do this.

At one point, I stumbled across Joe Holt, who was a developer on Illustrator for the Mac. I was always grateful for his friendship as he and I got along so easily. It was a commonality I’ve found in just a handful of people. At that point, I owned an Asteroids machine which was living in my dad’s house and I had the discovered that an Arcade in Oakland was closing, so I called Joe and on a Saturday, I got some cash, rented a truck and appliance dolly and Joe and I went up to scout out what they had.

We settled on a Robotron machine, paid for it, and hauled it back to Adobe where we set it up in a storage closet. We opened it up and set it up for free play and played a bunch of games. I returned the truck before it got too late and headed home.

Joe stayed late and pulled the ROMs from the game and put them into a ROM burner/reader and pulled the code. This is what I mean by resources. Since Adobe did a lot of embedded development, there were a number of machines at our disposal for reading/writing ROMS and they could handle the obsolete ROMs from the Robotron. Joe found a 6809 reference manual and wrote a 6809 disassembler in C and ran it on the ROMs, producing an ASCII listing which he put in a shared directory on his machine. Over the next few months, we went through the code and started commenting it and we figured out how the game worked at a fairly high level.

We had gotten stuck on some of the hardware addresses that were being used in the game, so I found the mailing address of Eugene Jarvis and asked him for some hints. He sent me a listing printed with an impact printer.

Scan-130712-0001

I found out the Robotron’s predecessor, Stargate, would run on a Robotron, so we bought some vintage ROMs from a local surplus house, downloaded Stargate, and burned some ROMs. Another engineer, an avid wood worker, built us a control panel that we wired up to play Stargate. We also set up Joust.

Since we had access to a hardware lab, we built a little panel of switches and buttons that we could wire up to a spare sound board and play with all the sounds in isolation of the game. Eventually, I ended up writing an emulator for the sound board and was able to get the sounds in realtime on a Macintosh Quadra.

It was so fantastic to be able to work and play in the same place. The reason it worked out so well was that Adobe had a very open and flexible approach to the hardware lab and so many of the engineers were willing and/or excited to talk about what they were working on or playing with.

I’m Old, Part I: Working on PostScript Printers

I was very lucky on a number of fronts in my career. In many things, it was a “right place, right time” confluence that worked out in my favor. For example, I knew 6502 assembly language as my second programming language and as a result had a fairly deep understanding of how software works at the lowest levels. I got involved in a program at Bell Labs in high school where I learned C one night a week. By the time I started college, I was already familiar with many of the concepts before I encountered them. And for no good reason, I did a private study in college on the language FORTH and did a reference implementation for the Macintosh.

Similarly, while I was in college, I worked at Bell Communications Research, doing work on Macintosh machines and Sun workstations before I decided to see what other opportunities lay before me. I managed to wrangle interviews at Apple and Adobe and thanks to some pull I had with an employee at Adobe (hey Curtis!), I landed a job there where, to my great relief, I discovered that development was being done on Sun workstations. Of course, I had no idea what PostScript was or how to do embedded software development, but I grabbed copies of the all the PostScript manuals on my first day and read up over the weekend. Oh hey – FORTH and PostScript share a lot in common. I can deal with this.

I took over on a project that was owned by DEC to work on their DECLaser 2200 series of printers. I was taking over the project from a woman who was going on maternity leave (hi Kathie!). On the day I started, she was in the middle of an early release and had no time to do much information transfer.

I took over, bouncing between my cube and her cube, trying to learn the ins and outs of not only PostScript, but also the sets of printers (there were two models that were going to be built from one code base), and their controllers. The underlying hardware was running on a National Semiconductor NS32CG16 processor which was sold to DEC as being a much better choice than the competing Motorola 68K series. National was lying through their teeth on a number of fronts, not the least of which was because the processor, which had an optional floating point unit, was designed in such a way that when the FP unit was installed, half of the machine registers (IIRC) magically went away, being dedicated to the FP unit. This meant that if you had to compile code to run with and without the FP unit, you had to assume that you had half as many registers as you might have. It made the code nearly impossible for a compiler to optimize, so the end result ran OK. Just OK.

In the middle of the project, I was given an awkward assignment to work with engineers from National to try and address the problems DEC was seeing. It involved a lot of benchmarks and we were stuck using GCC for the main compiler, but the National engineers were telling us that if we used their compiler, we’d get better performance, but the output file format was incompatible with the rest of out tool chain for making code that would embed on this controller. I was required to die on that hill.

Still, in the process, I learned a lot about development and the benefit of having a solid QA department who was on your side. To keep organized, I kept a binder where I put copies of the bug reports. I had them in sections for done, in progress, and needs to be done.

One of the problem with printer bugs is that sometimes it takes a long time just to reliably reproduce a bug. For example, there was a bug that was reported that would occur once in a while with only a certain set of serial settings when the printer was printing in duplex (both sides of the paper). I think I ended up using close to an entire case of paper finding and fixing that bug. It turned out to be a bug in the low-level code that handled serial input through a ring buffer. The stars had to align just right in terms of rate of input and interrupts for that one to happen. I found the bug and made sure that it made it into the main code base so that future printers wouldn’t have the problem. This was the “right thing”, but it turned out that it didn’t really matter because most of development at that point was focused on the next version of PostScript, which had none of the old I/O code.

Working on that printer was really entertaining because it was my first exposure to an ICE (In Circuit Emulator), which was a chunk of hardware that plugged into the CPU socket and acted like the main CPU, but let me set instruction level breakpoints, look at registers and memory, and generally figure out what was going on.

In the process of working on this printer, I learned a lot about corporate/political processes. There was a bug that opened up on the printer wherein the PostScript code would draw a solid 50% gray box on the page then render an image of a teeny-tiny helicopter scaled up to fit the box. When rendered on the page, the helicopter was supposed to cover the entire gray box, but on my printer (and in fact on all PostScript printers), there was a gray hair line on two edges – either the top or bottom and the left or right. Math said it shouldn’t happen, but it did.

I spent a week on this bug. After a lot of hacking and fiddling, I was able to determine that it was due to floating point error and that the location of the hairlines was predictable based on the floating point implementation and the margins settings of the print engine. DEC accepted my analysis and closed the bug. My boss was shocked – DEC opened this bug on every printer they worked on with Adobe. It was a tradition. Oh, we’re working with Adobe on a printer? Let’s just open this bug up on it. Suck it DEC, I have the answer!

At one point, as this project progressed, I had gotten a little behind and DEC was getting unhappy. They had a number of high priority bugs that they wanted fixed and were worried about the schedule. In total, there were around 40 bugs open on the printer. I managed to sort through their high priority bugs fairly quickly and then pounded through the rest. We had a conference call with them where they were assuming that I hadn’t fixed the high priority bugs and were ready to do some intense horse trading on them. When they asked about the status, I said, “I fixed all the bugs.” There was a long pause. “All the high priority bugs?” “No,” I replied, “all the bugs.” There was another long pause. “Is that OK? I mean, I’m pretty sure I can put some of them back if you’d like…” Peals of laughter. They faxed me a thank you card later in the day.

Unfortunately, there was one solid fuck-up in the printer and it was all on me. The printer had a tiny amount of non-volatile RAM that was used to store settings like the number of pages printed, the serial settings and so on. Through many changes and revisions, I had managed to subtly break the NVRAM code such that it worked great as long as the NVRAM had been written to once. If it had never been written to, the printer ended up going to a dreaded procedure named “CantHappen”, which would make the printer reboot. So when a brand new printer had the PostScript cartridge installed, it would boot, then try to read NVRAM, fail and reboot. Lather, rinse, repeat. DEC was in a panic about this because the printer was in production and the ROMs that went into it were already “masked”, which means that instead of burning each ROM individually, they were making the chips directly with my bug in them and they had already paid for thousands of them. I came up with a solution which was a chunk of code that would initialize the NVRAM and print a happy message on the front panel. Adobe footed the bill to have this code put onto a one-time-programmable cartridge and shipped with all the version 1.0 printers. You put in the OTP cartridge, it formatted the NVRAM and made it so that the version 1.0 printer would work. I wasn’t proud that that bug had made it into final release, but DEC and Adobe were amicable about the solution.

Credit Card Insecurity

Ingenico-iSC250-3

There’s no doubt that credit cards have overtaken other forms of payment, especially for online transactions. Unfortunately,¬†many sites that accept credit cards have payment systems that have security flaws in them, not the least of which include storing the credit card information. This is supposed to make it easier for repeat customers – lower the barrier to entering payment information and it will make it easier to buy something next time. In reality, it just provides another vector for attack.

My credit card has been replaced 3 times in the past year and a half, and every step has been done incorrectly.

The first misstep is how I find out about it – by failing to make a purchase, usually at a very inconvenient time. The credit card company has at least 3 different ways to contact me, some extremely expedient, but they don’t use them. Instead, they require a failure and a subsequent angry call from me.

The second misstep is that they need to send me a replacement card through express mail. Honestly, they could have already sent the replacement at the time of failure.

The third and most important misstep is that every time this happens, I ask what the cause was. Who was responsible for causing this inconvenience to me? I have been told every time that my card issuer has not been informed and they can’t tell me. The problem here is that this offers me no recourse – all that can happen is that I have to go through the first two missteps repeatedly.

The reason why this information is not provided is two-fold. First, the companies that are responsible are typically big targets. They think that it would be marketing suicide if the world found out that their data was compromised. Second, the companies are big enough to grab the credit card companies by the short and curlies to require them to keep it quiet.

Hiding a failure in data security makes the problem worse, not better.

Instead, what should happen is that the companies should own their mistakes. Publish the cause, publish the process by which it will be fixed, publish the progress, publish how that class of error will be prevented in the future.

Hiding the scapegoat also is a break in the free market. Without the information, I don’t have the agency to make a wise buying decision. Should I eschew credit at this establishment? Should I use paypal? Should I use virtual credit card numbers? Should I decline to create an account, but just use a guest checkout? Should I avoid the establishment until it has a fix in place?

Coding is hard, I get that. It’s been my profession for most of my adult life. I’ve made my share of bugs and mistakes. I’ve found that the best way to handle any of the issues has been to be honest and up front about it; to understand the problem fully and propose fixes. Hiding the problem – even with a fix – is sub-optimal.