I’m Old, Part LI: Hacking Sleep

When I was working on the Acrobat team, the work was intense and copious. It didn’t help that I came onto the project late so for search tools, I was way behind. I found myself working 10-12 hours a day and frequent weekends. This is the pattern that you’d expect from a project that falls into the “land grab” scenario. At the time that Acrobat was being built, there were a couple of other competitors and we had to ensure that we dominated the space. The way to do that in land grab is to be there first and to be a dominant force that can’t be dislodged. That meant big goals, high standards, and long hours to meet them.

At the time, I decided that I was going to try to hack my sleep pattern and see what the minimum amount of sleep I could get and still be a functional engineer. I did a basic divide and conquer approach and found that I was good at 6 hours for months at a stretch. I wasn’t happy or healthy, but nobody else was either so I was in good company.

I would come home, take care of dinner (if it hadn’t been supplied at work), took care of my cats, watched some TV, read, and then went to sleep at midnight until the alarm went off at 6. I got up, took care of my cats, had breakfast, showered, and headed to work and got in around 8:30. Then I worked until 6:30 or later.

One hobby I fit in at that time was baking. There were wonderful farmer’s markets in the area with incredible produce. I would buy all kinds of fruit when it was in season and that which I didn’t eat or can, I would use for baking experiments. About once a week, when I got up in the morning, I would put together a quick bread, pie, coffee cake, or cobbler and put it in the oven while I was showering. When I got into the office, I put it out at Karin Jurcevich’s desk. Karin was the department admin and ray of sunshine. Her desk was a focal point for a lot of the group. I would set up the treat along with some plates, napkins, and utensils, cut myself a serving then walk the perimeter and give the people who were at their desks the simple message, “Karin’s desk” and watch the hordes descend.

At one point, I started doing more extreme baking experiments. I made two checkerboard cakes – one from box mixes and one from scratch using Joy of Cooking recipes (IMHO, the Joy of Cooking recipes tasted better, had better texture, and were closer in overall size. Also, the difference in time between making them from scratch or mix and cleaning up was negligible). At one point I made a cherry pie that included the Acrobat logo sculpted on the top in dough. It was fun. Somewhere there exists a picture of that pie. I thought I had scanned it, but I couldn’t find it.

Of course, I later found that the amount of sleep I needed in order to remain productive is far less my 6 hour experiment. That number changed when I had kids. I routinely had weeks on end where I had 4 hours a night on average. I looked like a wreck at times and there were days when I came in and announced that I’d had 8 hours of sleep. Over the past 3 days.

At one point, Rick Minerich and I went a conference in Las Vegas. Our flight out got canceled due to heavy thunderstorms and we got put up in a cruddy hotel and rescheduled for a flight early the next morning. We didn’t get into the hotel until late and we hadn’t had dinner, so we ordered from a nearby delivery pizza place and shot the shit until entirely too late. We had to set an alarm for around 4:30 in order for both up us to be able to shower and make the shuttle. I woke up before the alarm by 5 minutes and used that time to bring all systems on line. The alarm went off and Rick stirred and started trying to fiddle with the coffee maker. I got in the shower and was out, dry, and ready to go. Rick just got his first cup of coffee and was in a state of shock that not only was I up and, bright-eyed and bushy-tailed, but I had done all of that without coffee (hate the stuff). The answer was simple: my kids ruined my sleep patterns and getting four and a half hours of sleep was a luxury.

To be fair, Rick was decidedly not a morning person in general, so I don’t blame him for being out of it that early.

And while the sleep experiments were intellectually interesting to me, I can’t say that I recommend them. Over time, I’ve come to believe that land-grab mentality is dangerous for both the health of the employees and the precedent it sets for corporate culture. Since Acrobat, I have only ever done one all-nighter and have done my best to protect my teams from having to do the same. How do you do this? By planning better, making better estimates, and if pulling the extra hours looks tempting, quote the holy trinity of late software:

  1. Lose a feature
  2. Ship later
  3. Lose quality

Pick 1 – and if you don’t pick one, you’re going to get at least 1 and likely all of them.

I’m Old, Part L: Cultural Exposure

For better or for worse, most of software engineering in Silicon Valley in the 1990’s was very white and very male. A good thing, I guess, was that there was a fairly wide variety of restaurants outside of this prosaic culture readily available: Chinese, Japanese, Mexican, Spanish, Indian. I had found a nice Indian restaurant in Sunnyvale called The Empress of India, which I loved. They had a menu if you really wanted, but frequently Jeannie, the chef would come to your table and ask you if you were hungry and she would just cook for you. The beauty of this was that you didn’t get stuck in a rut and Jeannie always delivered wonderful food.

Lunch at Adobe was taken by most engineers in the company cafeteria, which was quite decent (certainly much better than the one at Bellcore), but wore thin soon enough. Sometimes you just needed to get out and have something else once in a while.

Once, Eswar Priyadarshan invited us to join him at an Indian restaurant on Castro Street that he said had very good home cooking. Eswar ordered for us, if I recall correctly, and insisted that we eat with our hand using thumb and the first two fingers to pick up food and/or using bread as a scoop. When I say hand, I mean right hand. He was very particular about that. It was interesting especially for Kevin Binkley who was a southpaw.  Eswar told him to eat with his right hand and Kevin asked him why and Eswar simply said, “sh-sh-sh.” waving at him in a gesture to put his left hand down. I was now curious. I asked, “but why…” and was also interrupted with “sh-sh-sh.” Fair enough. I learned later that eating with your left hand is considered disrespectful.

While this experience was not a huge thing – I firmly believe that teams are made better by having breadth in culture and gender. To learn and to respect another’s culture is to effect a change in yourself and how you look at the world and in turn how you look at problems.

In the Hitchhiker’s Guide to the Galaxy, Douglas Adams at one point uses the phrase, “fire missiles at right angles to reality.” It’s meant as an impossible task. Every job I’ve had has involved tasks that have been supremely difficult (albeit not impossible since I’ve succeeded at most). Sometimes to solve these problems it’s helpful to think beyond your own borders and prejudices. To be immersed in another culture is to give yourself a nudge in that direction. To remain homogeneous is to never grow.

Why Upgrades Aren’t Always Easy – The Horrors of Components

The software component market is great. If you’re trying to make an application, that needs a particular feature, you can often just buy a component that gives you that feature. In addition to buying a component you have other choices.

  1. Build it yourself
  2. Rely on the operating system or language support libraries
  3. Try and find an open source solution

#2 is OK if it’s there.

#3 is fine as long as it’s current and not abandonware.

#1 is usually only if you can afford the development or there is no other choice.

Buying or licensing a component is often the right solution and is often the cheapest thing short and long term because your time is spent integrating a technology not learning it, implementing it, debugging it and integrating it.

The problem that unfortunately comes up is the “can’t go forward, can’t go back” issue. I’ll give you an example of this. This did not actually happen – what actually happened is far worse, but it’s close enough.

At my last job, I created a general PDF library for manipulation and generation of PDF documents. One use of this was that I created an add-on to our OCR product line that let you create PDF documents with each page containing the original scanned image as well as the recognized text which was invisible so when it was highlighted, it made it look like the scanned text was selected. The quality of the output was typically better than that those bundled with OCR engines and it cost less and worked with all engines.

We had several customers who used it and were happy. The initial version had a page limit due to a bug, so I fixed that a few years later. By that time, the original OCR engine we licensed and sold was no longer in our product line and we weren’t allowed to sell it anymore. It had even lapsed from our nightly builds.

And a customer called up and needed more pages than we could offer. The problem was that the new PDF code, modular though it was, couldn’t just be put into the old version. It wasn’t compatible because of OS/language component changes. I couldn’t go backwards. We offered the customer a different engine running on our current version. Nope – because of other component dependencies, they couldn’t go forward. Great.

Because we were a customer-focused company, we ended up doing was figuring out how to get the new PDF code built in a way that it would work with their build, as is. It was a pain in the ass and fortunately, it was just a “one off” so the customer was happy.

Now consider all the machines running custom software with Windows XP that were recently affected by a worm that exploits a security flaw. In fact, I saw a system that runs image analysis on 3D dental x-ray pictures two weeks ago at my dentist’s office that still runs XP. Why? Almost assuredly it’s due to a “can’t go forward, can’t go back” problem similar to what I outlined above.

Why do these problems happen? Sometimes it’s because a component provider no longer supports an older product on a newer operating system. Sometimes it’s because a component provider has gone out of business. Sometimes it’s because a component depended on a bug in the operating system to function. Sometimes it’s because it costs too much to upgrade.

How can we minimize this kind of problem?

  1. Select component suppliers with solid support and a contract that spells out how long the component will be kept up to date and a good bug-fix policy.
  2. Select only components that are supported for the project lifespan of your product.
  3. When budgeting a project, require that there is a line item for operating system/software upgrades including field service.
  4. Consider a requirement that all source and build systems go into escrow.
  5. Include a system replacement clause in the event that a “can’t go forward, can’t go back” problem comes up before the end of the contract terms with training if need be.
  6. If you’re building the component, for the love of all things holy, don’t rely on OS bugs or undocumented features.

And if you are buying such a system, make sure that your vendor can do all these things. If you go with a low-ball contract that doesn’t include these things, then you are risking a much higher cost in the future – either for new hardware or for new software or both. If you can’t plan to pay for that from the beginning, how will you pay for it in a crisis?

I’m Old, Part XLIX: The Stanford Invasion

As I mentioned before, after Acrobat 1.0, there was a bit of shake up in the planning stages of Acrobat 2.0. There were plans afoot to get a UNIX version (but which one?), would DOS be a supported platform in the future, how would full-text search fit in, what about that whole OCR team anyway and so on.

I finally got to move onto the Search team full time. At this point, I was way behind the Windows side and had a huge amount of work to do to catch up. This meant that I was no longer working on the Mac viewer anymore. Mike Pell was doing several jobs (release engineering/installers, UX work, etc). So management brought in two young men from Stanford – Gordon Dow for the Mac Platform and Alan Padgett for the Windows platform. That they came from Stanford should not have been a surprise. There were a number of people on the team with a Stanford education (CS or otherwise), and there was always a lot of talk about Big Game. I neither went to Stanford not have ever had a particular interest in football, so that who thing was totally lost on me.

Both Gordon and Alan were monster coders with all the benefits of youthful energy and a fresh college education where C++ was the new hotness. When I left college a few years earlier, C++ was an idea that only existed in a bizarre compiler called cfront which turned C++ code (slowly) into C code. Microsoft had adopted C++, but there were still no full implementations on the Mac. We had used Think C for Acrobat 1.0, but it was fairly limited in its scope.

Gordon and Alan did a lot of the grunt work. For example, I had thunking mechanism that was friendly to C and Pascal calling conventions and hoisted it over to Gordon who refined it for use in Acrobat as well as later figuring out what to do when a C call into a C++ callback threw an exception and other mind-bending puzzles.

Gordon and Alan were an interesting pair – really kind of an odd couple. Gordon was very intellectual, introverted and reserved. Alan was smart, extroverted, and very outgoing. At various points, they shared work space. Gordon put on headphones because Alan programmed with sound effects. No seriously. It seemed like when he finished a particular phrase, it would be punctuated with something like “Shuh-keuh!”

Also brought onto the team was Steve Herskovitz as team lead, an experienced Mac engineer who had escaped from the ATM team. In the 2.0 product, a lot of the code was rewritten. My built-in search code, for example, got chucked. It was built for speed and light memory usage, but Moore’s law had eclipsed it entirely and instead Acrobat was going to have Daryoush Paknad’s Wordy algorithm (used for full-text indexing) instead. At one point, Steve called me into his office to go over some behavior that was in 1.0 and how it was in 2.0. He was showing me a highlight of some non-linear text and it looked awful to me compared to 1.0. I know why, too – I had sweated a lot to get the details of text highlighting correct and I took it as a personal affront to see what was happening to the product. I was being passive aggressive to Steve and he lost his temper – my fault entirely. He snapped me back to reality with one curt question: “are you going to talk or are you going to listen?” I reigned my behavior in and then tried to explain what I had done, how I did it and why and then left him with the task of emulating it as best he could. Not my problem anymore, but I learned a bit about my own pride in the process.

We had a small kitchen in our wing with a fridge stocked with soda, coffee and so on. Steve frequently brought in a grapefruit on which he wrote an abbreviation of his name: “Hersker”. One seeing one, I imagined that Steve was an agranthrope – a creature who turned into a fruit/vegetable during a full moon. I told him as much, but like many of my jokes, I think it fell on deaf ears.

It was hard seeing so much of my work getting thrown out wholesale, but most of it was for the best for the new architecture of 2.0. Besides, I was too busy trying get the Mac search engine done to spend a lot of time crying over those changes.

I’m Old, Part XLVIII: Leprosy

I spent a great deal of my youth hacking on the Apple II. I wrote games that used the low resolution graphics first and then later hires. Applesoft BASIC came with a brief vocabulary of commands for drawing in HIRES. You could plot points, draw lines, and there was also a very simple vector-based shapes that you could define. I had some trouble understanding the awful manuals, but my dad spent some time explaining it to me and then my brother Pat wrote a shape maker program that would let you draw the shapes out and get the data for you.

I wrote space games for the most part. Most of the games we had were space games and I took my inspiration from them (Jupiter Express by Gary Shannon, Star Cruiser by Nasir Gebelli, Rocket Pilot and Star Wars by Bob Bishop, Tranquility Base by Bill Budge). I found writing games in BASIC frustrating. The shape tables drawing routines were quick, but if you tried to do any kind of animation, the game flickered badly from the erasing and redrawing. Plus there were two shape drawing commands – DRAW which used paint drawing and XDRAW which used Exclusive-OR (XOR) drawing. For XDRAW you could draw and erase since two identical XDRAW commands are an identity operation. In this way, you could get cursors, for example, that moved over existing drawing on the screen without damaging it.

Interesting side note – the XOR drawing process was patented in 1980, but it was already in use on the Apple II at that time.

The problem with XOR drawing is that if you use XOR to move things around, you have have to be super careful if you’re using other drawing because you’ll end up with unexpected junk left behind. I figured this out early and gave up on it. I moved onto to using bit mapped shapes and custom blitters to draw them on the screen. You could use XOR here too, but for the most part I used a direct write – less to do per pixel and faster as a result.

One day after school, I biked to Summit to Stonehenge Computing to see what was new and to show off hacks. When I got there, there was another guy trying to get some help on his program. He had gotten a book on Apple graphics and was working his way through the examples and tried extending them. He had a program that made a stork walk across the screen from right to left while animating the legs. “I need some help with my code – my bird has leprosy.” I stepped in, “are you using XDRAW for the body and HPLOT for the legs?” “Yes” “OK – you can’t do that if they overlap, it will leave bits behind where they overlap. You need to use DRAW for the body and change the color to erase and draw.”

And this is how I met Jim Gittelson. Jim was a few years older than me and went to a private school. We started hacking together. We learned how to crack software protection on published games. For the most part, I dug though code to figure out where the game lived and what did what so we could end up with a minimal set of hunks that could be saved out and loaded back in again. Jim liked to figure out the track patterns to get Locksmith or Nibbles Away to copy them. Later, Jim started making dedicated hardware to help the tasks. For example, he built a peripheral card which had some ROM and ton of RAM on it. When you pushed a magic button, it took over the machine and copied chunks of system memory to his card. Then he reset the machine and wrote the memory out to disk. In essence, he was creating a snapshot of memory which could be loaded back. Clever.

At one point, we discovered RLE encoding and wrote some code the compress a ton of HIRES images onto disk (far more than could fit uncompressed) and then knocked together some code that would run in a loop loading each file in using RWTS (Read/Write Track/Sector) routines to pull in the data and decompress it live into the screen, along with code that was playing two voice music. It was a glorious hack.

Jim and I also were fans of the arcade game Defender. We made routine trips to local arcades to find the games and spend every quarter on them. Well, not quite. Jim collected Bicentennial quarters and always set those aside. The last time I contacted Jim, he had purchased a used Defender machine and had modified it so that it had a pause switch, that way he could start a game, pause it and come back to it later. Nice. I asked him if he still collected Bicentennial quarters. Yup. Still.

I still think it’s funny that a long friendship started because his bird had leprosy.

I’m Old, Part XLVII: Customer Support

When I started at Atalasoft, I was employee number 7. The company was engineer heavy for a tiny place, but that’s to be expected. We had Glenn, Dave, Seon-Young, Bill, and me all writing code. Bill was the CEO and routinely wore many hats. We had a goal of taking his compiler away from him and eventually succeeded.

One thing we didn’t have was support. Dave and Glenn did most of the support since they knew the products better than me. I mostly consulted.

Our sales grew and we were able to move into a new office. It was necessary; our office was packed to capacity – in the meantime, we had hired Sean and Lou into engineering and John into sales. It was also necessary that we had to take a more methodical approach to support. I wanted full-time support, but we weren’t flush with funds yet, so Lou put into place a process where we had engineers rotating into support. For each rotation, the engineer would answer the phones, take support cases from our Salesforce queue and also hit our forums.

Glenn had a rotation when he encountered a customer who not only he couldn’t please, the customer went as far as to royally piss of Glenn. This was a tremendously hard thing to do. Glenn was a quiet, reserved Texas transplant who was about as even keeled as you can imagine. I had never seen him pissed off before. He had put the customer on hold and explained to me just how awful this man was. I’m going to call him Jim Jones. Jim managed to rattle Glenn and Jim was still angry and demanded satisfaction. I had Glenn transfer Jim to me and I let him blow off some steam. I did what I could do to help him and he spent some time complaining about Glenn. After he was done, I promised him that he would never have to work with Glenn again. Ever.

After he hung up, I told Glenn that he would never have to talk to Jim ever again and that if he ever called in and showed up on caller ID, feel free to forward his call to me or to whoever else was available. I made sure that everyone else in the office knew the rule: if Jim Jones calls in, under no circumstances should Glenn have to speak with him.

We were talking about it a few weeks later and no sooner had we mentioned his name then Jim Jones called in again. It’s as if when we spoke his name, his ears started burning and he called in. We kept Glenn away from that call. I think the same thing happened again: we mentioned his name and he called in. It was like Voldemo…he who shall not be named. At that point, I started referring to him as Tim Tones or Slim Slones or Bim Bones or some other obvious alliterative variation on his name, just to avoid having him call in again.

From then on, whenever we got a new hire for any title, we made it clear that if Trim Trones called in, under no circumstances would Glenn speak to him.

Now, from Pim Pones’ point of view, I met his desire of not speaking to Glenn, but I couldn’t care less. I cared far more about Glenn. That I could meet both of their desires in one fell swoop was extra awesomesauce. The best part of it was that Thrim Thrones had absolutely no idea how much I was favoring Glenn here and as far as I know, he never will. Another satisfied customer.