Jay Paroline

When testing your work, it’s essential to always test it in the context that it’s going to be used in.

Last weekend, I helped a friend move her stuff down to the Orlando area. UHaul trailers are incredibly cheap compared to trucks, and my car has a hitch, so we got one of those, loaded up her stuff and headed south. Somewhere along the way, the hookup for the lights became disconnected and then dragged along the ground, getting all nice and melted in the process. I noticed this when we got to her new apartment and called UHaul. Awesomely, they have free roadside assistance for any time something happens to their trailer.

They sent out a guy, he looked at the hookup on the trailer, rewired it, and tested it with his wire testing doohickey before taking off. We had already disconnected the trailer from the car for parking purposes, and when he pulled in to fix the trailer he blocked access to my car, so we didn’t actually hook it up to my car to test it.

A few hours later, it’s time to go home and return the trailer and oh, crap, the lights don’t work. I had to call UHaul and have them come fix it again. Because of the way the trailer was hooked up originally, the tongue of the trailer managed to get on top of one of the wires on my side of the connector and eventually wore through it, something I hadn’t noticed previously. It also managed to blow a fuse at some point so my brake lights didn’t work at all even when the wiring was fixed. Fortunately, the UHaul guy was very friendly and also had all the parts one could ever need for fixing minor car problems, so he easily re-rewired the connections and fixed my fuse.

Still, a few extra minutes of checking the trailer in the context it was going to be used in (i.e. attached to my car) would have revealed the problems and saved UHaul several hours worth of labor (driving to and from the apartment complex; they were far away).

That was a non-technical example of the principle, but there are certainly plenty of technical ones as well:
When I wrote the code to handle our PayPal processing, PayPal helpfully provided a development sandbox for testing with. All my code was thoroughly tested, seemed to be handling every sort of error that I threw at it, processing successful charges properly and everything. Then when it was time to release we pointed it at the ‘real’ PayPal servers and suddenly it didn’t work anymore. A few (real) payments later and some minor differences between how the real servers worked and how the dev servers worked were revealed. No big deal, but again, some testing up front in the correct context would have prevented the issue from ever appearing in the first place.

I’m back from my mini-vacation and it sounds like things are crazy at the office, but I think even so that I should have time to start writing again and I look forward to doing so.

My first order of business tonight is to start reading Systemantics: How Systems Work and Especially How They Fail by John Gall which was recommended by Raymond Chen and therefore automatically must be worth reading. Because I’m sure many of you are too lazy to actually follow that link, here’s the relevant RChen review:

Systemantics is very much like The Mythical Man-Month, but with a lot more attitude. The most important lessons I learned are a reinterpretation of Le Chatelier’s Principle for complex systems (“Every complex system resists its proper functioning”) and the Fundamental Failure-Mode Theorem (“Every complex system is operating in an error mode”).

You’ve all experienced the Fundamental Failure-Mode Theorem: You’re investigating a problem and along the way you find some function that never worked. A cache has a bug that results in cache misses when there should be hits. A request for an object that should be there somehow always fails. And yet the system still worked in spite of these errors. Eventually you trace the problem to a recent change that exposed all of the other bugs. Those bugs were always there, but the system kept on working because there was enough redundancy that one component was able to compensate for the failure of another component. Sometimes this chain of errors and compensation continues for several cycles, until finally the last protective layer fails and the underlying errors are exposed.

That’s why I’m skeptical of people who look at some catastrophic failure of a complex system and say, “Wow, the odds of this happening are astronomical. Five different safety systems had to fail simultaneously!” What they don’t realize is that one or two of those systems are failing all the time, and it’s up to the other three systems to prevent the failure from turning into a disaster. You never see a news story that says “A gas refinery did not explode today because simultaneous failures in the first, second, fourth, and fifth safety systems did not lead to a disaster thanks to a correctly-functioning third system.” The role of the failure and the savior may change over time, until eventually all of the systems choose to have a bad day all on the same day, and something goes boom.

Time to hit the books!