[Business] My Product Sucks. I’m Sorry. Please Buy More. Part 1

Handling severe product failures

Nearly every company will at some point experience a product or product-issue that’s an utter train-wreck in some fashion. I worked for a manufacturer a few years back, and I’ve been dealing with hardware and software issues in the IT realm for many more years than that. In light of Toyota’s recent catastrophic issue with their accelerators [http://money.cnn.com/2010/01/28/autos/toyota_tylenol.fortune/index.htm?hpt=Sbin], it seemed like an appropriate time to share some of what I’ve learned about handling business calamities.

In the Fortune article I linked to above, they refer to this gas pedal issue as Toyota’s “Tylenol Moment,” referring to Johnson & Johnson’s 1982 crisis when somebody was poisoning their product and people were dropping dead. Tylenol was the victim in that case, so it’s not a perfect comparison, but there’s a good lesson there and it meshes somewhat with my experience. Johnson & Johnson turned the crisis to their advantage by doing the right thing. They pulled all of their product, THEN they changed their manufacturing process to include tamper-proof packaging to ensure it could never happen again. It cost a fortune up front, but even if they never quite made that money back (which I suspect they probably did), they did manage to stay in business. Had they failed to win back the trust of their customers, there probably wouldn’t be a product called Tylenol on the shelves right now, and there might not even be a Johnson & Johnson. Toyota’s at a similar decision-point.

Let’s recap the Toyota situation. As early as 2007, there appear to be documents showing that Toyota was aware that their gas pedals could continue to accelerate when they weren’t being depressed or even during braking. It’s unclear whether they did anything to fix the problem at that point, but whatever they might have done, it doesn’t appear to have been very much. Then people started to die. Including one incident where an off-duty State Trooper and his family dial into 911 [http://www.youtube.com/watch?v=03m7fmnhO0I] as their Lexus is rocketing down the road at 120 mph. They were all killed in the crash, but it was a wake-up call to America that something big was wrong with Toyota vehicles.

Toyota’s response was lackluster, ranging from blaming driver error to the placement of floor mats. They even recalled the floor mats and installed a special bracket that was supposed to hold them in place better. Except that the problem now appears to have had nothing to do with floor mats, and (again, allegedly) Toyota damn well knew it. All of the facts aren’t in yet, but all indications are that the issue is in the electronics, not the mechanics. The gas pedal wasn’t caught on anything, or jammed, or even necessarily being depressed. It looks as if the computer controlling acceleration would simply get stuck on “accelerate” and would fail to release. The engine would continue to rev no matter what, and the brakes weren’t strong enough to stop it. The only solution was to remove the engine’s ability to drive the vehicle, such as by putting the transmission into neutral (which disengages the gears that connect the engine’s crankshaft to the rest of the drive train) and then by turning off the engine once it’s safe to do so. Toyota has since recalled some 3.8 million vehicles and ceased the manufacture of new models until the issue is resolved. And hopefully not just resolved with a useless floor mat bracket, but actually fixed.

I had a very similar experience when I was a VP for a manufacturer. Our products were printers, not cars, and there was no risk that anyone would get hurt. Our problem was also related to faulty parts – in this case, a combination of an LED emitter and a sensor. The sensor’s job was to report to the printer’s software whenever the light from the emitter was blocked, as this indicated that a slip of paper had been inserted into a slot and the printer needed to print something. Except that our printers were continually reporting that there was always paper in the slot, even when there clearly wasn’t.

My job as Service Operations VP was to represent the needs of the customer and the interests of my company in getting this issue amicably resolved. Unlike Toyota, we sold to other businesses, not customers, but in both cases reliability and reputation were key elements in preserving future sales opportunities and relationships.

When the problem was first reported to us, we, like Toyota, initially reacted somewhat defensively, assuming (and suggesting to the customer) that it was most likely user error of some sort. Instead of floor mat issues, we proposed that there was probably paper debris being left behind in the slot that needed to be removed. The users were supposed to be aware of this potential issue and had been provided directions on how to be sure the slot was completely free of obstructions. Stupid users!!

Well, then we got our hands on a few of the failed units, and there was definitely no paper in there. Oops! Sorry! Didn’t mean to call you stupid there. Wait, who said stupid? I didn’t say stupid? Did you say stupid? I don’t know. What to go grab some lunch or something?

But here’s where it started to get ugly. You see, our customer had recently begun a massive (massive!) upgrade in their stores all over the country, replacing our older printers with nice fancy new ones. And they’re out there just banging away with hired help all over the country installing new printers. Installing. Installing. Installing. Meanwhile, our engineers and quality-control guys are trying to figure out why this printer insists it has paper when it doesn’t. Oh, and here’s another one doing the same thing. And another one. And another. Uh-oh. It looks like all of the units installed last month (or, say, half of them) are having the same issue. And as they investigated further, it became clear that all of the ones being installed that day would start to fail in about a month.

You see, we’d been given bad parts. Bad parts from a reasonably good supplier. Bad parts that had never, ever been bad in this way before, as far as we’d ever seen. But, ah-ha, the supplier had recently moved production of these LEDs to a new factory. And, they’d borked the production process, bigtime. The LEDs (I think. It might have been the sensors, but I’m pretty sure it was the LEDs that were the problem) would lose power and quickly become so weak that they couldn’t send a signal across the gap to the sensor. But, this didn’t happen quickly enough to ever be seen in our quality-control testing. It took a month or so (or thereabouts – I don’t recall the specific number of hours. But it was definitely several several weeks) for the LED to fail completely, at which point the printer was unable to function until the circuit board containing the LED was fixed.

Now we’ve got a new product that actually works worse than our old product because it’s defective (whereas the old product, that was being removed from service, technically still worked fine). The customer is removing something that works and replacing it with something that, after a month or so, doesn’t work. And we have no easy fix for the problem. How we’d handle this disaster would have a huge impact on whether or not one of our best customers would continue to view us as a key business partner or as a hated supplier with whom they used to do business. Tomorrow, I’ll detail the lessons I learned managing this, my “Tylenol Moment” and the lessons it think it offers to other business-people across industries who need to come to grips with a product that’s threatening to go down in flames and take the company with it.