Toyota can’t say for sure whether the recent rounds of recalls will definitively resolve the safety issues. What a mess they are in! But we shouldn’t be surprised – automotive systems are so complicated that it is frankly surprising we haven’t had more problems. I wouldn’t be at all surprised that the harsh light shining on Toyota today will be focussed tomorrow on Ford, or GM, or Hyundai, or even BMW. (Just to be clear – I am not privy to any information about Toyota’s operations or those of any other auto manufacturer – these are just my opinions). Embedded systems are the future because you can do so much with them. They swivel headlights in the direction that the car is turning. They speed up or slow down the windshield wipers to match the amount of precipitation. They detect slippage and adjust braking. They keep trailers from fishtailing. They turn off the car if you drink too much alcohol. And they do all these things in all kinds of conditions, year after year. And this is just automotive systems – airplane software is so extensive and interdependent that it will soon be common practice to do complete system dumps each time the plane lands, followed by a full systems analysis and safety check.
It’s the long-term integrity of the systems that is ultimately the cause of many problems with embedded systems. The systems are exposed to water, salt, grime, grease and dust. They are supposed to work just as well crossing the Mojave desert at 110 degrees as they do driving on a frozen lake in northern Ontario. This is for the same car, too: Since you can very easily drive your car from Ontario to California, the car manufacturers don’t catch a break by adjusting the software to target the location where the car is sold.
So what does this have to do with information architecture? Quite a bit, actually. Information comes gushing with firehose intensity into the head office of every modern manufacturer every day of the week, and little bits of it contain the seeds of your future destruction if you are unlucky and not careful. How do you make sense of this information? Well you can’t just dump it into a big information hopper and hope that the right person will type in the right search terms, say, “sticky gas pedal”, and magically infer that a disaster is looming. You must be watching for trends, so that when you finally see one emerging you can be well ahead of the blogosphere and the politicians, giving you time to quietly and efficiently fix the problem.
How do you proactively capture the information you need to avert disaster? You need people to capture defect reports. They need to be trained to spot apparent duplicates – when they find one they must merge the new information with the original defect. This “augmented” defect will eventually have enough anecdotal evidence attached to it that your engineers will be able to formulate theories and start testing them. The system must support the ability to link defects to each other in cases where there may be a causal relationship. Users must be able to trace from defects to designs, to specifications, to test cases, and to code. They must have the ability to predict defect rates by correlating defect trends with sales data and spec sheets.
The irony is that individually all these requirements seem to be easy to satisfy. You need a defect database and customer service reps – no problem. You need a database for your designs, spec sheets, and test cases, that can be linked to your defects – doable. You need trend analysis software – well, not so easy, but it exists in a variety of flavors. A friend of mine works at a company called Ivara in Burlington, Ontario. They help companies predict how long their manufacturing equipment will last, and help you devise a maintenance program that minimizes your fully amortized operating costs. Complicated stuff, sure, but useful in the right hands. Just buy the software, train up some techs, and get on with your business, right?
Well, not quite. The tools are often disconnected from each other and hard to use. Individually they may work all right, but the UI and reporting interfaces suffer when the data are stored in silos. Perhaps the biggest obstacle is a lack of process. Workflow helps each employee know what to do when incorporating information into the corporate repository. The defect report must be reviewed. If it seems serious enough it should get investigated more thoroughly to include a technical perspective, some trend analysis and a business and legal perspective. If the problem is bad enough it gets put into a fix pipeline, including perhaps a recall. At each stage the defect can be rejected for lack of sufficient detail or seriousness, but it should always remain in the system in case new information comes along. When you follow process the firehose of information is working for you – people are picking the right information out of the torrent, putting it into the right place, and notifying others when they need to look at something.
You need a way to coordinate activities when numerous applications are required to span the information lifecycle from initial defect to mass recall. Orchestration platforms like BPEL can be employed to enforce the workflow, though having just completed a BPEL project of my own I can assure you it’s no trivial matter. Some tool vendors put together end-to-end solutions through tight integrations of various tools, while others use a single engine to enforce the process from start to finish. Much progress has been made in these tools, but uptake is low in many industries. The companies that really understand the need to coordinate information acquisition are embedded systems manufacturers. I believe this is because:
- they sell millions of units (so statistical analysis is feasible);
- if the product fails people die (so the overhead is worth it to avoid lawsuits and damage to the company’s reputation); and
- once the product leaves the showroom it may be a long time before you get a system dump, if ever. Thus all incoming information is potentially useful information.
Microsoft issues patches every single week. What a luxury! SAAS vendors can easily fix a problem in production – just upload a patch to the server. But embedded systems vendors know that if you don’t follow a process you may find your CEO on a plane to Warshington to have a cozy chat with some decidedly unfriendly lawmakers.