Validation – Did we build the right thing?
The system’s been built, and shown to work as the designers intended. It’s almost time to release the system to waiting customers. However, the task of validating the system still remains. Validation checks the system’s function against the customer requirements, rather than the design requirements. Basically, validation means ensuring that the right system was designed, according to the customers.
Keep in mind that the term ‘customers’ encompasses a fairly large group. Customers are the folks who buy the system, but also various regulatory agencies and market forces or company traditions (think of the distinctive look of Apple products). These are the ‘regulatory’ and ‘internal’ customers, respectively. These customers’ needs must be addressed, and should be tested.
While validation is often not as technologically as difficult as verification, and doesn’t frequently need a test system, it does have other challenges. Customer requirements should be validated through customer input; often validation is carried out through customer interviews, or usage data at customer sites. Validation usually starts through beta testing. Beta testing usually involves sending customers who have a good working relationship with the company an complete or almost-complete system (a ‘beta’ unit) and collecting data on the customer’s usage and satisfaction with the instrument.
Collecting this data can be a logistical challenge, as regulatory bodies often require very specific tests to be executed during validation, and this means designing a test for the system. In addition, customers are busy people, and don’t always have time to add something else to their day, especially when they’re not getting a direct benefit from doing more work. Customers face a real ‘what’s in it for me’ problem (often called WiiFM), and so need to be convinced to do this one extra task.
Other customers don’t like being monitored, or don’t understand the purpose or data collection methods of a particular study, and so usage data is either incomplete or difficult to interpret. Interpretation of the data can be difficult, and so validation tests need to be designed so as to make data clearly interpretable and traceable to a customer requirement. Automating data collection as much as possible also helps reduce ambiguity. There will likely be some ambiguity, no matter how well-designed validation tests are.
To help both ease the customer relationship and help interpret the data, the customer relations team or marketing group will typically assist the technical staff in both designing and executing experiments. These folks act as the customer’s advocate on the team, to ensure that the customers’ needs are truly being met. Often, they’ll also act as ambassadors to customers, instructing any customers running validation, so that they understand the experiment and its purpose.
The word ‘validation’ often gets confused with verification, but it’s important to remember that there is a distinction, and that these are two very different kinds of testing. That said, they often get executed at similar times, and the next toy model will use that sense of simultaneity (to keep things to a manageable size).
Verification – Was it built right?
Apologies for the late post – I’ve been in the middle of moving, and the dust is finally starting to settle.
Now that the system is built, integrated, and optimized, we need to ensure that the system behaves as expected. This is known formally as verification, and the intent is to answer whether the system was built ‘right’. In this case, ‘right’ means that the system operates as intended, and was built as the designers wanted the system to look, ‘feel’, and behave. We can assume that, at this point, basic testing of the system design has been done to ensure compliance to specifications, but our verification testing is intended to go beyond basic, component-level functionality.
In order to ensure that the system operates as intended, we need to first define the conditions under which the system is intended to operate. These should be defined in the requirements document, either as a set of assumptions, or as explicit design requirements for the system. Operating conditions are often referred to as the ‘intended use’ of the system. Intended use provides context for the extent to which the system’s operation must be verified, but should not totally restrict the verification of the system unnecessarily. I’ll cover this in more detail in another post, but it’s important to remember that system usage should drive system verification.
With the system’s usage well-defined, requirements should be prioritized for verification. The requirements that should be verified first are the highest-priority requirements; that is, the requirements that would prevent the system from behaving or operating as its most valuable. Requirements that are considered ‘nonessential’ or ‘nice-to-have’ can be verified later than the essential requirements, but value to the various customers (as in the inaugural post for this blog) should be considered before ordering verification testing activities.
Verification can be executed in many ways. Typically, one verifies system function by testing the system’s requirements and analyzing the data from these tests to show that the system fulfills the requirements. However, systems can be verified by documentation, inspection, or modeling behavior.
When testing systems, we often need to design and build verification test units. This implies a new system to be designed, with new system design requirements, and possibly a new set of verification tests. Ideally, such test systems are smaller and more easily defined than the system to be tested, in order to keep the design manageable. Often, ‘off-the-shelf’ solutions can be purchased for verification from outside companies, with certificates of verification or analysis. In many cases, these certificates of analysis are sufficient to ‘prove’ verification of these test systems, though the team verifying system need to understand the possible consequences of relying solely on the vendor’s analysis or verification data.
Naturally, verification can’t cover all scenarios in which a system will be used, and the tradeoff that any testing team needs to make is between completeness and reasonable timescales for testing. So, the next time you experience a temporary glitch in any system that you’re using, think about how likely it is that the manufacturer and seller needed to test the scenario in which you’re using the sytem.
Toy Model – System Integration
Now that you’ve designed the components of a truly great TrapezoidMobile (TM), it’s time to start thinking about how to put it all together into a single coherent unit. Clearly, the TM’s chassis will have spaces to hold the supercomputer, engine, seats, and, well, everything else. Once all of these pieces are put into the chassis, their functions need to be coordinated in order to work together in an effective fashion.
Once the engine, brakes, drive train, and controls are in the chassis, they may make the TM move, but its operation could probably be optimized more. You can fine-tune the engine, and adjust the strength of the brakes in order to increase the ability of the TM to stop quickly or slow down from high speeds. The size and ability of the engine to move the TM faster will change the design requirements of the braking subsystem. This is an example of an interface requirement – the functional interface between the braking system and engine requires both sides to be strengthened. I’m specifying that this is a functional interface, since there may not be a well-defined physical interface between the brakes and the engine (after all, the engine acts on the TM through the wheels and the brakes act on the tires, which is another interface). These interface requirements mean more fine-tuning and optimization of system design.
In my ‘theory’ post for this subject, however, I alluded to discovering new functions and ‘hacking’ your system during integration. Let’s say that, when fine-tuning the TM’s braking system, you start to notice that, under the right conditions, you can generate electric current from the braking system. The supercomputer on board has a cooling system, which increases its capacity for rapid calculations. The cooling system requires power to effectively transfer heat away from the processor.
Here, integration means increasing the TM’s computing power by generating electrical power from the brakes and using this power to improve the efficiency of the supercomputer’s cooling system. The limitation of the ability to both generate power and cool the supercomputer is another design requirement or specification placed on the interface between these systems. By braking, the Fearless Trapezoid can calculate the areas of hypercubes and spheres in different topological spaces.
There is a not-so-subtle analogy here – hybrid vehicles work very similarly to what I’ve described (the Prius’ ‘regenerative braking’ is essentially this concept). Doing this kind of experimentation and tuning allows a development team to discover new system functions and uses, as well as extend and strengthen pre-existing functions and uses of the system.
I was a little hard-pressed to come up with good ‘accidental’ system improvements and discoveries, but I’d love to hear from anyone reading – what’s your experience with integration? Did you find that different parts of a system made the system better, or more complete?
Integration – how it all comes together
The truth of design specifications is that a single engineer is usually responsible for a subsystem (for a while, an unofficial title of mine was “the valves guy” for a project I worked on). Each subsystem with an associated specification needs to be integrated in order to make a coherent whole. This phase of design is called system integration, and is one of my favorite phases of design.
When you integrate, you’re not just bolting a bunch of functional subsystems together. If you do that, you still may have a system that works, but it won’t be pretty.
Note the camera phone
Instead, the integration phase is where parts can be made to come together seamlessly, and where the interfaces become even better defined. A well-defined interface between a camera and a phone may build a few pixels and camera firmware into a phone (like virtually every mobile phone out there). Another interface exists between components of a PC. If you’re old (like I am) and recall external zip drives, you know how buggy those interfaces could sometimes be. You could lose a lot of data very quickly if you weren’t careful. So, you needed to have a reliable interface between the hard drive and the external zip drive components.
The point I’m trying to make is that a well-integrated system is more than the sum of its parts. Often, the very act of integrating disparate elements creates new problems and innovative new solutions. Integration is the phase where you get to define functional interfaces, learn subsystem and component boundaries, and generally push the system’s limitations to learn what they truly are. It’s where you get to hack what you’ve just designed and built to add functionality, or increase value to the customer in new and exciting ways. You also often get to find new applications and capabilities of your existing system.
The Sony Walkman is a good example of well-executed system integration. Cassette players and portable radios both existed, but were not integrated, or made personal to quite the same extent. The Walkman gave everyone personal, on-demand music in their pocket. It also shipped with quick rewind and fast-forward capability, as well as a built-in radio, in a variety of new colors. Naturally, it became one of the most popular personal electronics products of all time.
Now, I’ve implied that a system is often greater than the sum of its parts. This may imply that a system that is only the sum of its parts is not an optimal system. That’s not necessarily true (see Wayne Wymore’s essay on this topic here - yes the title’s dumb, but the point is good), although it’s frequently true. I’ll talk about optimization strategies in a future post, but suffice it to say, optimization and integration often go hand-in-hand.
Now, I’m opening the floor to stories. What systems have you integrated and optimized lately?
Toy Model – Specification Setting
Let’s go back to The Fearless Trapezoid and the TrapezoidMobile for an example of design specifications. I’m not going to get too deep into the design specs (I’m not someone who knows how to design vehicles), but I’d like to provide a short example, given the previous discussion on the TM’s design requirements.
With the speed requirements of the TM, you can start to design a vehicle. In order to reach a higher speed, you usually need a bigger engine. You can start to design the specs for the size, RPMs, number of cylinders, and actual shape of the engine. You can design the shape and length of the TM’s body, as well as the size and shape of the tires.
The problems, as I alluded to, lie in the interfaces. A fast engine may cause TFT to spill the super math-juice used to power a fight against crime and bad geometry solutions. So, you need to trade off the aspects of design. I’ll get to tradeoff studies in another post (there is a quantitative way to do them), but for now, we just need to know that an improvement in one requirement may mean a degradation in another requirement.
The heavy engine and requirement for speed may mean that some hardware for the supercomputer needs to be left out, and that the supercomputer isn’t as super as it could be. This is where the interface may cause some problems, as you either have to trade engine size off for computer size, or computer size for engine size.
And I’m going to wrap it up before long. As you can tell, us systems engineers are not specialists in design specifications. This is where the really detailed engineering work comes in, but we are there to guide the process. I’m interested in hearing about interface issues, though. Anyone run into interesting interface specification mismatches out there? Comment below!
How it’s made – Design Specifications
With the requirements set and agreed upon (mostly), it’s time to actually start designing. I like to think of the product or design requirements as the “whats” and “how wells” of a system, and the design specifications as the “hows” of the system. There are many more exact definitions, but I think that conceptually captures it. Design specifications (I’ll be hereafter shortening this to ‘specs’) can take a wide variety of forms, but should have a description of how the particular component of the system works, and some numerical values on the operation of the system. For example, writing a spec for a bicycle wheel should include the average width and diameter of the wheel, the ideal tire pressure, whether the tube and the tire are separate or one integrated unit, how many spokes the wheel should have, and so on.
Specs are best left to the specialty (electrical, mechanical, chemical, software, etc.) engineers, but should be reviewed and guided by the systems engineer. The SE particularly needs to review operating spec range (the spec range or the spec envelope), and ensure that this operating range is achievable, reasonable, and repeatable. SEs typically need to ensure that all system spec ranges do not likely produce what’s called a ‘tolerance stackup’ problem. Tolerance stackup is a term borrowed from mechanical engineering, wherein pieces of a mechanical component will not operate properly, even if all are within their defined spec range.
To illustrate this, imagine you’ve designed a jar, with a lid. The jar’s mouth’s diameter can be a certain size, say, half an inch (0.5″), with maybe a tenth of an inch allowed of ’wiggle room’. In other words, the jar’s mouth can be anywhere from 0.4″ to 0.6″ in diameter. In this case, the 0.5″ diameter can be called the target spec, with 0.1″ of tolerance ‘on either side’. Now, say the inner diameter of the jar’s lid (the part that’s going to screw onto the jar) is designed to be 0.6″, with 0.15″ allowed on either side. The target sizes will allow the lid to screw very tightly onto the jar. If the jar’s diameter is slightly greater than the target spec, but the lid’s inner diameter is slightly smaller than the target spec, the lid may not fit onto the jar. Both pieces are within their particular spec envelopes, but, if the jar diameter is 0.6″ and the jar lid is 0.45″, they won’t fit together.
Most specialty engineers are not concerned with interfaces above and beyond their parts of the system. It’s up to the systems engineer to understand these interfaces, and determine how tolerances (or, more generally, the ‘error budget’) will cause the system to behave. In this post, I’ve largely described mechanical specs, because I think they’re the most concrete and therefore easiest to understand, but specs apply to anything quantifiable. For instance, the number of crashes of a software program can be a spec. Other specs can include things like volume (loudness) range of a stereo, brightness of a light, reproducibility of an action, and so on. Now that you’ve got some ideas of specs – what systems specs can you think of on a system? Are there any places where a system you use has broken down due to tolerance stackup?
OR in Engineering – The License Plate Game
Let’s take a break from our regular schedule. I wanted to talk about Operations Research (OR) and its use in systems engineering. OR is a field of research with a lot of components, but is primarily concerned with decision-making based on mathematical methods. Decisions try to fulfill a goal, and are subject to lots of rules and constraints. There is a strong tie-in with SE, since customer requirements provide a system’s goal, and design requirements provide other system constraints.
What makes OR hard is that very few (some would argue no) decisions in life are made with complete, perfect information. You never know in advance, for instance, whether putting on the red shirt or blue shirt is going to make a difference in your day (unless you work on the Enterprise – then always go with blue). To deal with this uncertainty, researchers and engineers deal with probabilities, which can be measured or, often, estimated from existing knowledge.
Of course, the first step decision-making under uncertainty is estimating probability of outcomes. That’s why I think Prof. MacLay’s posts about the license plate game are pretty neat. It’s a great example of how to estimate probability based on observation, applied to a pretty common experience. In science, estimating the probability of finding specific states’ license plates is a lot like estimating the early terms of the Drake Equation (specifically, the fp term of the equation).
Engineers get concerned with this kind of thing when there are rarely-occurring events that are expected during normal operation of the system. Something like, say, a hacker attack in a computer network. As a more appropriate analogy to my industry, let’s say you want to detect one or more rare genes because they’re cancer markers that indicate the type and potential treatment of a patient’s cancer. If the probability of the particular gene (or combination of genes) occurring in a given piece of tissue is rare, the product requirements should ensure that your detection system is sensitive enough to find it when it does occur, without a ‘false positive’ (unnecessary chemo or radiation therapy is terrible to a healthy person). The obvious tradeoff is determining the optimal sensitivity of your system.
To calculate that, you’d have to take into account the probability of finding your gene in patient tissue, the probability of your technology detecting that gene, and the probability of falsely determining a normal (noncancer) gene as a cancer gene. An OR solution may be able to optimize the detection requirement, given estimated (but accurate) probabilities of finding genes.
“Real” engineering isn’t this cut-and-dried, of course, and relies on a lot of assumptions for design. The license plate game model of design is a good, easy-to-understand model of other types of design.
Toy Model – Customer and Product Requirements
I apologize for the delay; I’ve been on a semi-vacation this week, and just got back. Anyway, I want to go back to the example of The Fearless Trapezoid (TFT) and designing a new TrapezoidMobile (TM) to discuss customer and product requirements in a more concrete example. For a review of this toy model, please go back and re-read this post. I’m including both types of requirements for requirements decomposition (something I’ll discuss later). We’ve determined that a new TM will add value to both society and TFT’s work, and now we need to determine exactly what form the new TM will take.
Since TFT is the only customer for our new system, we can sit down with our superhero and determine the super-needs of the user. Let’s say, through interviewing, we determine that TFT wants the new TM to have a supercomputer that will better determine answers to topology problems related to location of super-villains. TFT also wants the new TM to be fast, and be able to hold super-beverages. These 3 or so statements form our “Voice of the Customer”, and can be the basis for most of the explicit customer requirements. From here, we can say that a set of customer requirements based on TFT’s explicit input maybe:
- The new TM needs a supercomputer capable of finding enemies.
- The new TM should be very fast.
- The new TM needs to hold super-beverages.
A few assumptions about the TM are held in these requirements, however. For instance, ‘fast’ most likely refers to the ground speed of the TM. The capability of finding enemies implies that the TM has access to other technologies that can find enemies. And the ability to hold super-beverages implies that the TM must do so without spilling them, in a way TFT can actually drink these beverages. There are other implied requirements, of course, but I want to limit the length of this post. Implied customer requirements can lead us to some of our highest-level product design requirements – again, these won’t be a complete list, but will give a bit of a ‘flavor’ of product requirements. Indented requirements are at a lower level of abstraction (and closer to design specifications).
- The TM shall have a supercomputer.
- The TM’s supercomputer shall have a tracking system for TFT’s enemies.
- The TM’s supercomputer shall have a function for predicting future locations of enemies.
- The supercomputer shall be able to solve most graduate-school level mathematics problems.
- The TM shall travel rapidly to a destination.
- The TM shall have a maximum land speed of 200 miles per hour.
- The TM shall have a maximum air speed of 400 miles per hour.
- The TM shall have a maximum water speed of 200 miles per hour.
- The TM shall be able to stop within 5 feet from full speed travel.
- The TM shall hold super-beverages.
- The interior panels and upholstery of the TM shall be beverage-proof.
- The user shall have the ability to drink super-beverages at any time.
- Super-beverages on board shall include super-water, super-math-drink, and super-juice.
These are a (very) small snapshot of all possible requirements. I included an over-arching requirement, and its associated, decomposed requirements. These decomposed requirements should eventually drive down to design specifications (each could have several associated lower-level design requirements). The formality – including the word ‘shall’ and specificity – is required for making requirements binding and specific. The more specific, the better. I’m sure I’m missing several design requirements here – I’d love to hear a few more!
EDIT: Having a few problems with the CSS; I’ve un-bolded the decomposed requirements in order to clarify. I’ll try to fix this soon.
Driving down – product requirements
With customer requirements document, it now comes time to actually start designing something. Usually, first iterations of a design happen ‘on paper’ (more accurately, in a computer). Before putting virtual pencil to virtual paper, you have to start thinking about what your design actually needs to do, or look like. In short, you need to think about what you’re actually going to be designing, and plan it accordingly.
This typically takes the form of a set of product requirements, also called design requirements. Product requirements should be largely ‘design agnostic’, meaning that they don’t assume a final form of your product, and should have specific metrics associated with it, to the greatest extent possible. For example, when writing a set of product requirements for a paper-binding device (a stapler), one fundamental functional product requirement would be: “the device shall permanently affix two or more sheets of paper together”. This requirement is easily measurable, clear, and does not assume anything about the actual design of your product.
That said, being design agnostic in all of your requirements isn’t necessarily a good way to drive down to product specifications (the ‘hows’ of the design – specs will warrant another post). It quickly becomes necessary to introduce ‘levels’ of requirements. These are levels of complexity of the requirements themselves, starting from a very high level, with little description of the product, and going down to a low level, describing (for example) the color and shape of your product. The levels of complexity to be described should be agreed upon in advance by the design team. At a low enough level, your product requirements start to become product specifications, which describe the actual nuts-and-bolts design of the product. Defining specs is an activity best left to specialty engineers (again, this will be discussed in the future).
Design tradeoff decisions will need to be made during requirement definition, however. Just as in the customer requirements need to be prioritized (for more definition, see my previous post on customer requirements), product requirements also need to be prioritized. Priorities should also be agreed upon by an entire design team, if only to create a common set of rules for prioritization of design.
A lot can be said about writing good design requirements (in fact, much has been written about it), and there are some several guidelines. I’ll get more into those in a future post, but for now, it’s enough to make you, the readership, aware that product design requirements are the first stage of good product design.
A former mentor once told me that “without [product] requirements, you’re not even treading water – you’re just churning”. I think he was right about that – if you start designing something without thinking about what the design needs to do up front, you grow a system, rather than design one. In and of itself, this isn’t bad (cities are good examples of organic growth), but a lot of time and energy is eventually spent fixing issues that should have been identified in the first place.
The Big Idea
This is a post that’s probably destined to be permalinked from the FAQ section (or maybe from the “About” section) of the site. There is a plan for the blog.
You may have noticed that I’m picking a starting place and ‘drilling down’ in the design process. What I’m describing, over the course of the first dozen or so posts, is what’s usually called the ‘classic waterfall model’ of product design. Wikipedia has a description as it applies to software design. This isn’t the best model for product design, but it’s a good place to start, and at least is a good way to introduce concepts.
The waterfall model in pictures.
The intent of this blog is to talk about the systems development process and general engineering practices as they apply to everyday life. It’s interesting for me to think about how something like an airport, public transportation system, or bookstore works, from an engineer’s perspective. Using everyday examples, I’ll talk about specifics of each phase of the design process, and new tools to use. I don’t want to turn everybody into an engineer, just to get people to start thinking holistically about why the world is the way it is, and how things could possibly be improved.
Chances are, I’ll lean heavily on stuff that interests me – statistics and mathematical optimization under constraints; product design processes; design for ‘x’; things like that. I’ll explain new concepts as they arise, but I’m always soliciting feedback. This is why I need ‘customers’ (blog readers) – I can post about topics that interest me, but ideally, they should interest me and you. Please, comment if I’m starting to repeat myself too much, or if there’s something you’d like me to go on about. I’d like to make sure that this blog is more a conversation than a monologue – it’s my blog, but I want people to actually read it, so there’ll be some give-and-take here.
Anyway, that’s a ‘snapshot’ of what I intend to accomplish in this blog. Hopefully, we can work together on creating something interesting.