A friend recently shared with me a series of quotes complied by the statistician George Box in one of this books. While Box did not take credit for all the quotes, he must have felt that the collection of them was instructive enough for them act as truisms which provide insight to the world of statistics. I’m no statistician, but I find that the sentiment of many of these quotes are just as at home in the world of computer aided engineering. In CAE, we analyze data coming from simulation models and learn by building and tweaking these models to perform our own “experiments”. So I’d like to share how I see that these quotes fit nicely into the life cycle of the typical projects conducted by CAE engineers, and I hope everyone can smile and nod at the familiarity of shared experiences.
Project Initiation and Planning
“To find out what happens when you change something, it is necessary to change it.” I knew a manager who insisted that all reports coming from his group must contain at least two analysis for comparison. Then you could properly tell a story: “The critical feature on this part is this radius. Notice that it works in the proposed design and the part breaks when I change it.” A single analysis can never deliver assurance that the result is a reasonable prediction of reality and not a symptom of a flawed model.
“Only in exceptional circumstances do you need or should you attempt to answer all the questions with one experiment.” The previous quote made the case to anticipate the need to run multiple analysis. But assuming the sufficiency of a single analysis is just as naïve as believing you can plan all the required analysis in advance. Each project is unique and will contain its own set of twist and turns that must be addressed as they arise. The best plan is to start a project with a minimal analysis intended to uncover the subsequent questions.
“When running an experiment the safest assumption is that unless extraordinary precautions are taken, it will be run incorrectly”. This quote is a particular favorite of mine! Learning to mitigate mistakes is essential a keeping a project on track. Of course “extraordinary precautions” such as experience and meticulous attention to detail can, and should, minimize disruptions, but it is just as important to manage our frustration arising from the work of others (or yourself!) so the project maintains a positive outlook.
“The best time to plan an experiment is after you’ve done it” – R.A. Fisher. A droll quote, but so very true. It is important to learn from each project and refine your techniques for the next project. Of course no matter how much learning you put into your next plan, you will realize that too was not perfect.
“All models are wrong; some models are useful.” This famous and profound quote is attributed to Box himself. Too often we talk about numerical accuracy as an absolute, and we lost sight of the fact that these analyses are only numerical approximations of the real world. All models will contain errors – the question is whether the model’s inaccuracies are acceptable relative to the use case. In airplane design, there is typically a very coarse finite element model that is used to calculate the global behavior of the structure and understand how loads are transmitted. This model is perfectly useful in that context, but would be completely useless to get stress at a component level. A good model is only as accurate as needed.
“It is better to solve the right problem approximately than the wrong problem exactly.” To me this is a corollary of the previous quote. A simulation that reasonably predicts physical reality is the very basis of virtual prototyping. Running multiple analyses (the original quote above) on this predictive model can lead to amazing some engineering insight of complex systems.
Project Wrap Up
“The most exciting phrase to hear in science, the one that heralds the most discoveries is not “Eureka” but ‘Now that’s funny …’” –Isaac Asimov. When looking at your results, don’t be quick to dismiss inconvenient data. These unexpected results can often uncover a software bug, a flaw in the model, or even overturn a previous held belief. This is not to say you should expect to discover the next theory of relativity, but your data may well provide evidence to overturn the conventional logic on how your company designs its top selling widgets.
“There has never been a signal without noise”. After all the data has been collected, you still need to describe the outcome of your project to others. This is an underappreciated aspect of all projects but your efforts can be wasted if you can’t effectively communicate. Don’t let anyone walk out of the room without a clear understanding of your message. You should put your conclusions up front and don’t let your message (the signal) get lost in the minutia of unimportant details (the noise).
- Thought Leader Thursday: Philosophical Quotes on Life as a CAE Engineer - June 15, 2017
- Thought Leader Thursday: Seven Mistakes to Avoid in Getting a Fit - September 15, 2016
- Thought Leader Thursday: Design Exploration versus Design Optimization - April 21, 2016