Modeling and Simulation

Over the past three weeks I have heard references to "computer modeling" used in references to the current financial mess, global warming and future automobile sales.

It got me thinking. In all the discussion regarding modeling I have not heard any discussion as to the validity of the various models. I may be the ONLY person that my readers have met who has actually been responsible for moving an M&S effort through the process used by the Department of Defense to qualify M&S.

In the Department of Defense there is a process used to qualify M&S and is called "Validation, Verification and Accreditation". And, each step along the way has a series of checks to ensure that the process isn't swayed by bias or a desired outcome.

A few years ago I was tasked with running the VV&A of a hybrid device which was both software and hardware and emulated the actions of an anti-ship cruise missile. The M&S was being developed as it was unsafe to fly a physical missile close enough to a populated ship to be safe.

The process began with a letter from the Navy's Commander, Operational Test and Evaluation Force (COTF) which validated a requirement for M&S citing ship safety. The letter provided direction which the M&S developer by providing about ten requirements of the Model.

My boss at the time, was a T&E manager in the Navy and he funded an organization to tailor an existing M&S suite to meet the requirements created by COTF. The organization required about 12 months to complete the M&S after which time the validation process began.

The M&S validation process was comprised of hundreds of simulation runs of the Model in a closed-loop lab and then to compare the Model's performance with that which had been observed by the missile in the real world. The validation process compared the actions of the Model against the real anti-ship cruise missile: essentially, it measured the level of goodness of the model. Upon completion of the validation process a Report was prepared for my boss which compared on a line-by-line bases the requirements provided by COTF against the results from the Model's validation tests.

The Model wasn't perfect, but if I recall correctly it was good enough on eight of the ten requirements and not good enough on two of the ten requirements. This resulted in an Accreditation by the T&E manager for the use of the Model in the areas where its performance closely replicated that of the real article. The critical take-aways were:

* The Navy had specific measurable requirements
* A M&S was developed to meet the measureable requirements
* The performance of the M&S was measured against the actual article
* Shortcomings of the Model were documented and limitations were clearly understood
* The cognizant Navy decision maker chose to accept the results which came from the Model in eight of the ten areas where requirements existed

So. Why I am giving this inconsequential discussion of the VV&A process used by the DoD? Well, it is to highlight how complex the M&S process is in the DoD. I deal with other engineers and physicists and the rules in which we operate are defined by the physical world around us.

I would ask my readers to pay close attention anytime the media hypes M&S which draws any conclusions or imagines a future. As complex as my M&S was, I was fortunate to have immutable laws of physics to bound my problem.

Bottom Line: M&S is hard work which requires the user to get his hands dirty to ensure that it accurately replicates the environment of interest. It would be very easy for M&S modelers with a lax process to insert bias into their Models should a desired political outcome be considered more important than revealing the truth.