Evaluation at the Highest Level
The ultimate measure of success of a wargame project is the satisfaction of the sponsor. However, there are a number of obstacles to using this directly to evaluate a wargame project:
- The sponsor may be hard to reach for feedback.
- The sponsor may be isolated from the results of a game project, relying on staff assessments or briefings (which may include biased reporting).
- The value or shortcomings of a wargame project may be clearly evident only after a passage of considerable time.
The sponsor's staff may be able to evaluate the results of a war game functioning as a surrogate for the actual sponsor. The staff's evaluation may be in the form of a specific response. Or it may be derived by observing if the staff use the wargame results in their planning and decision making.
Alternative Evaluation Mechanisms
Use of Best Practice
Following best practice, as recognized by the community, should raise the likelihood of a successful project. But this comes without a guarantee.
Data Collection and Management Plan
The Data Collection and Management Plan (DCMP) serves as a link between the sponsor's objectives and issues and the analysis product. Over the course of a wargame project, the sponsor's objectives and issues may change; so the DCMP should be reviewed periodically with the sponsor or with the sponsor's staff to ensure, where possible, that changes are accommodated. There may be circumstances where changes are not possible; these should be captured in Constraints, Limitations, and Assumptions (CLA).Constraints, Limitations, and Assumptions
The CLA may be used to set a benchmark for evaluation. As indicated elsewhere the CLA should never be used as an excuse for shoody practice. However, documenting the CLA provides the sponsor, the analysts, and the gamers a framework for discussing what the wargame should deliver and what it will not.
Once a game is in progress the DCMP provides a checklist against which to confirm that the necessary data is developing. If monitoring the DCMP determines that anticipated opportunities are being missing, this will be an indication that the game results may not meet expectations. In some cases deficiencies of this sort can be resolved by inserting new activity into the DCMP (and into the scenario). But it may be that not such resolution will be possible.
Short-term and Longer-term Evaluations
At the conclusion of a war game, participants may be polled for what went well and what did not. The participants can also be polled on whether they believe the results are valid, consistent with similar studies, and useful going forward. This can certainly be valuable. But views may be tainted by enthusiasm created from playing the game.
Consistency with previous studies should not be an overriding criterion for success of a war game. War games can be particularly valuable if they reveal new information or give unanticipated results. However, if wargame results are inconsistent with previous studies, the team should apply some diagnostics to determine the reason.
When the gaming organizations has an enduring relationship with the sponsoring organization, some periodic review may be in order. One of the main benefits of this is that repondents will have the context of several games for comparitive purposes. They may then point out techniques that worked better than others or components (e.g., scenario or analysis) that was particularly well done on one game. This sort of comparitive feedback can be particularly valuable.