Analyze and Report Phase
The fifth and final phase of a wargame project is to analyze the collected data and related material and using this to produce reports of results. As indicated in the diagram this consists of two steps.
Step 14: Reviewing and Processing Data
The data that analysts may use comes from many sources. The three links below provide a few examples of how data might be collected. These examples are not exhaustive, and some will be inappropriate for certain types of gaming. Nevertheless they are offered as a launching pad for a new study.
- Numbers from sources where quantification is appropriate
- Feedback from players, and from others
- Records of events and other activity from the conduct of the game
• Real-time Review. Reviewing data should be conducted in parallel with conducting the game. This level of review need not be comprehenive, however it is critical to impose quality control while the data is being collected.
• Data Quality Control. This real-time quality control, applied while the game is underway. The purpose is to ensure that the data being collected will be suitable for the required analysis. Some discrepacies in data collection that may be diagnosed from real-time review include:
- observers were out of position to collect data at the time and location required
- written comments or observations are unreadable or use confusing or imprecise language
- computer-based systems malfunction (hardware or software problem)
- collection procedures were inappropriate or inadequate, e.g., computer systems were overwhelmed by volume or pace of responses
- data is inconsistent with expectations (due to bad procedures, software failure, etc.)
- equipment fails (e.g., audio or video recording)
The data-collection malfunctions listed above can happen, and the list is not exhaustive. The sooner malfunctions are discovered, the sooner rectification can start. Because war gaming involves considerable resources, it is critical to take corrective action while data collection opportunities are still available (while the game is in progess).
• Processing Data. Given the diversity of data that may be collected in a wargame project, it would be a challenge to provide procedures that would be universally applicable. Some games will depend largely on a gist of the discussions, possibly supported by interviews with participants. Other games will be more dependent on numerical data, perhaps casualty numbers from a combat simulation or message loads on a command and control system or a communications network.
• Emerging Insights. As data review proceeds the analysis group should scan for emerging insights. As observations accumulate, the group may feel they have preliminary insights. As the game continues, confidence may grow that some preliminary insights have been confirmed, but these should be shared so they are critiqued by participants.
Preliminary Insight – The synthesis of a set of observations that reveal a capability or a warfighting impact. Insights include new thoughts or patterns that emerge as an analysis team looks at observations and reviews them in light of a larger body of knowledge within an operational context.
Emerging Insight – The evolving insights that are produced on a cyclical basis during the conduct of the game. They are intended to capture in real time, what the analysis group is learning and are used to periodically update the other participants. The emerging insights may be withheld from game players during play to avoid biasing their decision making.
Initial Insights Report – An evolving document that should grow as events are conducted during the game. The intent of this report is to compile one comprehensive document that ties all insights together from all of the events or phases of the game. The report may have separate sections for preliminary and emerging insights since the quality threshold for the evidence from the game for preliminary insights may be lower than for emerging insights. Where insights are controversial, this should be identified and supporting evidence should be provided for contrarian points of view. This report is generally produced within 30 days of the end of a game, but sooner for a smaller game.
Final Analytical Report – A detailed analytical report containing the final insights and supporting evidence from the game, based on post-event analysis and synthesis of observations, instrumented data, surveys, and interviews obtained throughout the game in order to produce the insights. The final OA Report is generally produced within six months of the end of a large and complex game, but sooner for simpler games.
• Preparation for the Quick-look Report. Providing a quick-look report is covered in Step 13, but collecting the necessary data and other evidence is a task for the analysis team (and hence covered in Step 14). The Quick Look Report to the players (and other participants) should include emerging insights and, if time permits, preliminary insights. Feedback collected during the Quick Look should be added to evidence already collected. The feedback may include material that confirms or contradicts an insight. Particularly when there is feedback that contradicts an insight, it should be supported by evidence. Even if the evidence is only unsupported opinion from players it should be collected by the analysis group and balanced against the material already collected.
Step 15: Reporting Results
A Candidate Table of Contents
Most analysis organizations will have report format suitable for the results of a war game. If this is lacking the outline below can be considered, with parts that are obviously inappropriate discarded.
Chapter 1 Introduction
- Report Organization
- Chapter 2 Literature Review
- Relevant Past Findings
- Parallel Activity
- Data Sources
- Chapter 3 Exercise Framework
- Organizational Design
- Key Systems
- Schedule of Events
- Chapter 4 Analysis Framework
- Methods, Models, and Tools
- Constraints, Limitations, and Assumptions
- Analysis Products
- Chapter 5 Analysis Results
- Supported Insights, with summary of evidence
- Nascent Insights, more study required
- Unsupported Assertions, with supporting and contrary evidence
- Chapter 6 Summary
- Participants or Participating Organizations
- The Scenario and Vignettes
- Backgrounds of Roles and Organizations Represented in the Scenario
- Orders of Battle
- Descriptions of Weapon, Sensor, and Communication Systems and of Platforms
- Detailed Data (supporting Chapter 5)
• Data Visualization. Much of the analytical data and evidence will be quantitative. Much of this material will be suitable for visual presentation, e.g., graphs and charts.
Data visualization has received considerable attention in recent years. Several authors have provided very good references for improving skills in this area. The references are not specific to wargame results but are excellent for advice and examples:
• PowerPoint and Printed Reports. Presentation of preliminary and emerging insights on a screen (e.g., using PowerPoint) may be suitable for the Quick Look Report. Viewers will have experienced the game, so will have contextual knowledge to interpret such findings appropriately. For other types of reports PowerPoint style is unsuitable. Generally, this style has a considerable lack of contextual material and readers will make assumptions (sometimes wrongly) above the meaning of headlines and partial sentences. A well-written printed report should always be included in a game project.
• Web Pages and Wikis.