adf logo

Australian Defence Force

Background for the Basic and Advanced Analytic Wargaming Courses from the Naval Postgraduate School in Canberra, 17 February to 6 March 2020

Analyze and Report Phase

The fifth and final phase of a wargame project is to analyze the collected data and related material and using this to produce reports of results. As indicated in the diagram this consists of two steps.

Step 14: Reviewing and Processing Data

The data that analysts may use comes from many sources. The three links below provide a few examples of how data might be collected. These examples are not exhaustive, and some will be inappropriate for certain types of gaming. Nevertheless they are offered as a launching pad for a new study.

  • Numbers from sources where quantification is appropriate
  • Feedback from players, and from others
  • Records of events and other activity from the conduct of the game

• Real-time Review. Reviewing data should be conducted in parallel with conducting the game. This level of review need not be comprehenive, however it is critical to impose quality control while the data is being collected.

• Data Quality Control. This real-time quality control, applied while the game is underway. The purpose is to ensure that the data being collected will be suitable for the required analysis. Some discrepacies in data collection that may be diagnosed from real-time review include:

  • observers were out of position to collect data at the time and location required
  • written comments or observations are unreadable or use confusing or imprecise language
  • computer-based systems malfunction (hardware or software problem)
  • collection procedures were inappropriate or inadequate, e.g., computer systems were overwhelmed by volume or pace of responses
  • data is inconsistent with expectations (due to bad procedures, software failure, etc.)
  • equipment fails (e.g., audio or video recording)

The data-collection malfunctions listed above can happen, and the list is not exhaustive. The sooner malfunctions are discovered, the sooner rectification can start. Because war gaming involves considerable resources, it is critical to take corrective action while data collection opportunities are still available (while the game is in progess).

• Processing Data. Given the diversity of data that may be collected in a wargame project, it would be a challenge to provide procedures that would be universally applicable. Some games will depend largely on a gist of the discussions, possibly supported by interviews with participants. Other games will be more dependent on numerical data, perhaps casualty numbers from a combat simulation or message loads on a command and control system or a communications network.

• Emerging Insights. As data review proceeds the analysis group should scan for emerging insights. As observations accumulate, the group may feel they have preliminary insights. As the game continues, confidence may grow that some preliminary insights have been confirmed, but these should be shared so they are critiqued by participants.

Preliminary Insight – The synthesis of a set of observations that reveal a capability or a warfighting impact. Insights include new thoughts or patterns that emerge as an analysis team looks at observations and reviews them in light of a larger body of knowledge within an operational context.

Emerging Insight – The evolving insights that are produced on a cyclical basis during the conduct of the game. They are intended to capture in real time, what the analysis group is learning and are used to periodically update the other participants. The emerging insights may be withheld from game players during play to avoid biasing their decision making.

Initial Insights Report – An evolving document that should grow as events are conducted during the game. The intent of this report is to compile one comprehensive document that ties all insights together from all of the events or phases of the game. The report may have separate sections for preliminary and emerging insights since the quality threshold for the evidence from the game for preliminary insights may be lower than for emerging insights. Where insights are controversial, this should be identified and supporting evidence should be provided for contrarian points of view. This report is generally produced within 30 days of the end of a game, but sooner for a smaller game.

Final Analytical Report – A detailed analytical report containing the final insights and supporting evidence from the game, based on post-event analysis and synthesis of observations, instrumented data, surveys, and interviews obtained throughout the game in order to produce the insights. The final OA Report is generally produced within six months of the end of a large and complex game, but sooner for simpler games.

• Preparation for the Quick-look Report. Providing a quick-look report is covered in Step 13, but collecting the necessary data and other evidence is a task for the analysis team (and hence covered in Step 14). The Quick Look Report to the players (and other participants) should include emerging insights and, if time permits, preliminary insights. Feedback collected during the Quick Look should be added to evidence already collected. The feedback may include material that confirms or contradicts an insight. Particularly when there is feedback that contradicts an insight, it should be supported by evidence. Even if the evidence is only unsupported opinion from players it should be collected by the analysis group and balanced against the material already collected.

Step 15: Reporting Results

A Candidate Table of Contents

Most analysis organizations will have report format suitable for the results of a war game. If this is lacking the outline below can be considered, with parts that are obviously inappropriate discarded.

    Chapter 1 Introduction
    • Purpose
    • References
    • Objectives
    • Scope
    • Intent
    • Report Organization
  • Chapter 2 Literature Review
    • Relevant Past Findings
    • Parallel Activity
    • Data Sources
  • Chapter 3 Exercise Framework
    • Characteristics
    • Architecture
    • Organizational Design
    • Key Systems
    • Schedule of Events
    • Training
  • Chapter 4 Analysis Framework
    • Methods, Models, and Tools
    • Constraints, Limitations, and Assumptions
    • Analysis Products
  • Chapter 5 Analysis Results
    • Supported Insights, with summary of evidence
    • Nascent Insights, more study required
    • Unsupported Assertions, with supporting and contrary evidence
  • Chapter 6 Summary
  • Annexes
    • Participants or Participating Organizations
    • The Scenario and Vignettes
    • Backgrounds of Roles and Organizations Represented in the Scenario
    • Orders of Battle
    • Descriptions of Weapon, Sensor, and Communication Systems and of Platforms
    • Detailed Data (supporting Chapter 5)

    Other Issues

    • Data Visualization. Much of the analytical data and evidence will be quantitative. Much of this material will be suitable for visual presentation, e.g., graphs and charts.

    Data visualization has received considerable attention in recent years. Several authors have provided very good references for improving skills in this area. The references are not specific to wargame results but are excellent for advice and examples:

    • PowerPoint and Printed Reports. Presentation of preliminary and emerging insights on a screen (e.g., using PowerPoint) may be suitable for the Quick Look Report. Viewers will have experienced the game, so will have contextual knowledge to interpret such findings appropriately. For other types of reports PowerPoint style is unsuitable. Generally, this style has a considerable lack of contextual material and readers will make assumptions (sometimes wrongly) above the meaning of headlines and partial sentences. A well-written printed report should always be included in a game project.

    • Web Pages and Wikis.

Documenting the Progress

• Project Workbook. In times past this was typically a large binder kept in the Study Director's office for safekeeping. In modern times, it is less likely to look like a physical book. Rather it may be a folder on a file system or even a wiki. The workbook may include copies of emails that discuss aspects of the study, and certainly emails where commitments of resources have been made. Whatever the format, the Project Workbook should be a repository of all material relevant to the study.

• Study Plan. The Study Plan should be issued early in the life of the project. It should be signed off by the sponsor and will constitute a contract between the sponsor's organization and the study team. Within some organizations there may be other documents that are equivalent, e.g., a tasking directive. See more on the Study Plan in the Design Phase. When the Study Plan is issued it may not yet be clear that a war game will be part of the study. Alternatively, if it is clear that wargaming should be a major part of the study, the study plan may lay out aspects like the format of the game activities and where to find players.

• Analysis Plan. The analysis plan will go into considerably more detail on how the study will be structured and what methods, models, and tools may be used. A significant part of the Analysis Plan will be the Data Collection and Management Plan.

• Data Collection and Management Plan (DCMP). The DCMP is a flexible document usually presented in a spreadsheet format. On the left of the spreadsheet will be a decomposition of the sponsor's objectives, including issues, and sub-issues (if their is sufficient complexity to go this far). For the analysis group, there should be a row (perhaps more) for each essential element of analysis (EEA). Reading across such a row should identify what measure will be used to determine what the analysis team will be looking for to address the EEA. Depending on the complexity of the project, many other aspects may be provided in other columns in the row. The DCMP format is flexible and analysts may wish to add columns to address, for example, what sort of methods, models, and tools may be required; when and where to anticipate that the players will be dealing with the corresponding issue or sub-issue, what training may be required for observers so they will be competent to assess the player responses.

• Other Inputs. For most studies that include a war game, the above documents are the minimum that should be developed and maintained by the Study Team. For a small study these may be sufficient to proceed. Some other documents that may be appropriate are:

 • Specifications for Models and Simulation Support. Models or simulations, particularly if complex or implemented on computer systems, should have an appropriate level of specification. There are many references on documenting how computer-based models or simulations should behave. The documents used on a wargame project should specify behaviours and also outline the testing regime that will be used to confirm the software behaves appropriately.

 • Data for Models, Methods, and Tools. MMT typically rely on appropriate data to function appropriately. The Study Team should confirm that sources of reliable data will be available when the MMT are to be used.

 • Network Diagrams for Communications. In a complicated war game (where there are more that a RED and a BLUE player facing each other), it can be helpful to lay out diagrams showing how the communications network should operate. If actual practice departs from this, a series of diagrams may be necessary showing both the anticipated network and the real network. Such network diagrams will be useful, for example, in explaining where messages were deplayed or corrupted or where one party was misled by information provided by another.