Naturalistic Decision Making Tools

Recently, I polled many of the leaders of the Naturalistic Decision Making (NDM) community about the tools and methods that they’ve developed and have used. By “tool” I meant something that could be used to get a job done, and that included conceptual models. However, we did not include methods that are primarily designed for doing laboratory research involving controlled experiments.

The following colleagues helped me to identify all these tools (in alphabetical order): Cindy Dominguez, Julie Gore, Robert Hoffman, Devorah Klein, Laura Militello, Brian Moon, Emilie Roth, Jan Maarten Schraagen, and Neelam Naikar.

The result was a list of 42 methods, much more than I had expected. But after all, the NDM community has been going strong since 1989.

The tools fell into 9 categories. Here is a link to the complete set of tools, along with a key reference for each. Adam Zaremsky took the lead in adding the short descriptors and references.

Knowledge elicitation tools, primarily methods for doing Cognitive Task Analysis such as the Critical Decision Method, the Situation Awareness Record, Applied Cognitive Task Analysis (ACTA), the Knowledge Audit, the Cognitive Audit, and Concept Maps.

Cognitive specifications and representations, such as the Cognitive Requirements Table, the Critical Cue Inventory, the Cognimeter, Integrated Cognitive Analyses for Human-Machine Teaming, Contextual Activity Templates, and Diagrams of Work Organization Possibilities.

Training approaches, including the ShadowBox technique, Tactical Decision Games, Artificial Intelligence Quotient, On-the-Job Training, and Cognitive After-Action Review Guide for Observers.

Design methods, e.g., Decision-Centered Design, Principles for Collaborative Automation, and Principles of Human-Centered Computing.

Evaluation techniques such as Sero!, Concept Maps, Decision Making Record, Work-Centered Evaluation.

Teamwork aids, e.g., the Situation Awareness Calibration questions, the Cultural Lens model.

Risk assessment methods: the pre-mortem.

Measurement techniques such as Macrocognitive measures, Hoffman’s “performance assessment by order statistics” and four scales for explainable Artificial Intelligence.

Conceptual descriptions: these are models like the Recognition-Primed Decision model that have been used in a variety of ways.

Undoubtedly this compilation will expand in the years to come, but it seems useful to have it on hand as a reference and guide.