A Sense Making Experiment

A Sense Making Experiment – Enhanced Reasoning Techniques to Achieve Cognitive Perception

This week we’re diving a bit deeper into some techniques around increasing sensitivity to weak signals and generating more scenarios and ideas when planning.

This is a paper that was presented at the 12th international command-and-control research and technology symposium by Gary Klein, David Snowden, Chew Lock Pin, and Cheryl Ann Teh.

It goes over an experiment that was done with military and intelligence teams. We’ve talked a bit about this experiment before since the results are what helped develop the anticipatory thinking model. Instead of looking at that model, but this time we will dive into these specific techniques that were used with these teams and explore where they were and were not effective.

This experiment essentially asked the question “what causes us to be insensitive to weak signals? And how can we improve?”. This was examined by giving seven of these military/intelligence teams two different exercises. One was a military scenario that was strategic and also designed to feel familiar to some of the things the teams might’ve experienced in the past. This allowed an increased chance of teams just going along with their assumptions based on surface level info. The next exercise was a homeland security type scenario where the objective was to take in a ton of information, some of it irrelevant some of it conflicting, and determine when, where, and how this eminent threat would take place.

For the experiment they had some ideas around what was it that might cause individuals and teams to miss weak signals, such as:

1. Cognitive biases including
   • confirmation bias
   • recency effect
   • availability heuristic
2. Compartmentalize knowledge

The Techniques

Crystal Ball

This is an exercise that was developed by Martin Cohen to help with constructive criticism of situation assessments. The exercise is designed to avoid two main problems. One is that when we do this for ourselves, a part of us may be hoping that we don’t find any big flaws. And the second is that in a team environment, individuals can be hesitant to critique ideas of other members.

The way the exercise works is that after the team has made an assessment, they are told that they have the right information, but the assessment is wrong. The group then goes back and generates alternate assessments. This can help the team prevent fixation on any one pattern and think in a different way that might reveal flaws.

Ritualized Dissent

This is a method that was developed by the Cynefin Centre designed to help defeat pattern entrainment, where once we get an idea we can get stuck in it, and also groupthink.

First different groups are formed, where they all go through the same process of assessment or evaluation of the situation. Next, each group sends a single spokesperson to the other groups. The spokesperson then presents their groups ideas.

Here’s where it gets interesting. After presenting, the spokesperson turns around so their back is to the people they presented to and is no longer allowed to speak or otherwise engage with them. Now, those that heard the presentation will try and dismantle the ideas.

The idea here is that by not being able to engage, the spokesperson is not just thinking up a rebuttal, but is truly listening to what is being said. At the same time in trying to dismantle those ideas, the audience may begin to see flaws in their own ideas.

This seems like something that could be difficult to do depending on your specific teams, though they do say that the idea of turning around helps depersonalize what’s going on a little bit. I’ve been on some teams where I can see this being really effective and others where it might be a bit more of a concern.

Attractors/Barriers (AB) Framing

The idea in this exercise is to generate and use abstracted language to talk up about the characteristics of the system in such a way as to avoid just trying to predict outcomes and also make it more clear what things can be changed or other traits.

This is done by saying that interactions take place within barriers and around attractors. Or instead “attractors attract and barriers repel”. Typically a metaphor will be provided along with this. The idea being that you must describe the situation in terms of attractors and barriers not in terms of causality. Then identify what of those are in your control or maybe you which ones are stable etc…

They warned that this only really works when you’re describing some sort of uncertain future or at least one that’s uncertain from the perspective of the person participating. That means that this does not work for retrospectives.

I found this exercise interesting as an idea, but it seems really awkward and perhaps hard to implement. They don’t go into a a lot of detail, but at the end they asked participants to rate exercises and this one was rated as being conceptually difficult. To such a degree that the authors recommend developing the technique further.

Future Backwards

This technique is similar to prospective hindsight approaches that you may have heard elsewhere. It was developed as an alternative to scenario planning in an attempt to expand how many possibilities are considered. This can take place over a very short time, say 10 to 15 minutes, or could go on for several hours depending on the needs of the group and the consequences of the operation.

The authors recommend the following steps:

1. Identify what the current state of the situation is in terms of whatever limited number of decisions being faced
2. Step backwards in time to the last turning point that gave rise to the current situation
3. Continue to keep stepping backwards until going further wouldn’t really seem to change anything
4. Now identify two possible future states, one just ridiculously good and one just ridiculously bad. These should not be realistic.
5. Now find a way to make those futures happen by going through the past turning points and as needed insert a major accident or surprise.

The idea here is that when we are reasoning hypothetically around something that is said to be very certain, like one of those future states, then you work more creatively than we would if we just said there were these uncertain future states.

Situation Awareness Calibration

The authors briefly mention this idea of using an exercise to help reveal what others on the team are thinking so that common ground can be maintained and potentially give insight from diverse viewpoints.

The authors don’t really detail what they did before this is an exercise but do mention they asked some questions during it to gather some data, such as:

• “What is the immediate goal of your squad/team?”
• “What are you doing to support this goal?”
• “What are you concerned about?”
• “What is the current threat location, size, and intention?”
• “What do you think the situation will look like in 24 hours?”

Results from the Experiment

Looking at the data of the resulted from this experiment the authors were able to stay that using the AB framing and future backwards techniques led to increased awareness of weak signals and situation alternatives to varying degrees compared to their control group.

AB mostly helped with overall understanding of intent for the mission and the purpose of information gathering, whereas future backwards lead to understanding of previously unconsidered scenarios. Future backwards also narrowed the gap between what groups discussed as the state of situation and their proposed plan versus what individuals role in their narrative as the state and proposed plan.

Something to be aware of is that in examining the narratives that the participants produced, the researchers had noticed that there were already some individuals sometimes as much as half of the group who were already noticing the weak signals. There was even a case where a single individual had come as far as almost predicting the entire scenario.

The problem is that when the signals were brought up with the team they were often dismissed and in some cases they will brought up at all. After the interventions all of the teams were more attentive to those weaker signals being brought up.

Additionally, after both future backwards and ritualized dissent the level of debate increased. The authors note that they don’t have any data to back this up but their impression was that deference to rank also seems to go down, which indicates more people willing to speak up or listen to other ideas regardless of rank.

Takeaways

• There are processes and routines that can help teams and individuals generate more hypothesis and ideas and prevent straying down the “garden path” in thinking.
  • Based on just this paper alone it can be difficult to decide if these approaches would work with your team.
• Individuals were already noticing weak signals much of the time, but being dismissed by their teams.
• While the authors indicate that the attractors barriers exercise was effective in some ways they do indicate that it was consistently voted as conceptually difficult. I think this is something important to keep in mind if we’re going to implement some of these things with our own teams or during game day type setups.
• Using prospective hindsight approaches like the future backwards technique is effective in generating more scenarios to account for.
• Future backwards also narrowed the gap between what the group had discussed as the plan and status the situation and what individuals thought it was or should be.
• Many of these techniques are potentially appropriate with modification for our pre-mortems or game days.