The learning review: Adding to the accident investigation toolbox

Only 1 more regular issue left before the new Resilience Roundup launches! There’s still time to get a discount as an early adopter here.

The learning review: Adding to the accident investigation toolbox

This paper not only goes over what a learning review is, but also how the organization (in this case the United States Forest Service) came to adopt it, as well as why it’s a useful tool.

The USFS has a number of duties, but the main one focused on here is that they are an organization that is responsible for wildland (not city) firefighting.

The authors also give a good overview of some of the pitfalls of accident investigation of complex systems (the kind we likely all deal with in software).

Not only is this a good starting point for how a similar system might be implemented, but it also allows examination of current systems. After reading here, I strongly encourage you to look for features you might adopt or places where your current system may better serve.

The old way

The Serious Accident Investigation Guide (SAIG) was the old way the US Forest Service investigated accidents. Even thought it was revised in 2005, it was still firmly rooted in techniques and models used for mechanical, linear events, which is not how humans work.

Also, SAIG encouraged a focus on a timeline. Instead, the learning review process started being developed to build context around decisions and actions.

Before the US Forest Service implemented the learning review, incidents were treated like criminal investigations with lots of secrecy. Predictably, this sowed a lot of distrust and people became less willing to share information.

The Learning Review

It was brought about because there were over 400 wildland firefighter line of duty deaths between 1995-2015. During this period, though accidents were investigated “no substantive changes in operation or policy resulted from investigation.”

Typically accident investigations blamed people without investigating why actions were taken. At the same time, accident rates were trending upward.

The Learning Review process attempts to focus attention on how conditions relate to behaviors and decisions without ascribing blame.

The learning review starts with some grounding principles:

• That operators are well intentioned and are working within the system to meet goals.
  • This touches on the local rationality principle, that an action or decision must have made sense at the time or it wouldn’t have been performed, no one shows up to do a bad job.
• Incidents can be a by product of the uncertainty that is inherent in complex systems.
• Before an incident leaders are responsible for knowing how the organization functions. At this stage “traditional” methods of accountability are useful.
  • “After an incident, prevention is based on learning. The organization is accountable to learn all it can from the incident”
• Conditions affect decisions and actions. Making these conditions visible and understanding them can help share future recognition and reaction.

From these principles some tools were developed:

• The complex narrative. This is the recognition that human memory and recall isn’t perfect, regardless of how it is gathered. Included are attempts to avoid unintentionally biased language where possible.
  • This in turn shapes how interviews are conducted. Since we recognize that recall isn’t perfect, we then expect different accounts to conflict, and let go of the need to produce a single “true” account.
    • These conflicts can help to raise different questions that a single account could not. When there is a conflict we can begin to ask things like “were they aware of it?” “how did they resolve it?”
• Network of influences map. This is based on Rasmussen’s Accimap, but focuses on “influences” instead of “causes”.
  • This is because the organization found it easier to explore connections and ask questions. When they were focused on “causes,” there was a pressure to draw a direct line between events (possibly requiring firm evidence), but when discussing “influences” intuition and experience were easier to discuss.
• A focus on normal work. “Prevention is forward looking and our processes were retrospective.” In response, the network of influences map and complex narrative are presented to a focus group to help determine if the conditions are in common with normal work.

Based on what the above tools help reveal, they are then categorized into one of the following 4:

1. Conditions outside of leadership control
2. Conditions that, if changed, will have meaningful impact but may take a long time. (These are usually cultural the authors note.)
3. Conditions that, if changed, will meaningfully impact operations and can be changed quickly.
4. Conditions that have minimal impact if changed.

Recommendations on what to change and how based on these categories is not handed down form on high, but developed in collaboration with those closest to the work.

Your toolbox

This is an opportunity to examine your investigation toolbox, do you have tools that let you do some of the same things? Does this approach to learning extend your toolbox?

Takeaways

• Linear systems of accident investigation don’t match how humans actually work.
  • Attempting to apply them anyways can be misleading if not outright harmful.
• A complex narrative is captured, letting go of the need to create one “true” account of an incident or resolve differences between accounts.
• Artifacts of the learning review like the network of influence map are shown to focus groups to help determine if conditions in the incident are a part of normal work.
• What to change and how is determined in collaboration with those closest to the work.
• Conditions affect decisions and actions. Making these conditions visible and understanding them can help share future recognition and reaction.