Bootstrapping Multiple Converging Cognitive Task Analysis Techniques for System Design

Welcome back! You probably have a lot in your inboxes already about COVID-19 so I won’t tell you what I’m doing or anything like that. All I’ll say is that we talk here about listening to expertise at the sharp end and learning from what went well as well as what didn’t, which I think can apply here too.

Bootstrapping Multiple Converging Cognitive Task Analysis Techniques for System Design

This is a chapter by Scott Potter, Emilie Roth, David Woods, and William Elm from Cognitive Task Analysis. They go over how to use cognitive task analysis to design systems, specifically how it can be used to influence the software development process.

As the authors explain:

“the goal of cognitive task analysis (CTA) is to uncover the cognitive activities that are required for task performance in a domain in order to identify opportunities to improve performance through better support of these cognitive activities.”

Essentially, cognitive task analysis is a way to examine work to help determine what tools and features of tools could be created to help with things like situation assessment and anomaly response. Sounds like a lot of things that we deal with in software.

Before writing, the authors looked at the literature that existed at the time and found that there were a number of different techniques that already existed, but many of them have various aspects in common. The authors say that they found that doing CTA effectively means really understanding the field that they’re looking at and using multiple techniques so I don’t think that knowing the exact boundaries or types of methods is necessarily required here.

All CTA methods have pretty much the same goal, to reveal what cognitive activities that individuals and teams could gain support on that would help improve their performance.

We’ve talked a bit about CTA previously , but this chapter is especially interesting since the authors point out that often, the output of CTA are artifacts that aren’t usually compatible with next steps. In this case, software development.

The “bootstrapping” bit is in the title, because instead of just applying a single CTA method, the authors advocate for iterating, applying multiple techniques over and over to help clarify and build understanding.

They draw a diagram with to axis at represent the two perspectives that need to be continuously considered throughout the process. First is the “fundamental characteristics of the domain”. This has to do with the way things exist today, before further tooling or support is developed and the things that make work in that space difficult. This includes things like:

• Why are less experienced people (in the domain) less effective?
• Why do experts use the strategies that they do?
• What are some hard cases where support would be especially helpful?

The second perspective is that of the practitioners and how they respond given the demands in their field. This is another way out looking at what makes things in that field difficult and what strategies may already be known at least by experts. Also looking at how those with less experience can further provide information on where support is needed.

Though most of the focus of the chapter is about CTA techniques, the authors emphasize that when looking for and applying techniques it is important to focus on what the output needs to be in order to be used, as opposed to the exact details of each method.

“In performing a CTA it is important to utilize a balanced suite of methods that enable both the demands of the domain and the knowledge and strategies of domain experts to be captured in a way that enables clear identification of opportunities for improved support”

Essentially, no matter how good your analysis is, if it’s in a form that no one else can then pick up and use, it doesn’t matter. So if the artifacts are diagrams and charts and the intention is to allow someone to develop software support tools, there’s probably a mismatch there unless that person happens to also be familiar with your analysis and its methods.

Getting Started

So how do you actually get started with CTA? Here is a brief overview of what the process looks like from beginning to end. Even without familiarity with specific CTA techniques it should allow you to get started with investigating and learning about difficulties in a given area. Of course, this depends a lot on your organization and your access to the people that you wish to support and build tools for.

The authors say that “for an experienced researcher conducting a CTA, one rarely has to start from scratch for each analysis.” I find this a little funny in an article that talks about bootstrapping since it potentially obscures some of the ways to get started if you don’t have experience.

The author suggests that insights in this case can come from research in “similar worlds.” So even if you don’t have experience performing the research you could potentially learn from other research that has been conducted. If you’re not sure what to read for your given area of intended investigation, send me an email, I’m happy to make a recommendation if I can.

There are a few things you can take from existing research in a similar world, such as:

• What approaches to use
• Where to focus your attention
• What types of scenarios to build for simulations or asking questions

A good thing to keep in mind is that the process of this analysis is essentially building a model of the analysts understanding of the domain and its problems. First you can start with what you know about the domain, this is often not much, but that’s OK, it’s just a starting point. I think since many of us are likely to think of ourselves as in the same field as those we may be building tools for, we need to be especially careful early on to not treat our perspective as the only one or as fact.

Next might be to interview the people you wish to help, if you have access to them. If not, perhaps you can do some observation. Once you have some information and some background research, you can begin making a prototype.

The prototype helps in case the predictions made by the interviews or research on how the work would change for those using the tooling wasn’t accurate. The prototype can be used and is likely to help you gather more information and form additional hypothesis on what support could look like.

“CTA techniques appropriate for this phase of the analysis include storyboard walkthroughs, participatory design, wizard-of-oz techniques, rapid prototype evaluations and observation of performance using simulations of various degrees of fidelity.”

Finally the authors provide some guidance on how to evaluate how the effort is going:

Criteria to consider in developing and evaluating a CTA process should include:

1. efficiency of the CTA in itself (Are the resources being invested in the CTA activities commensurate with the value of the results being obtained? )

2. validity of CTA (Does it capture what it is like to function in the field of practice?)

3. effectiveness of CTA in design (Does the CTA point to what is likely to be useful support? Does it help generate new aiding concepts and innovations? Does the CTA help to identify the bounds of aiding? Does it help avoid typical design errors? Does it generate ideas that can be readily converted to system requirements to guide system design and testing?)

4. tractability of CTA results in design (Are the products of the CTA documented in a way that can be meaningfully reviewed, tracked, and updated not only throughout the CTA phase but also throughout the entire system design life-cycle? Does it support distributed communication and coordination of design team members within and across organizational boundaries? Do the products of the CTA make contact with artifacts utilized in the software design process and can the results of the CTA be integrated into the software and product development process?)

5. predictive power of CTA (Does it help anticipate the impact of the introduction of new technologies and aiding concepts on practitioner performance? Does it predict how new technological power can change roles, expertise and error? Does it help address the envisioned world problem;)

Takeaways

• There are a number of CTA methods, but they all have the same goal: discovering the cognitive tasks that people and teams could use support on to help improve their performance.
• Doing CTA effectively is more than just applying a single technique.
  • Instead, multiple techniques can be applies repeatedly.
• To help increase effectiveness, the artifacts that come out of CTA need to be designed or translated into the right format for their use, such as for software development.
• Two perspectives should be kept in mind throughout the process:
  • That of the domain, what things are inherent in it as it exists today and what features make what problems difficult?
  • That of the practitioners, how do they respond given the constraints and difficulties of the domain?
• Prototypes can be developed to help generate further hypothesis and help increase the accuracy of predictions around how the work will change with new tooling.