⭐Verbal reports as evidence of the process operators knowledge

This copy of the paper is on DeepDyve, a service that offers a free trial.

Verbal reports as evidence of the process operators knowledge

This is a paper by Lisanne Bainbridge, if that name sounds familiar, it’s because she’s most famous for writing The Ironies of Automation. This is one of her lesser known papers that she wrote befores essentially disappearing from the field.

Though it’s not quite as well known, it’s still helpful to us. Mainly it helps guide us as we think about and perform interviews or otherwise learn from other experts.

We can use those skills to learn about incidents or just to learn in general from other experts or as a way of understanding how normal work is done.

Observation is useful and important, but doesn’t tell us about what is going on in someone’s head. This means it doesn’t tell us about what strategies they used, why they might have used it, or what other things they may have considered. When we want to uncover the cognitive work, we can’t rely on observation alone.

Notice I say “alone.” My hope is here is that after reading this you’ll be more inclined (and more able) to combine the two methods, not just choose one over the other. They should be used in conjunction with all other sources of evidence where possible.

What constitutes other evidence will vary across each situation and if you’re focusing on the task itself, like trying to develop training, or if you’re focusing on the individual, like you might if investigating an incident.

From a research perspective, verbal data isn’t valid for all uses, but it’s still useful for most of the cases we care about. Of course people don’t have perfect memories, so there are hazards with this approach, but when we’re aware of them, we can begin to work around them.

Additionally, people don’t have some special access to their cognition. This might seem strange, we of course are aware of the results of our thinking, but no is special view into the process itself.

She cites another study that shows that when people are asked why they did something, they aren’t machines, they don’t scan their memory for an answer to the question, they do much as we might as observers and form theories.

That’s right, we don’t truly examine our memories (which are not perfect anyway) but instead construct an answer that is essentially a guess. Now, it could be a very accurate guess or inaccurate, but still just a theory.

Verbal reports as evidence

There isn’t a necessary or guaranteed correlation between what someone says when they describe their process and what actually happens in their head (mental behavior). This means that from a scientific or rigorous perspective, you can’t use what those people to prove or disprove your theories about what happens in their head.

Despite this it is possible that there is some correlation between what people say about the process and what they do.

Then in order to get the most accurate and useful information, you’d want to do what you can to get the most correlation between the report and the behavior as that also maximizes the validity. You’d also want to balance that against what a given reporting technique (survey, interview, etc…) could tell you.

Valid uses of verbal data

Because you can come up with a hypothesis from anything and from anywhere, then verbal reports are a valid way to generate hypotheses. Bainbridge mentions specifically using it to generate hypotheses around mental behavior and use that to predict the “non-verbal behavior”.

She notes that there are problems with the idea, but it’s possible that they could be used to test theories of how verbal reports are generated.

Static Simulation

This is where someone is in a simulation of a realistic scenario and “talks through” what they would do, without time pressure. Bainbridge describes them:

the simulations are normally relatively simple ‘paper and pencil’ techniques

Some forms of tabletop exercises could fit under this heading.

Oddly, Bainbridge describes this as an “extension of the structured interview,” which means anything that helps maximize information from structured interviews could help here, but I disagree. You may have a pre-built scenario, so in that sense they are similar, but the environment they tend to take place in is so different that I don’t find it especially useful to group them together like this.

Just as with other ways of forcing verbalization, static simulation and talking through it, makes the operator use a sequence that may not really exist.

Additionally, while the removal of time pressure can encourage discussion or narration, it can also change the way the simulation as done, since time pressure of course exists in the real world and can affect the work.

Types of distortions in verbal reports

Parts of a task may be done unconsciously, if this happens, then someone may give a report of what they think they do, but it may or may not be what they actually do.

Also, some tasks may involve doing multiple things at once. In these cases, trying to force something into a sequence in order to verbalize it may lead to an inaccurate or incomplete report.

Additionally, if someone lacks the vocabulary to explain what they do, the report they give may make it seem like they don’t have the knowledge they do.

Further, having to give a report at all makes some people self-conscious, so they may do the task differently and also report differently about it than they might otherwise. This is similar to some of the challenges of interviewing. And like interviewing, its important that people feel safe to share information.

I hope that you’re now better equipped to use these techniques (mixed with others!) to help further your own learning and discovery. If you still have questions or want to share how you’ll use it just drop me an email: Thai@ThaiWood.IO.

Takeaways

• Verbal reports (like interviews) are a way for us to learn about the strategies that experts use.
• These skills could be used just to learn more about a specific skill or to investigate an incident or just to better understand normal work.
• It isn’t perfect though and is best mixed with other data like some form of observation.
• Understanding where and how verbal reports fit help us maximize the useful information we receive from them while minimizing issues with the accuracy or usefulness of that information.
• We don’t have any special insight into our own decision making process, we tend to create theories that explain our behavior when asked “why?,” instead of reviewing memory.