I’ll be at REdeploy in SF this week. If you’ll be there too, please come by and say hi!
This is a paper by Emilie Roth and Emily Patterson where they discuss how observational studies can be used to learn about challenges and opportunities in changing how people work.
I see this as relevant to us in software, especially if we create tools for others like SRE’s might. These methods can be used to help us understand what features of a system should be preserved and also where there is room to change and enhance cognitive support.
You’ll see here that there is a lot of time put into these studies, across multiple observers and groups, but I don’t believe such a high bar is required to begin to use and benefit from such an approach.
Observational study is different from other types of scientific research you may be with. The goal here is not to eliminate most variables or make a repeatable experiment to test hypothesis, but instead to discover things that may help form a hypothesis. As a result, observational study is shaped by the observers and the methods and frameworks they bring to bear as well as what study questions are intending to be answered.
We learn about this approach through a look at two different case studies. One with train dispatchers where a move from analog radio to digital communication was being considered. A second with a nuclear power plant that was being upgraded to have computer controls and interfaces and wanted to know where additional training might be needed.
Train dispatchers
The first case study looks at a group of train dispatchers. They have the responsibility of monitoring and controlling track usage and schedules. Each individual dispatcher is responsible for a particular area.
Their normal way of working is to use regular analog voice radio, but this was determined to be an overloaded channel of communication and a digital datalink was considered. The study was intended to answer two main questions:
- What activities could be better supported with digital communication options?
- What existing features of the system are important to dispatchers?
In order to answer these questions two observers, each sitting with a dispatcher, began studying how train routing and traffic management functioned. During lower work periods they would also ask questions of the dispatchers.
In the first phase they observed in high workload, low workload, and handoff situations. Next, they interviewed experience dispatchers. Then they applied the same approach at a freight dispatch center to see what things were in common and what things differed in the other center. Finally to verify and expand on their findings they returned to the original dispatch center.
Ultimately they learned that unplanned demands on track and thus schedule as well as having to replan around outages or delays was cognitively difficult.
To accomplish these things dispatchers were:
- Monitoring trains outside of their territory
- Anticipating delays
- Balancing multiple track demands
- Making rapid decisions
As a result dispatchers were tracking (no pun intended):
- Where trains are
- If they’ll get to their destination on time
- If delayed, for how long?
Because of taking the time to observe they realized that dispatchers use the radio to listen to other communications that were not necessarily meant for them but help them plan. These communications might be around maintenance or between trains when they would broadcast when they were leaving a station. Also the dispatchers themselves would broadcast something when they realized that they had information that others might need. As you may recognize this is similar to how voice loops were used in mission control (see issue 21 for more on voice loops).
What changed as a result?
The authors tell us that some incidents, though they don’t specify which ones, confirmed that radio is an overloaded medium and that some communications could go to some new media where the information might be a better fit. For example, data link would allow for some information to be displayed visually. This is interesting because it is sort of the opposite move from what we’ve seen in many fields, where typically the overloaded medium is visual.
It was also clear that the ability to listen in and hear about things that were not specifically in their territory needed to be preserved. As a result a follow-on study was conducted that had a datalink set up with and without broadcasting and compared them in simulations. Ultimately, it was shown that data link would be effective, significantly more so with the broadcast capability preserved.
I’m not highlighting this to say that we need to do exactly this every time we build or change tooling, but I think that we can still follow much of the model. They learned what the difficulties actually were for the practitioners and as a result built a proof of concept and then deployed it. That sounds like a very doable and useful process, some of which likely happens for many of us already.
Upgrading a nuclear power plant
The next study they highlight applies the same methods in a slightly different situation. In this case upgrades to replace old control stations with a computerized system had already been decided on. The questions here were:
- What aspects of the new interface were clear improvements over the old way?
- What new, unanticipated challenges or issues could arise with the new interface?
To answer these questions five operating crews were observed during one week of training and interviewed after two days to get the operators perspective. They watched crews operate the simulator which allow them to see how they would actually interact with the new functions.
In the current system in an emergency a shift supervisor would a paper emergency plan, reading out loud for the two or more operators that were there. The shift supervisor job was to keep track of where they were in the planning response whereas operators were to read out the board values needed and to take action as directed.
The new computer interface changed this, automatically delivering the needed values to the shift supervisor as they were referencing the plan in the computer. This also allowed operators to better focus on their own work.
What this study changed
After observing, it was realized that having the numbers automatically delivered to the supervisor helped them work faster, but made it difficult for operators to know what others’ situational assessment was or where they were at in the process.
While it made some tasks easier, it also introduced a new demand, that the supervisor needed to remember to inform the team and also formulate what to say. Fortunately, further observation showed that handling this extra cognitive burden improved with training and experience.
The study also looked at if the crew was able to detect if the computers guided decision making aids were off track. There were only 3 instances where they were able to observe this and each time the crew was able to detect this and get back on track. But it’s unclear if this was representative of how other crews would perform or if this would be the case for other ways the system may get off track.
Takeaways
- Observational study can be used to discover new things like how existing solutions support work, where it could be improved, or where problems may arise.
- Unlike other research, the goal isn’t to control every variable to test a hypothesis, but instead to observe and possibly create a hypothesis
- Observational study is shaped by what frameworks observers bring and what study questions they are attempting to answer
- This is a tool that we can begin to apply in similar situations, especially when developing tools for others.
Subscribe to Resilience Roundup
Subscribe to the newsletter.