The Risks of Autonomy: Doyle’s Catch

As I do each year, I’ll be spending the rest of the year preparing and planning new content. See you back here in 2021!

The Risks of Autonomy: Doyle’s Catch

This is a short paper by David Woods, which serves as an excellent reminder, especially for us in software about the hazards of introducing new technologies without plans or designs on how they’ll used and adopted in the real world.

The observations the author makes come from attending a number of self driving car briefings as part of DARPA’s Grand Challenge, where some of the vehicles required as much as “75 engineers from one of the top engineering universities in the world over 18 calendar months.”

Unlike some of the research this paper is directed almost directly to us. Even the title highlights a view point often overlooked in software. Automation is usually seen as something that is almost universally good, it’s almost never seen or evaluated as a risk.

I’ve discussed other dangers of automation elsewhere, but Doyle’s catch highlights another risk. That new risks can be overlooked, especially where demos have been made using some sort of idealized version of the world.

Computer-based simulation and rapid prototyping tools are now broadly available and powerful enough that it is relatively easy to demonstrate almost anything, provided that conditions are made sufficiently idealized. However, the real world is typically far from idealized, and thus a system must have enough robustness in order to close the gap between demonstration and the real thing.

Notice that Doyle’s catch doesn’t even begin to talk about resilience, even robustness (which can be significantly easier to achieve than resilience) can improve the situation.

They presumed that because capabilities could be demonstrated under some conditions, extending the prototypes to handle the full range of complexities that emerge and change over life cycles would be straightforward.

While this might be a possibility, hoping it is so is not enough. Instead, Doyle’s catch helps us can front the question of how can we “close the gap between the demonstration and the real thing?”

Closing the gap between the demonstration and the real thing requires the development of new methods to manage creeping complexity and the associated costs.

Part of the issue is overlooking the change required over the life cycle of the technology. It needs to be “poised to change” and will face:

• surprise events
• Different use conditions and contexts
• Even though self about driving cars have software that allows them to adapt, they will still experience shortfalls in that adaptation where humans will have to fill the gap

If you had any doubts that Woods is addressing us software to folks:

Handling life cycle dynamics will require an architecture equipped with the capacity to adjust performance over a wide dynamic range. This is, in part, the target of extensible critical digital services and closely related to resilience engineering.

Instead of doing that is so common and what this challenge did, we can “test for brittleness rather than feasibility.”

Specifically we can use the turn around test, where we ask: “how much work does it take to get a system ready to handle the next mission/case/environment, when the next is not a simple parametric variation of the previous demonstration?”

As you probably guessed, most autonomous systems do badly on this measure, including the self driving car in the challenge.

“Doyle’s Catch highlights how demonstrations can be brittle in ways that are unappreciated. But when the demonstration encounters the full complexity and scale of real-world deployments, the forms of brittleness undermine the viability of a system and require people in various roles to adapt to fill the gaps. As a result, there is a need to assess the brittleness of envisioned systems as they move from demonstration to deployment and across its life cycle.”

Takeaways

• Doyle’s catch is the gap between a prototype or simulation and the complexity of the real world.
• Ignoring this gap is one of the ways that new technologies can create new challenges,
• While future about engineering efforts may solve some problems, its important to consider how to close the gap between the demo and what the real world will require.
• “Closing the gap between the demonstration and the real thing requires the development of new methods to manage creeping complexity and the associated costs.”
• When examining a prototype system we can test for brittleness instead of feasibility.