Trust in Autonomy

Editors Note: This is the third installation in a series addressing the future of autonomous aerial systems training and acquisition.  This article addresses one of the largest barriers in the mind of both defense professionals and the public at large – trust in autonomous systems.

By Nicholas J. Helms

The first installation of this series introduced Systems Training as a concept for training autonomous air vehicles like human wingmen or co-pilots. The next installation will put flesh on the bones of this concept. In between, however, we are delving into the concept of trust.  Logically, this required a new definition of autonomy – Airmen should conceptualize autonomy as a blended spectrum of tasks that, when combined, facilitate a generally understood capacity for action. This definition, discussed in part two and which assumes humans and machines as an inseparable entity, allows Airmen to see their interactions with machines similar to their interaction with student aircrew. This week, we will reinforce the long-term nature of this trust development.

Autonomy has raised concern on the topic of trust. According to Graham Warwick, an author for Aviation Week, trust in autonomy “has been identified as the biggest challenge” to accepting these types of technologies as useful. Dr. Jason Ryan, a Human and Autonomy Engineer at Aurora Flight Sciences, described the iterative modeling of the human-machine relationship as the “calibration of trust.” He said that humans do not build trust in machine autonomy, they calibrate trust over time. Thus, trust in autonomous air vehicle development is situation dependent, and long-term trust can yield powerful results. Familiarity with an autonomous machine is a major contributor to trust.

Humans are not equally autonomous. In other words, consistent with the Airmen’s definition of autonomy, humans have different capacities for action. A toddler that is restricted from playing near the stairs has less capacity to act than a teenager, a teenager under curfew has capacity to act than an adult, and new employees have less autonomy than trained colleagues. Likewise, Airmen do not trust an undergraduate pilot student, a mission-qualified wingman, or a mission-commander at the same levels. Aircrew members are tested initially during training and throughout their careers during upgrade training programs. Responsibility with an aircraft is earned progressively over time.

Trustworthy behavior is situation dependent. In mathematical models of trust the dominant strategy over time occurs when a person trusts until their partner defects from cooperation, but eventually finds the grace to trust again. This is described as a “generous tit-for-tat” behavior.    Aircrew are not above tit-for-tat trust behaviors with each other. Airmen invest, and then reinvest training in trustworthy teammates with high aptitude. In his book Outliers, Malcolm Gladwell highlighted profound long-term effects on success stories as a result of tit-for-tat trust behaviors in students. Consistent with Gladwell’s example, an aircrew demonstrating early aptitude in academics may ultimately succeed, not because he was indeed better, but because he got more specific pre-flight briefings, which enabled more challenging airborne missions, which led to more substantive flight debriefings, which earned him an early qualification upgrade, and so on.  If generous tit-for-tat were an aircrew trust strategy, then a weak-aptitude aircrew member could eventually persevere to high levels of success, albeit after his or her high-aptitude peers have already done so.  As is the case with humans, long-term trust behaviors towards machine autonomy will have powerful impact.

Familiarity plays a role as humans calibrate trustworthiness. MITRE and Google researchers studied the factors that influenced trust in autonomous vehicles, and familiarity with past performance was overwhelmingly important. Out of twenty-eight trust influences, statistics on past performance was the top factor, an “understanding of the way” the vehicle worked was ninth, and “past experience” was tenth. These trust influences outscored performance measures like accuracy, effectiveness, and processing capacity for an autonomous vehicle. This demonstrates a human’s propensity to trust those things that are familiar while simultaneously accommodating performance vulnerabilities for the trust subject.

Once a new technology catches on, the way trust spreads in a population is predictable. In The Tipping Point, Malcolm Gladwell likened “the phenomena of [trust]” to the spread of viruses years before people had a shared understanding of the term “viral”.  Simon Sinek also described the manifestation of trust in his book “Start With Why.” Both authors cited Geoffrey Moore’s “Law of Diffusion of Innovations” to characterize the trust gaps between different portions of a population towards technology. In general, a small portion of population is eager to trust a new technology, the majority of the population requires time to trust that same technology, and another small portion of the population is perpetually skeptical to trust. Therefore, timing is an important factor in the development of new technologies. Over time, personality differences temper enthusiasm between adopters of technology, skeptical majority consumers, and cynical technology laggards. Trust in a technology can be accelerated if that technology is useful and if the population has broad access to it.  Trust in autonomous air vehicles are in the midst of a similar long-term pattern.

In Wired for War, P.W. Singer highlights timing’s role in fulfilling technological trust gaps. He said that timing enables innovation to succeed “at just the right moment that both technology and the marketplace are ready to support.” MQ-1 Predator and MQ-9 Reaper aircraft achieved popular familiarity during US conflict in Afghanistan and Iraq, when all of the major defense contractors and “more than 50” innovators got into the business of drones. As people became more familiar with Predator and Reaper aircraft, they came to understand these particular systems as a complicated remote-controlled aircraft with simple autonomy. When it came to decisions and actions, the only task a Predator or Reaper aircraft could reliably be trusted to accomplish independently was to return-to-airbase in the event of lost remote-control datalink.  Nonetheless, the timing of the Predator and Reaper development matched with the demand for Predator and Reaper capability during US combat operations enabled familiarity and trust in machine autonomy for a large number of Air Force aircrew members.

Familiarity helps operators calibrate trust against fair expectations for machine autonomy.  Humans tend to trust what they can see because they are visual animals. But humans are also capable of discriminating sensory thresholds and experiencing sensory adaptation to more accurately tune into an environment. Thus, human familiarity with the types of feedback that an autonomous air vehicle can provide helps the human to calibrate trust in the system amidst different contingencies. Over time, operators will calibrate trust in the context of specific expectations for a given machine capability. For example, MQ-9 operators accepted the “bad” (datalink integrity) with the “good” (range and persistence) because the overall capacity for military benefit of the man-machine team was worth the tradeoff.

A different combat environment would change MQ-9 operators’ trust in the system. An environment that contests the command and control datalinks, for example, would demand more machine autonomy to achieve the same capacity for action. Conversely, a dynamic environment with unanticipated threats, targets, or weather, may benefit from human remote-control. Durst and Gray said that “for an unmanned aerial vehicle, a fully autonomous intelligent asset does not assure the best mission completion status in the case of an ever-changing scenario or even a time sensitive target.” Machine autonomy fulfills delegated decisions and actions. However, to successfully complete a task without human intervention, those decisions and actions must account for all versions of the task, including contingencies. Humans use powerful cognitive techniques called heuristics to translate solutions from one problem to another. To date, machines are not capable of accomplishing this same translation. This serves as a significant source of skepticism towards machine autonomy. But rather than focus on trust in the most difficult of environments amidst the most fundamental of human and machine processing differences, let us look back at other past examples of human-machine trust that increased capacity for action.

Machine autonomy is important to remote-controlled unmanned air vehicles, but being unmanned does not directly translate to being autonomous. In terms of electronic control, one could argue that the fly-by-wire flight control system makes a manned aircraft tele-operated.  The pilot of an F-16, F-117, or F-35 does not directly set a flight control surface deflection.  Instead, the fly-by-wire flight controls balance stability variables and make decisions about where to position flight control surfaces to deliver the pilot’s intended effect, like commanded pitch rate or G-force. Pilots would lose inputs to these flight controls if wires between the control stick and the processor were severed, but fly-by-wire aircraft mitigate the interruption of aircraft control with redundant electronic flight control pathways. Whether these flight control systems are deterministic is not as important as their responsiveness, end-result predictability, and familiarity towards operators over time.

The pilot may not be in direct mechanical control of a flight control surface, but over time he will gladly rely on the flight control computer to decide where flight controls are positioned in order to preserve aircraft stability. Consistent with the human-machine definition of autonomy, increased participation of the machine affords increased capacity for action as a pilot can now control an otherwise unstable F-16, F-117, or F-35. To the human pilot, mathematical determinism demonstrated by flight control surfaces is not as important as the effect-based determinism that provides stability by the flight control system. The exact position of a flight control surface dependent on 20 inputs per second is the least of the pilot’s concerns, but the preservation of pitch, roll, and yaw flying qualities at all corners of the flight envelope is what matters. Similar to the MQ-9, familiarity enables the pilot to calibrate trust in fly-by-wire aircraft over time as potentially unpredictable flight conditions are rendered predictable by the machine’s participation in the flight control team.

Increased machine participation in being welcomed by aircrew that are more and more familiar with the attributes of a trustworthy machine. Beyond what we have already understood regarding familiarity, timing, and compensation, a trustworthy machine enables responsive transitions between human and machine decision space. USAF F-16s employing dive-bomb deliveries at 450 nautical miles per hour are protected by Auto-GCAS, a rule-based logic that protects the aircraft from collision with the ground. Among other rules, the collision avoidance capability is active unless the F-16 is in an aerial refueling configuration, a landing configuration, the airspeed is too low to facilitate 2G flight maneuvers, or the pilot ejected out of the aircraft. These rules help to prevent the machine from making erroneous maneuvers. That way, the pilot can execute a mission without being interrupted by a machine decision that undermines the very safety it is supposed to support. When this USAF F-16 capability is triggered, the transition from human to machine, and back to human control is seamless. In other words, if the machine predicts that the human will crash, then the aircraft takes over, flies to a safe condition, and quits maneuvering so the pilot can fly again. The transition between human and machine is seamless.  Since USAF F-16 delivers accurately within its machine decision space, operator trust is not complicated. Pilots know what to expect, and how to best exploit machine autonomy over time.

Autonomy as it is defined in this series, a balance between human and machine capacity to act, highlights the importance of familiarity in developing trust. Humans calibrate trust with machines over the long-term.  Familiarity with machine autonomy helps the human calibrate trust in the context of specific tasks, machine capabilities, and specific environments. Trust in autonomy is not defined by perfection. For both manned and unmanned aircraft, familiarity with the machine allows the human to develop techniques that simultaneously leverage machine strengths and compensate for machine weaknesses. These techniques benefit the utility of the combined human-machine team and bolster human trust in reliable machine performance.

Fundamentally, trust is calibrated through an experience of cooperation or conflict, and it is vital that human aircrew have thousands of opportunities to model cooperation and conflict with machine teammates. Generous tit-for-tat behaviors do not reinforce trustworthy interdependency if machine software is locked in a development cycle that renders the human unable to develop trust.  Therefore, an Airman’s trust in autonomy will be less dependent on what autonomy does today than it is on how rapidly autonomy can adapt to novel use in a complex environment tomorrow.  This is not because Airmen have low standards for autonomy; it is due to natural human psychology.  Treating autonomy as a malleable human-machine team enables Airmen brains to work as they always have, as long-term complex adaptive systems.  The next installation will compare reductionism to complexity, reinforcing the concept of training our autonomous air vehicles like our student human aircrew.

Nicholas J. Helms is a graduate of the USAF Academy with a Bachelor of Science in Human Factors Engineering.  He is a distinguished graduate of the USAF Test Pilot School with over 2,000 hours piloting multiple aircraft, including the F-16, MQ-9, T-38C, and C-12J.  He has flown missions in support of Operations Noble Eagle, Iraqi Freedom, and Enduring Freedom.

Disclaimer: The views expressed are those of the author and do not necessarily reflect the official policy or position of the Department of the Air Force or the U.S. Government.

Print Friendly, PDF & Email

Leave a Reply