theory could reach a level of âblind trustâ or. âcomplacencyâ in which operators cease to monitor the automated system at all. However the quantitative models of ...
ARE OBSERVERS EVER REALLY COMPLACENT WHEN MONITORING AUTOMATED SYSTEMS? Neville Moray Department of Psychology University of Surrey, UK. A recurring worry in recent applied research on the role of humans in highly reliable automated systems has been the fear of “complacency’, or the tendency to trust automation too much, with the consequence that faults or abnormal function go undetected. Existing evidence does not support the conclusion that operators are complacent. Rather, it supports the notion that in any complex dynamic system even an operator who is well calibrated with respect to the probability of faults, who shows eutactic behaviour, and who behaves optimally cannot be expected to detect all faults. The question of what strategy should be adopted when monitoring a highly reliable system is discussed. Trust, Reliability and Complacency. As the quality of automation has improved and automated systems have become increasingly reliable, the role of human operators is increasingly to exercise supervisory control (Sheridan, 1976, 1997). In such a mode their task is to monitor displays and interfaces to detect faults or abnormal operating states of complex plant. The high reliability of modern automated systems means that such abnormal states occur only rarely, and it has been reported that operators may become “complacent”. The word “complacent” refers to “self-satisfaction which may result in non-vigilance based on an unjustified assumption of satisfactory system state” (NASA definition). Note that the phrase is pejorative and puts the blame on the operator. Empirical studies have shown that trust in reliable automation certainly increases with experience of reliable performance (Muir and Moray, 1996; Lee and Moray, 1992, 1994), and in theory could reach a level of “blind trust” or “complacency” in which operators cease to monitor the automated system at all. However the quantitative models of trust developed by Lee and Moray (1992,1994) and by Moray,
Inagaki and Itoh (1999) show a mathematical form which suggests that such is not the case, at least in the systems studied by those researchers. Rather, if anything, trust seems to asymptote at a level slightly less than is deserved on the basis of the experience of the operators. What then is the evidence for complacency? All the studies which claim to have found complacency base that claim on the fact that observers miss signals that indicate abnormal system states (see e.g., Endsley and Kiris, 1995; Mosier, Skitka, and Korte, 1999; Singh, Molloy, and Parasuraman, 1997; Parasuraman, Mouloua, and Molly, 1996.) But missing a signal is not necessarily an indication of complacency: it could be simply a sign that the signal-to-noise ratio of the signal is very low, making it hard to observe even when the operator is dedicated completely to observing the source of the signal. To be fair however those who talk of complacency generally provide readily detectable signals, so that such an interpretation of missed signals is unlikely to explain all missed signals. However, to consider such a possible explanation draws attention to the fact
that a failure to detect a signal is not in itself a sign of complacency. Consider the following thought-experiment. Defining complacency An observer must monitor two displays, which cannot be viewed at the same time, for example two pages of computer displays each of which fills the screen completely if selected. The maximum rate at which the pages can be switched is once every second. The signals which are to be detected last for 1 second. Signals to be detected are highly detectable, and if viewed are never missed. On Page A a signal occurs, if at all, every 7th second, on Page B every 13th second. This is known to the observer. What is the optimal monitoring strategy? Clearly the observer should select Page A at time t = 7, 14, 21…seconds, and Page B at times t = 13, 26, 39….seconds. this is indeed an optimal strategy in the strong sense of optimal. All signals will be detected until t = 91 seconds. At that moment let a signal occur on both Pages, so that the signal will be detected on whichever page is currently in view, and missed on the other page. We conclude from this example that a perfect observer, who knows exactly when a signal may occur, is fully vigilant, and is not complacent, may nonetheless fail to detect a signal. Hence the failure to detect a signal cannot in itself be a sign of complacency. Complacency is about monitoring, not about detection. The notion of complacency is not currently well defined, and until a formal definition is given, complacency, which implies a defect in the performance of the operator, should not be invoked as an explanation of missed signals. In the thought-experiment the failure is one of system design, more specifically of display design, not of complacency. A formal definition can be given as follows. The behaviour of an observer is sceptical if a source is sampled (monitored) more frequently than is warranted by the statistics of occurrence
of the signals to be detected (more frequently than optimal). The behaviour of an observer is complacent if a source is sampled (monitored) less frequently than is warranted by the statistics of occurrence of the signals to be detected (less frequently than optimal). The behaviour of an observer is eutactic1 if a source is sampled (monitored) at a rate perfectly calibrated by the statistics of occurrence of the signals to be detected (at an optimal rate). Only if an observer samples a source of information less frequently than he or she should on the basis of their past experience of the statistics of events can they be said to be complacent. Clearly a signal will be missed if the source is not sampled: but where attention must be divided among several sources even optimal sampling cannot guarantee detection. Since none of the reports of complacent behaviour has defined what is the optimal or eutactic rate of monitoring, it follows that no observers have been shown to be complacent. Indeed a re-analysis of the data in the experiments of Parasuraman et al. (1996) strongly suggests that their observers were eutactic with respect to the presentation rate of signals, despite the fact that they missed signals. Monitoring rare signals The above analysis questions whether any evidence exists for complacency, while not denying that important signals are missed. It also raises the very important question, one which will become more and more important as reliability of hazardous and/or risky systems increases, of how often an observer should monitor a “perfectly reliable” system, that is, one which has never been known to fail in the past. This is a particular case of the general monitoring problem, which can be thought of as the problem of allocation of attention. In pursuit of a quantitative model which can provide an eutactic norm against which to detect complacency, we should consider worst case situations such as those used in the thought-
experiment above. That is, we must consider monitoring as a queuing problem in which attention can be given to only one source at a time. This approach is chosen for two reasons. The first is that other models, such as parallel attention, multiple resource models, etc., have not been specified in sufficient detail for them to be used as the basis for a quantitative predictive model. The second reason is that in many real situations attention is constrained by eyemovements, or by the need to page through computer displays sequentially. The world often demands sequential sampling. It has been suggested (Moray, 1986) that the best starting point for such a general model is Carbonell’s model which determines sampling intervals on the basis of an expected value measure of the desirability of an observation. That is, the choice of a sampling instant is related to the probability of an incident of interest being present, weighted by the value (or cost) of the observation. Thus the source of a rarely occurring signal will be sampled only infrequently, but if it is very important to detect the signal, because of its associated hazard the frequency will be increased. We can think of the frequency of sampling being related to the product of risk and hazard. Such a model is a plausible norm for situations where signals (faults, abnormal states) occur with a frequency which can be estimated, even if only after many hours of experience by the observer. But a completely reliable source does not provide a norm. The probability, based on past evidence, that a source which has never failed will fail at time t is clearly zero, regardless of the value of t. Indeed, far from it being complacent never to sample a completely reliable system, such a strategy is completely rational. On the basis of past experience, there is no instant at which it makes sense to sample a completely reliable source, particularly since any time spent sampling such a source will reduce the time available to monitor other sources which may be known to be less than 100% reliable.
While such a conclusion is rational, it is clearly not acceptable in real work situations. The problem is that it instantiates, in a real way, the problem of induction. We know that the fact that a system has never failed in the past does not preclude it from failing in the future, and we expect the operator-observer to detect any such failure when it occurs for the first time. In other words, when dealing with real systems such as power stations, aircraft, military situation assessment, etc., we are not content with eutactic behaviour, let alone complacent behaviour. What we intuitively require is rational sceptical behaviour – but at what frequency? A possible approach is as follows. Consider the dynamics of a fault. Let the time from the occurrence of a fault until the dangerous consequences are unavoidable (the incident is unrecoverable) be T seconds. Let the time required to take action to prevent the unrecoverable consequences be t seconds. Then the system which has never suffered a fault should be sampled with a frequency related to (T – t).w, where w is a weight related to the severity of the consequences of an unrecoverable accident. Note that such a strategy will not guarantee the timely detection of all faults when they first occur. But as we saw in the thought-experiment, no strategy, not even an optimal strategy, will guarantee the timely detection of all faults. An optimal strategy will certainly guarantee the detection of the maximum number of fault signals, but cannot guarantee that they will all be detected, especially in large systems. Equally, it is never the case that a failure to detect signals is evidence of complacency: only a failure to sample, to monitor, the source of information at less than eutactic rates can be that. Note also that this is not just a matter of semantics. To claim that an operator missed a signal because of complacency is to have recourse to the classic tactic which we all claim to eschew, namely, put the blame on the operator and say that “human error” was the cause of the problem. What we should be doing is to explain that even if the operator is
performing optimally no certainty can be guaranteed in stochastic systems. The only way to approach certain detection of signals is to have the operator monitor only one source, which is impractical, or to have strong alarms associated with all occurrences of abnormal states and faults. The latter also gives rise to problems as is well known, and such alarms must be much more reliable than the system itself. Conclusions 1. Current research does not provide evidence for complacency in monitoring rare events, and provides some evidence that such monitoring approaches eutactic strategy. 2. Even optimal monitoring cannot detect all signals. 3. Research is needed on a normative optimal model for sampling 100% reliable sources. 4. Only when such a model is available can the design of alarms, training for monitoring, etc., be rationally based. 5. Blaming the “complacency” of operators for a failure to detect signals is tantamount to placing the blame on “operator error”, and is just as unacceptable asa is the latter.
Mora y, N., Inagaki, T., & Itoh, M. 1999. Allocation of function, situation adaptive automation and fault management in time-critical tasks. Journal of Experimental Psychology: Applied. (in press). Mosier, K., Skitka, L. J., & Korte, K. J. 1999. Muir, B. M., & Moray, N. (1996). Trust in automation. Part II. Experimental studies of trust and human intervention in a process control simulation. Ergonomics, 39(3). 429-461. Parasuraman, R., Mouloua, M., & Molly, R. 1996. Effects of adaptive task allocation on monitoring of automated systems. Human factors, 38(4), 665-679. Sheridan, T. B. 1976. Towards a general model of Supervisory Control. In. T. B. Sheridan and G. Johannsen (Eds.), Monitoring Behavior and Supervisory Control, New York: Plenum Press. 271-282. Sheridan, T. B. (1997). .Supervisory control. In G. Salvendy (Ed.) Handbook of human factors (2nd edition). New York: Wiley. 1295-1327. Singh, I. L., Molloy, R., &Parasuraman, R. 1997. Automation induced monitoring inefficiency: role of display location. International Journal of Humn-Coimputer Studies, 46, 17-30.
References 1
From the Greek word for “well trained” Endsley, M. & Kiris, E. O. 1995. The out-of-theloop performance problem and level of control in automation. Human factors, 37(2), 381-394. Lee, J. D. and Moray, N. (1992) Trust, control strategies and allocation of function in human-machine systems. Ergonomics.35, 1243-1270. Lee, J. D. and Moray, N. (1994) Trust, self confidence and operators' adaptation to automation. International Journal of Human- Computer Studies. 40, 153-184. Moray, N., "Monitoring Behavior and Supervisorysory Control," In Handbook of Perception and Human Performance, (eds., K. Boff, L. Kaufmann, and J. Beatty) Wiley, 1986.
