Evaluating Vigilance in a Dynamic Environment - UCF College of ...

PROCEEDINGS of the HUMAN FACTORS and ERGONOMICS SOCIETY 56th ANNUAL MEETING - 2012

1586

Evaluating Vigilance in a Dynamic Environment: Methodological Issues and Proposals Grace W. Teo, James L. Szalma, Tarah N. Schmidt, Gabriella M. Hancock, Peter A. Hancock University of Central Florida Decades of vigilance research have contributed much to our understanding of the factors affecting sustained attention. However most of what we know about vigilance has been from studies employing tasks that involve relatively static stimuli presented on relatively uncluttered backgrounds. This bears little resemblance to many modern day vigilance tasks. The present study discusses the challenges and issues in applying the vigilance paradigm and methodology to a dynamic task requiring vigilance in an IED detection task.

Copyright 2012 by Human Factors and Ergonomics Society, Inc. All rights reserved. DOI 10.1177/1071181312561316

INTRODUCTION Vigilance or the ability to sustain attention over time has been extensively researched in both laboratory and field settings for over sixty years. Several psychophysical dimensions and other task factors have been investigated (for reviews see Davies & Parasuraman, 1982; Warm & Jerison, 1984), but the majority of studies have employed static displays that present stimuli in discrete trials. The stimuli in vigilance studies typically consist of selected letters, numbers, lines, shapes or symbols that may or may not have been degraded or masked. Examples of such tasks include detecting different line lengths (Warm & Dember, 1985); differently-shaded circle pairs (Warm, Finomore, Shaw, Funke, Hausen, Matthews. Taylor, Vidulich, Repperger, Szalma & Hancock (2009); differentiating “O”s, “D”s and backwards “D”s (Shaw, Matthews, Warm, Finomore, Silverman & Costa, 2010; Helton, Shaw, Warm, Matthews & Hancock, 2008); symbols of aircraft “flying along certain paths” (Reinerman-Jones, Matthews, Langheim & Warm, 2010); digit pairs that met a certain criterion (Warm, Howe, Fishbein, Dember, & Sprague, 1984; Szalma & Teo, 2010) among others. While these are relatively easy to obtain and modify for use in programmatic research, they may bear little resemblance to the relevant features of detection tasks in operational settings, such as items in a baggage, defects in products in an assembly line, and improvised explosive devices (IEDs). In addition, as part of the experimental design and control, vigilance tasks usually feature carefully-determined set size or number of distractors, as well as temporal parameters such as a fixed stimulus duration and fixed inter-stimulus intervals (ISI). This enables the manipulation of factors such as event rate, and it is also necessary for the computation of outcome measures (i.e. proportion of hits, false alarms, measures of sensitivity and bias). Training Sustained Attention The importance of maintaining sustained attention during monitoring has led to efforts to develop training regimens to improve performance. The most widely used method for vigilance is the provision of knowledge of results (KR) regarding

detection performance. KR has been found to effectively improve performance during training as well as during subsequent test session in which feedback had been withdrawn (Becker, Warm, & Dember., 1994; Szalma, Miller, Hitchcock, Warm, & Dember, 1999). However, there is also evidence that such transfer of training is specific to task categories (Becker et al., 1994; Szalma et al., 1999). It therefore seems unlikely that a generic task to train for detection of threats in multiple domains will be effective. It is more likely that any task derived for vigilance training will need to incorporate elements representative of the particular operational environment. Traditional vigilance tasks may be ill suited to this purpose. Most of these tasks consist of presentations of single stimulus events against a neutral, uncluttered background in discrete trials. In contrast, many operational settings are dynamic environments in which the flow of information may be continuous and the environment surrounding targets is comprised of a multitude of irrelevant objects. One domain with such characteristics is the detection of IEDs. Soldiers on a mission move through environments in which targets are embedded in a matrix of irrelevant stimuli; the target placement is both spatially and temporally uncertain and thus difficult to predict; and the tactical situation is such that the pace of movement through terrain should not be too low, providing limited time epochs in which to inspect a scene for indicators of IEDs. Vigilance Tasks and Visual Search Vigilance task that employ visual stimuli comprising targets and non-targets/distractors can be thought of as a visual search task sustained over a prolonged period of time, which may result in a decrement of search/detection performance. As with vigilance studies, visual search studies have also largely been based on static stimuli presented for a pre-determined duration with a few exceptions. For example, Kunar and Watson (2011) devised a visual search task that incorporated some elements found in real-world targets. These included motion, avoiding targets that appear and disappear abruptly, and having a complex display. Their results revealed that empiricallybased visual search principles did not apply to the complex displays that were more representative of operational environments.


T These findingss suggest that perhaps vigillance research h may bbenefit from an a alternative paradigm. p Hen nce, a goal forr our reesearch was to develop a traiining module for f vigilance in n the coontext of IED detection usin ng a video gam me-based plat-form.. Empirical su upport for the utility of a dynamic d video o gamee environment for training vigilance v is su upported else-wheree in these procceedings (Teo et e al., 2012). In n this paper wee identiify several issues to be reso olved in the development off such a task, because it differs substantially frrom traditionall tasks used for trainiing sustained atttention.

1587

C Cardboard box iin Location #2

Carddboard box targget

dardization off Stimuli Stand SSelection of tarrgets and targeet locations. An n important featurre of experimen nts is the stand dardization of stimuli s to be presented. Howeverr from reports and a interviewss of military persoonnel, it appearrs that IEDs can n take on many y different formss and these are constantly evo olving. Hence one o of the firstt issuess was determin ning what form m targets would d take. To avoidd confounding vigilance v with target discrimiination per se, vigilaance researcherrs typically select stimuli thatt can be detectedd under alerted d conditions on most trials (e.g., 85-90% of trials)) using a two-aalternative forcced choice proccedure. Hence, an iniitial step in dev veloping a videeo game-based d vigilance task k was to determine wh hich stimuli an nd stimulus loccations were ns on ~90% of trials t in a twodetectable under aleerted condition hoice task. alternnative forced ch A As it was moree important to train t on the ab bility to sustain n attenttion during the search than to train on memory m for an n exem mplar or template, a limited nu umber of targeets were select-ed for the task. A total t of six objjects commonlly encountered d ment were seleected as poten-in a rrural Afghan village environm tial IIED indicatorss: barrel, batteery, bottle, deead dog, rock,, dboard box (seee Figure 1). In addition, to o trashbbag, wire, card furtheer standardize the stimuli, seven s scenes (the ( context off the sttimuli) in the dynamic envirronment were considered ass possibble locations for target placcement. The detectability d off the taargets in the different d locatio ons was tested d under alerted d condiitions in a seriees of pilot studiies. Trashbag in Location L #1

Bottle in Loocation #2

Bottle targget

Rock in Loocation #3

R Rock target

Trashbaag target

Figurre 1. Examplees of targets to be detected inn a rural village scenaario. Note that during the taskk the menu in tthe lower rightthandd corner of eachh image was noot present.


Tablee 1 s (detectab bility of the varrious stimuli in n Resullts of the pilot study the dif ifferent locatio ons)

on of approxi-Wee adopted a crriterion of a rate of detectio matelly 0.9 to indicaate that the target-location paair was not too o difficcult or easy to detect under alerted conditiions. From re-sults of one of the pilot p studies, itt appeared thatt to avoid ceil-s target-ing oor floor effectss with detectaability of the stimuli, locatiion pairs such as “Bottle-Loccation 7”, or “C CardboardBox-Locattion 2” (i.e. haas detectability y as 0.9) should be used (seee Tablee 1). Furthermore, it was alsso found that targets t such ass the baarrel and wire should be avo oided as these were too easy y or diffficult, respectiively, to detect regardless of the t location. Disstractors and Movement M Pacce. Unlike a sttatic laboratory y displaay, the present vigilance task, being on a vid deo game plat-form,, may be situatted in a contin nuously movin ng environmentt compprised of multip ple objects thaat were irrelevaant to the task. Addittionally, in thaat dynamic virtual environmeent, the numberr and tyype of distracttors (i.e. set sizze) would be difficult d to fix. To adddress this, the vigilance task k was made succh that the par-ticipaant had to mo ove through th he same virtuaal environmentt repeaatedly so that the environment would beccome monoto-nous. Participants would w also nott be allowed to control theirr nvironment an nd so are pre-moveement through the virtual en venteed from explo oring the env vironment themselves. Thiss helpeed standardize the entire ex xperience of th he task acrosss particcipants. Annother pilot stu udy was cond ducted to deterrmine pace off moveement. The pacce should not be b too fast such h that detection n wouldd be almost im mpossible or too slow such th hat participantss are abble to eventuallly detect every y target. For th he present task k a pace of 1.8 meterss/second was optimal o for train ning purposes. t placemeent Spatiial aspects of target boratory vigilaance task with static displays,, In a traditional lab there is usually a fixation f point on o which the participant fo-cusess on before each trial. This is to ensure that each trial is ass uniform as possiblee. The dynamicc environment however, pos-pear in the pe-es a ddifferent challeenge as targetss may first app ripherry, but move to o the center off the screen, an nd then out into o the periphery again n as the particiipant walks tho ough the envi--

1588

ronm ment. Moreoveer, when moviing through ann environmentt, the vvisibility (size,, perspective) of objects woould change as a well.. This experiennce of optic fflow should bee considered in n deterrmining target pplacement for the vigilance ttask. Taarget Salience. With the use oof dynamic stiimuli, the back ky grounnd/context agaainst which tarrgets are placeed can be very differrent. To mainntain some coonsistency of target salience acrosss targets, the color/shading of the nearby objects should be cllose to that of the target. Forr instance, placcing the yellow w contaainer near sim milarly light-coolored burlap sacks or lighttcolorred walls was effective for the detection task developed for ttraining vigilaance. The objjective is to develop targeet placeements that doo not capture attention but iinstead to have targeets that are placced in locationns that render tthem detectable by paarticipants if thhey are paying attention. A second issue cconcerning targget salience em merged from the use oof the dynamiic environmennt. Targets hadd to be placed whilee the participaant is moving through the vvirtual environ nmentt, but having taargets appear/ddisappear withhin this continu uous fflow would be unrealistic andd would resultt in a “pop-out” effecct, which woulld render the targets too salient. This waas addreessed by placinng targets behind other objects so that they y are ooccluded whenn first ‘appeariing’ in the virrtual scene, bu ut becom me visible as the participantt moves througgh the environ nmentt (see Figure 2)). Bottle Occluuded

Botttle in view

Figurre 2. Bottle occluded initiallyy, but comes innto view later Tem mporal aspects of target placcement Annother benefit of the strategyy to use occlussion was that iti allow wed the time inn which targetts were visiblee to the particiipant to be calculatted – the stimuulus onset timee. The stimulu us n durattion (time on screen) was tthus defined aas the duration betw ween the targett/stimulus onseet time (i.e., w when it became visibble to the obserrver) and the tim me at which thhe target moved out oof view. Ann additional chhallenge with uusing a dynam mic environmen nt in thhe vigilance tassk was the diffficulty in definning the occurro rencee of an event. This had impplications for ccomputation of the ffalse alarm ratte. In typical vigilance studdies stimuli are preseented in discrette trials that innclude a fixed nnumber of non nsignaal events, so thhat computationn of a false alaarm rate is posssible . In a continuoous video gam me environmennt, however, the numbber of false alaarms committed should be annalyzed, and the


derivaation of signal detection meeasures may not n be possiblee unless a way to com mpute a false allarm rate is dev veloped. mance is typicaally analyzed in n blocks of tri-Viggilance perform als (ooften referred to o as periods on n watch). The demarcation d off blockks is usually so omewhat arbitrrary and is baseed on the mostt conveenient way to ensure equivaalent numbers of events forr both signals and no on-signals with hin each block. Although thiss b used in videeo game-based tasks, a usefull approoach can also be alternnative is to crreate blocks of o trials based d on repetitivee moveement through the scenarios. In order to creeate a task with h the m monotonous character of vigiilance, we creaated a scenario o in whhich participan nts ‘moved’ th hrough an enviironment to an n end ppoint, at which h time they retturned to theirr starting pointt via thhe reverse routee. This movem ment was then repeated r multi-ple times, with each h trip out and back constitutting one block. g., 2 or 3) are recommended d Howeever, multiple repetitions (e.g for eaach block as treating t one reepetition (out and a back) may y provide insufficientt data for stab ble estimates of o change overr time ((see Teo et al.,, 2012).

1589

Target preesent but not clickked on

Clicked on an area with no target

Figurre 3. False alarrm committed as a result of identifying the wronng target (targeet circled in redd). Thus, the obbserver committeed both an erroor of commissioon and one of oomission.

ning a correct//incorrect resp ponse Defin hin a dynamicc Duue to the naturre of the vigilance task with videoo game setting,, participants sh hould be requiired to respond d with a mouse-click k on the locatiion in which they think thee targett was placed. This allows the researcherr to determinee whethher the responsse was actually y to a target or to another partt of thee visual field. Using U a key-strroke response typical t in vigi-lance research wou uld not reveal such information. Howeverr ocation necessiitates delineat-requirring a mouse-cclick on the lo ing aan area around d the target th hat, when click ked, would bee scored as a hit. Thee size of this “cclick- area” wo ould depend on n vement as mou use-click accuracy would bee (i) the pace of mov nt through thee more difficult the faster the pace of movemen n adequate areaa scenaario, and (ii) sizze of the targett. Providing an of acceptable mousse clicks for acccurate respon nding is neces-ounding of targ get detection with w limitationss sary tto avoid confo in pssychomotor accuracy as described by Fittts’ Law (Fitts,, 1954)). Wiith such a “cliick-area”, falsee alarm would d be a mouse-use-click is nott click when no targeet is in view, orr when the mou withinn the click-area of a target, suggesting that the object thatt promp mpted the click was w not a targeet. Misses wou uld then consti-tute ffailure to click within the durration in which h the target wass on sccreen. Note th hat in contrast to previous vigilance v taskss usingg discrete trialss, in continuou us video gamee-based tasks a particcipant can com mmit both a faalse alarm and d a miss in thee same time frame. Fo or instance, it is i possible to click c on an areaa h no target is present p and to not attend to a of a sscene in which targett placed in another area of thee scene (e.g., see Figure 3).

miliarization off targets Fam n Thhe vigilance tassk is typically one that is eassy to do well in whenn one is alert aand attending too the task. Thee discrimination n itselff does not generally require much exposurre for familiariizatioon. When usingg dynamic stim muli, familiariziing participantts on thhe stimuli hadd to include thee movement ccomponent, and thus video clips thhat show a seggment of the eenvironment in n uwhicch the target is placed may acchieve better reesults for stimu lus ffamiliarizationn than using sstatic images of the targetss. Whenn such instrucction was not pprovided the eeffectiveness of o feedbback training uusing the task was reduced ((see Schmidt et e al., 22012). In conclusion, thhe use of a video game platfoorm which pro ovidess a dynamic eenvironment annd stimuli cann enable devellopmeent of vigilancce tasks that reesemble those encountered by y moniitors in operatiional environm ments. This meethod of study ying vvigilance has higher predicttive and conteent validity (aas well as face validitty) and may yyield a more complete underrstandding of how taask characterisstics interplay to affect vigiilancee performance.. However, theere are challengges that accom mpanyy the use of dynnamic stimuli and environmeents; specificallly, thhe loss of conttrol at the leveel achieved in typical discrete trial aapproaches. A Articulating som me of these issuues and propossing ssome possible solutions reppresent efforts taken toward ds bridgging the gap beetween laboratoory studies andd the operation nal env nvironment. Thhe issues to bee considered iin developing a video gameebasedd monitoring taask are summaarized below. • •

•

Pace of movement thrrough the scenaario Numberr of target categgories (should be sufficient to avoid m making the taskk too simple but not so many y that vigiilance effects aare confoundedd with memory y set sizee. Four or fivve targets are recommended based onn the present w work). Placemeent of targets inn scenario


• • • • •

Occlusion of targets Degree of clutter of irrelevant objects in scenes Definitions of period on watch Use of repetitive movement through scenes to stimulate the monotony typical of vigilance tasks Defining correct and incorrect responses for mouse clicks on target areas. ACKNOWLEDGMENT

This work was supported by grant W5J9CQ-11-C-0019 from the Army Research Institute, J.L. Szalma, Principal Investigator. The views expressed in this work are those of the authors and do not necessarily reflect official Army policy. The authors wish to thank Dr. Jennifer Murphy for providing administrative and technical direction as the Technical POC on the grant. REFERENCES Becker, A.B., Warm, J.S., & Dember, W.N. (1994). Specific and nonspecific transfer effects in training for vigilance. In: M. Mouloua & R. Parasuraman (Eds.), Human performance in automated systems: Current trends (pp. 294-299). Hillsdale, NJ: Erlbaum. Broadbent, D.E. (1971). Decision and stress. London: Academic Press. Davies, D.R., & Parasuraman, R. (1982). The psychology of vigilance. London: Academic Press. Dittmar, M.L, Warm, J.S., Dember, W.N. (1985). Effects of knowledge of results on performance in successive and simultaneous vigilance tasks: A signal detection analysis. In: R.E. Eberts and C.G. Eberts (Eds.), Trends in ergonomic/human factors II, (pp.195-202). Amsterdam: Elsevier. Fitts, P.M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 6, 381–391. Helton, W. S., Shaw, T., Warm, J. S., Matthews, G., & Hancock, P. (2008). Effects of warned and unwarned demand transitions on vigilance performance and stress. Anxiety, Stress & Coping: An International Journal, 21, 173-184. Kunar, M.A. & Watson, D.G. (2011). Visual search in a multielement asynchronous dynamic (MAD) world. Journal of Experimental Psychology: Human Perception and Performance, 37, 1017–1031. Reinerman-Jones, L., Matthews, G., Langheim, L. K., & Warm, J. S. (2011). Selection for vigilance assignments: A review and proposed new direction. Theoretical Issues In Ergonomics Science, 12, 273-296. Shaw, T. H., Matthews, G., Warm, J. S., Finomore, V. S., Silverman, L., & Costa, P. r. (2010). Individual differences in vigilance: Personality, ability and states of stress. Journal Of Research In Personality, 44, 297-308. Szalma, J. & Teo, G. (2010). The joint effect of task characteristics and neuroticism on the performance, workload and stress of signal detection. Proceedings of the Human Factors and Ergonomics Society, 54, 1052-1056.

1590

Szalma, J.S., Miller, L.C., Hitchcock, E.M., Warm, J.S., & Dember, W.N.(1999). Intraclass and interclass transfer of training for vigilance. In: M.W. Scerbo and M. Mouloua (Eds.), Automation technology and human performance: Current research and trends (pp. 183-187). Mahwah, NJ: Erlbaum. Teo, G., Schmidt, T.N., Szalma, J.L., Hancock, G.M., Hancock, P.A. (2012. ) The effect of knowledge of results for training vigilance in a video game-based environment. Paper presented at the 56th Human Factors and Ergonomics Society Meeting, Boston, MA. Warm, J.S., Finomore, V., Shaw, T.H., Funke, M.E., Hausen. M.J., Matthews, G., Taylor, P., Vidulich, M.A., Reperger, D.W., Szalma, J.S., & Hancock, P.A. (2009). Effects of training with knowledge of results on diagnosticity in vigilance performance. Proceedings of the Human Factors and Ergonomics Society, 53, 1066-1070. Warm, J.S., Jerison, H.J. (1984). The psychophysics of vigilance. In: J.S. Warm (Ed.), Sustained attention in human performance (pp. 15-59). Chichester United Kingdom: Wiley. Warm, J. S., Howe, S. R., Fishbein, H. D., Dember, W. N., & Sprague, R. L. (1984). Cognitive demand and the vigilance decrement. In: A. Mital (Ed.), Trends in ergonomics/human factors I (pp. 15-20). Amsterdam: Elsevier Science Publishers.