This is a title

Download PDF
8 downloads 0 Views 535KB Size Report
Comment
Bell Street, Dundee. University of Abertay,. Bell Street, Dundee c.martin@abertay.ac.uk l.ball@abertay.ac.uk j.archibald @abertay.ac.uk l.carson@abertay.ac.uk.
7RZDUGVWKHLPSURYHPHQWRIVHOIVHUYLFH V\VWHPVYLDHPRWLRQDOYLUWXDODJHQWV

&KULVWRSKHU0DUWLQ 6FKRRORI&RPSXWLQJ  (QJLQHHULQJ6\VWHPV 8QLYHUVLW\RI$EHUWD\ %HOO6WUHHW'XQGHH [email protected]

/HVOLH%DOO 6FKRRORI&RPSXWLQJ  (QJLQHHULQJ6\VWHPV 8QLYHUVLW\RI$EHUWD\ %HOO6WUHHW'XQGHH [email protected]

-DFTXHOLQH$UFKLEDOG 6FKRRORI&RPSXWLQJ  (QJLQHHULQJ6\VWHPV 8QLYHUVLW\RI$EHUWD\ %HOO6WUHHW'XQGHH j.archibald @abertay.ac.uk

/OR\G&DUVRQ 6FKRRORI6RFLDO +HDOWK 6FLHQFHV 8QLYHUVLW\RI$EHUWD\ %HOO6WUHHW'XQGHH [email protected]

Affective computing and emotional agents have been found to have a positive effect on humancomputer interactions. In order to develop an acceptable emotional agent for use in a self-service interaction, two stages of research were identified and carried out; the first to determine which facial expressions are present in such an interaction and the second to determine which emotional agent behaviours are perceived as appropriate during a problematic self-service shopping task. In the first stage, facial expressions associated with negative affect were found to occur during selfservice shopping interactions, indicating that facial expression detection is suitable for detecting negative affective states during self-service interactions. In the second stage, user perceptions of the emotional facial expressions displayed by an emotional agent during a problematic self-service interaction were gathered. Overall, the expression of disgust was found to be perceived as inappropriate while emotionally neutral behaviour was perceived as appropriate, however gender differences suggested that females perceived surprise as inappropriate. Results suggest that agents should change their behaviour and appearance based on user characteristics such as gender. Keywords: affective computing, virtual agents, emotions, emotion detection, HCI, computer vision, empathy.

1. INTRODUCTION

EXVLQHVV ZLWK WKHP LQ WKH IXWXUH DQG ZKHWKHU WKH\ ZLOOUHFRPPHQGWKHPWRRWKHUSRWHQWLDOXVHUV /LQ  +VLHK  -RKQVRQ HW DO    %\ DGRSWLQJ DIIHFWLYH FRPSXWLQJ SUDFWLFHV LQ 66, GHVLJQ VXFK DV JLYLQJ FRPSXWHUV WKH DELOLW\ WR GHWHFW DQG UHDFW LQWHOOLJHQWO\WRKXPDQHPRWLRQVDQGWRH[SUHVVWKHLU RZQVLPXODWHGHPRWLRQVXVHUH[SHULHQFHVPD\EH LPSURYHG .OHLQ HW DO  -DNVLF HW DO  :DQJHWDO   $IIHFWLYH DJHQWV KDYH EHHQ IRXQG WR UHGXFH IUXVWUDWLRQ GXULQJ KXPDQFRPSXWHU LQWHUDFWLRQV +&,V  .OHLQ HW DO  -DNVLF HW DO   WKHUHIRUHZHDUHLQYHVWLJDWLQJWKHLUHIIHFWLYHQHVVDW LPSURYLQJ QHJDWLYH DIIHFWLYH VWDWHV LQ D 667 XVHU GXULQJDVKRSSLQJVFHQDULR:HSURSRVHDV\VWHP ZKLFKZLOOGHWHFWQHJDWLYHDIIHFWLYHVWDWHVLQDXVHU DQG H[SUHVV DSSURSULDWH HPSDWKHWLF UHDFWLRQV XVLQJDQDIIHFWLYHYLUWXDODJHQW  7ZR VWDJHV RI UHVHDUFK ZHUH LGHQWLILHG  7KH SXUSRVH RI VWDJH  UHSRUWHG LQ 0DUWLQ HW DO LQ SUHVV  ZDV WR LQYHVWLJDWH ZKHWKHU HPRWLRQDO IDFLDO H[SUHVVLRQV DUH SUHVHQW GXULQJ 667 XVH WR GHWHUPLQHZKHWKHUDYLVLRQEDVHGHPRWLRQGHWHFWRU ZRXOGEH VXLWDEOHIRUWKLVV\VWHP7KHSXUSRVH RI VWDJH UHSRUWHGLQ0DUWLQHWDOXQSXEOLVKHG ZDV

7KLV SDSHU GHVFULEHV UHVHDUFK ZKLFK FRQWULEXWHV WRZDUGV WKHGHYHORSPHQWRI DQHPSDWKHWLFV\VWHP ZKLFK ZLOO GHWHFW DQG LPSURYH D XVHU¶V DIIHFWLYH VWDWH GXULQJ D SUREOHPDWLF VHOIVHUYLFH LQWHUDFWLRQ 66,  WKURXJK WKH XVH RI DQ DIIHFWLYH DJHQW  6HOI 6HUYLFH7HFKQRORJLHV 667V DUHWKRVHZKLFKDOORZ DSHUVRQWRREWDLQJRRGVRUVHUYLFHVIURPDUHWDLOHU RU VHUYLFH SURYLGHU ZLWKRXW WKH QHHG IRU DQRWKHU SHUVRQWREHLQYROYHGLQWKHWUDQVDFWLRQ667VDUH XVHGLQPDQ\VLWXDWLRQVLQFOXGLQJKLJKVWUHHWVKRSV VXSHUPDUNHWV DQG WLFNHW NLRVNV  7KH XVH RI 667V PD\ SURYLGH EHQHILWV VXFK DV LPSURYHG FXVWRPHU VHUYLFH IRUH[DPSOHDOORZLQJKRXUDGD\GD\V D ZHHN VHUYLFH  UHGXFHG ODERXU FRVWV DQG LPSURYHG HIILFLHQF\ &KR  )LRULWR    /HVV WKDQ  RI FDXVHV IRU GLVVDWLVIDFWLRQ ZLWK 667 LQWHUDFWLRQV ZHUH IRXQG WR EH WKH IDXOW RI WKH FXVWRPHU 0HXWHU HW DO  3XMDUL   LQGLFDWLQJ WKDW WKHUH LV D QHHG IRU EXVLQHVVHV DQG 667PDQXIDFWXUHUVWRLPSURYHWKHVHLQWHUDFWLRQVLQ RUGHUWRUHGXFHFDXVHVIRUGLVVDWLVIDFWLRQ 0DUWLQHW DO XQSXEOLVKHG   7KH IUXVWUDWLRQ FDXVHG E\ D QHJDWLYH 66, FDQ KDYH D GHWULPHQWDO HIIHFW RQ D XVHU¶V EHKDYLRXUDO LQWHQWLRQV WRZDUGV WKH UHWDLOHU LPSDFWLQJWKHOLNHOLKRRGWKHXVHUZLOOFRQWLQXHGRLQJ ‹7KH$XWKRUV3XEOLVKHGE\%,6/ 3URFHHGLQJVRIWKH%&6+&, 3HRSOH &RPSXWHUV;;9,%LUPLQJKDP8.



Work In Progress

Towards the improvement of self-service systems via emotional virtual agents C. Martin, L. Ball, J. Archibald, L. Carson

WR JDWKHU RSLQLRQV RQ WKH DSSURSULDWHQHVV RI VHYHUDOHPRWLRQDOIDFLDOH[SUHVVLRQVZKLFKPD\EH H[SUHVVHGE\WKHYLUWXDODJHQWGXULQJD66,

RUV\QWKHVLVHGVSHHFK$IIHFWLYHYLUWXDODJHQWVDUH WKRVH ZLWK DIIHFWLYH DELOLWLHV LQFOXGLQJ DIIHFW GHWHFWLRQ DQG HPRWLRQDO H[SUHVVLYHQHVV 3LFDUG -DNVLFHWDO3HWHUVHWDO 7KH XVH RI DIIHFWLYH DJHQWV LQ +&,V KDV SURGXFHG HQFRXUDJLQJ ILQGLQJV IRU WKH LPSURYHPHQW RI WKHVH LQWHUDFWLRQV LQFOXGLQJ FDXVLQJ D XVHU WR LQWHUDFW ORQJHU ZLWK HUURUSURQH .OHLQ HW DO   RU GLIILFXOW WDVNV :DQJ HW DO   DQG UHGXFLQJ IUXVWUDWLRQ -DNVLF HW DO    :KLOH WKHVH ILQGLQJV LQGLFDWH SRWHQWLDO EHQHILWV RI DIIHFWLYH DJHQWVWKHVWXGLHVE\.OHLQHWDO  DQG-DNVLF HWDO  UHOLHGRQD:L]DUGRI2]PHWKRGRORJ\ ZKHUH WKH UHVHDUFKHUV PDQLSXODWHG WKH DJHQW ³EHKLQG WKH VFHQHV´ WR VLPXODWH DXWRPDWHG EHKDYLRXU  7KH VWXG\ E\ :DQJ HW DO   UHTXLUHG XVHUV WR PDQXDOO\ VHOIUHSRUW WKHLU HPRWLRQDOVWDWH2XUSURSRVHGHPSDWKHWLFV\VWHP ZLOOEHIXOO\DXWRPDWLF  7KH DSSURSULDWHQHVV RI DQ DJHQW¶V HPRWLRQDO EHKDYLRXU IRU WKH VLWXDWLRQ LQ ZKLFK LW LV XVHG KDV EHHQIRXQGWRDIIHFWWKHDJHQW¶VEHOLHYDELOLW\ /LP  $\OHWW WKHGHJUHHRIWUXVWWKHXVHUKDVLQWKH DJHQW &UDPHU HW DO   DQG WKH XVHU¶V VWUHVV OHYHO ZLWK LQDSSURSULDWH HPRWLRQDO EHKDYLRXU LQGXFLQJVWUHVV %HFNHUHWDO ,IRXUV\VWHP LV WR EH VXFFHVVIXO LQ UHGXFLQJ QHJDWLYH DIIHFWLYH VWDWHV LQ XVHUV GXULQJ IUXVWUDWLQJ LQWHUDFWLRQV WKH HPRWLRQDO EHKDYLRXU RI WKH DJHQW PXVW EH DFFHSWDEOH WR XVHUV  6LQFH DSSURSULDWHQHVV RI HPRWLRQDO EHKDYLRXU YDULHV ZLWK VLWXDWLRQ D VWXG\ ZDV FDUULHG RXW WR HYDOXDWH D UDQJH RI HPRWLRQDO IDFLDO H[SUHVVLRQV WR GHWHUPLQH ZKHWKHU DQ\ RI WKHPZHUHDFFHSWDEOHLQDSUREOHPDWLFVHOIVHUYLFH VKRSSLQJ WDVN  $FFHSWDEOH HPRWLRQV ZRXOG EH LQFOXGHG LQ RXU HPSDWKHWLF V\VWHP ZKLOH LQDSSURSULDWHHPRWLRQVZRXOGEHH[FOXGHG

2. LITERATURE REVIEW 6HYHUDO VWXGLHV KDYH IRFXVVHG RQ KDYLQJ FRPSXWHUV UHFRJQLVH HPRWLRQV IURP IDFLDO H[SUHVVLRQV (VVD  3HQWODQG  7LDQ HW DO 9DOVWDU 3DQWLF.RWVLD 3LWDV .XPDQRHWDO/XFH\HWDO6DWL\DQ  1DJDUDMDQ    $V UHSRUWHG LQ 0DUWLQ HW DO LQ SUHVV LQVWDJHRIWKLVUHVHDUFKZHH[DPLQHGWKH IDFLDO $FWLRQ 8QLWV $8V  GLVSOD\HG GXULQJ 66,V $8VDUHGHVFULSWRUVIRUWKHPRYHPHQWRILQGLYLGXDO IDFLDO IHDWXUHV ZKLFK FRPELQH WR IRUP IDFLDO H[SUHVVLRQV .DQDGH HW DO    3URWRW\SLFDO GHILQLWLRQV RI $8 DFWLYDWLRQ SDWWHUQV KDYH EHHQ LGHQWLILHG IRU HDFK RI (NPDQ¶V   EDVLF HPRWLRQVDQJHUGLVJXVWIHDUKDSSLQHVVVDGQHVV DQG VXUSULVH /XFH\ HW DO    ,Q RUGHU WR GHYHORS DQ DIIHFW GHWHFWLRQ V\VWHP IDFLDO H[SUHVVLRQV ZKLFK RFFXU GXULQJ 667 LQWHUDFWLRQV QHHG WR EH LGHQWLILHG VR WKDW D GHWHFWRU FDQ EH WUDLQHG  :H LQWHQG WR XVH +DDUOLNH IHDWXUH H[WUDFWLRQ (VVD  3HQWODQG  .XPDQR HW DO  6DWL\DQ  1DJDUDMDQ   WR GHWHFW IDFLDO H[SUHVVLRQV GXH WR LWV HDVH RI WUDLQLQJ DQG LPSOHPHQWDWLRQ  6RFLDODQGHPRWLRQDOLQWHOOLJHQFHWKHDELOLWLHVRIDQ LQGLYLGXDO WR GHDO UDWLRQDOO\ DQG HIIHFWLYHO\ ZLWK VRFLDO VLWXDWLRQV DQG WKH HPRWLRQV RI RWKHUV 6DORYH\  0D\HU  3DQWLF  5RWKNUDQW]  3DQWLF   KDYH EHHQ DUJXHG WR EH LPSRUWDQWIRUVXFFHVVIXOVRFLDOIXQFWLRQLQJ 6DORYH\  0D\HU    1DVV HW DO   VKRZHG WKDW KXPDQVDSSO\WKHVDPHVRFLDOUXOHVWRLQWHUDFWLRQV ZLWKFRPSXWHUVDVWKH\GRWRLQWHUDFWLRQVZLWKRWKHU KXPDQV,ISHRSOHKDYHVLPLODUVRFLDOH[SHFWDWLRQV IRU FRPSXWHUV DV WKH\ GR IRU RWKHU KXPDQV DQ LQWHUDFWLRQZLWKDV\VWHPZKRVHSHUFHLYHGVRFLDORU HPRWLRQDOLQWHOOLJHQFHIDOOVEHORZWKHLUH[SHFWDWLRQV PD\SURYHXQVDWLVIDFWRU\7KHUHIRUHLWLVLPSRUWDQW WKDW D FRPSXWHU¶V SHUFHLYHG VRFLDO DQG HPRWLRQDO LQWHOOLJHQFHLVDFFHSWDEOHWRXVHUV 0DUWLQHW DO LQ SUHVV3DQWLF3LFDUG3LFDUG   ,Q RUGHU WR SURJUHVV WRZDUGV WKH SURSRVHG HPSDWKHWLF VHOIVHUYLFH V\VWHP VWDJH  RI RXU UHVHDUFK LQYROYHG GHWHUPLQLQJ ZKHWKHU DQ\ HPRWLRQDO EHKDYLRXUV DUH IRXQG WR EH DSSURSULDWH RU LQDSSURSULDWH GXULQJ D SUREOHPDWLF 66, $SSURSULDWH EHKDYLRXUV ZLOO EH LQFOXGHG LQ WKH DELOLWLHV RI RXU HPRWLRQDO DJHQW ZKLOH LQDSSURSULDWH EHKDYLRXUVZLOOEHH[FOXGHG  9LUWXDO DJHQWV DUH SHUVRQDV LQYROYHG LQ D +&, ZKLFK LQWHUDFW ZLWK D XVHU YLD D FRPELQDWLRQ RI D JUDSKLFDOUHSUHVHQWDWLRQSULQWHGWH[WDQGUHFRUGHG

3. METHOD ,QRUGHUWRLGHQWLI\$8VZKLFKRFFXUGXULQJ66,VLQ VWDJH  FXVWRPHUV DW D VXSHUPDUNHW ZHUH ILOPHG XVLQJ VHOIVHUYLFH FKHFNRXWV  &XVWRPHUV ZHUH DSSURDFKHG ZKHQ WKH\ ZHUH TXHXLQJ DQG DVNHG ZKHWKHU WKH\ ZRXOG OLNH WR EH ILOPHG ZKLOH WKH\ VFDQQHG WKHLU VKRSSLQJ  7KRVH ZKR DJUHHG ZHUH GLUHFWHG WR D VHOIVHUYLFH FKHFNRXW ZKHUH D YLGHR FDPHUDZDVVHWXSRQDWULSRG7KHFDPHUDILOPHG WKH SDUWLFLSDQW¶V KHDG DQG VKRXOGHUV IRU WKH GXUDWLRQRIWKHLULQWHUDFWLRQ  7KLUW\ SDUWLFLSDQWV WRRN SDUW LQ WKH VWXG\  PDOH  IHPDOH   ,Q RUGHU WR PLQLPLVH GLVUXSWLRQ WR FXVWRPHUV DQG WR LQFUHDVH WKH DSSHDO RI WDNLQJ SDUW SDUWLFLSDQWV ZHUH QRW DVNHG DQ\ TXHVWLRQV RU UHTXLUHG WR SHUIRUP DQ\ DGGLWLRQDO WDVN DSDUW IURP UHDGDQGVLJQWKHLQIRUPHGFRQVHQWIRUP$OWKRXJK LW PD\ KDYH EHHQ XVHIXO WR JDWKHU GHPRJUDSKLF GDWD LQIRUPDWLRQ DERXW SDUWLFLSDQWV¶ WHFKQRORJLFDO

352



Towards the improvement of self-service systems via emotional virtual agents C. Martin, L. Ball, J. Archibald, L. Carson

EDFNJURXQG DQG WKHLU FXUUHQW DQG SUHYLRXV H[SHULHQFHV ZLWK 667V WKLV VWXG\¶V PDLQ DLP ZDV VLPSO\ WR FROOHFW GDWD RQ HPRWLRQDO IDFLDO H[SUHVVLRQV IURP DV ODUJH D JURXS RI SDUWLFLSDQWV DV SRVVLEOH WKHUHIRUH WKH UHVHDUFKHUV ZDQWHG WR PD[LPLVH WKH DSSHDO RI WDNLQJ SDUW  7KH FROOHFWHG IRRWDJH ZDV YLHZHG E\ D UHVHDUFKHU DQG IDFLDO H[SUHVVLRQV ZHUH PDQXDOO\ LGHQWLILHG 7KH IRRWDJH ZDVGLYLGHGLQWRFOLSV±VKRUWVHTXHQFHVRIIRRWDJH FRQWDLQLQJ HLWKHU D IDFLDO H[SUHVVLRQ RU D QHXWUDO IDFH  &OLSV FRQWDLQLQJ D QHXWUDO IDFH ZHUH GLVFDUGHGDQGWKHUHPDLQGHUZHUHDQDO\VHGE\WKH UHVHDUFKHU LQ RUGHU WR GHWHUPLQH ZKLFK $8V ZHUH SUHVHQW DW SHDN HPRWLRQDO H[SUHVVLRQ ([SUHVVLRQVH[KLELWHGDVDUHVXOWRILQWHUDFWLQJZLWK D PHPEHU RI VWDII FRPSDQLRQ RU WKH UHVHDUFKHU ZHUH GLVFDUGHG DV WKH\ ZHUH QRW HOLFLWHG E\ WKH 667 GHYLFH  ,W LV SRVVLEOH WKDW VRPH $8V ZHUH H[KLELWHG E\ WKH SDUWLFLSDQWV EXW QRW REVHUYHG E\ WKH UHVHDUFKHU GXH WR IDFWRUV LQFOXGLQJ WKH UHVROXWLRQRIWKHUHFRUGHGIRRWDJH [SL[HOV  WKH OLJKWLQJ LQ WKH VFHQH DQG SDUWLFLSDQW IHDWXUHV VXFK DV KDLU VW\OHV JODVVHV KDWV VFDUYHV DQG IDFLDOKDLU  7KH VWXG\ FDUULHG RXW GXULQJ VWDJH  UHTXLUHG SDUWLFLSDQWV WR DWWHPSW D FRPSXWHUEDVHG VHOI VHUYLFH VKRSSLQJ WDVN  7KH WDVN ZDV VLPLODU WR PRVWRQOLQHVKRSSLQJH[SHULHQFHVLQYROYLQJDGGLQJ LWHPV WR D YLUWXDO VKRSSLQJ EDVNHW  7ZHQW\VL[ SDUWLFLSDQWV WRRN SDUW  PDOHV  IHPDOHV  ZLWK DQDYHUDJHDJHRI\HDUV(WKLFDODSSURYDOZDV REWDLQHG IURP WKH 8QLYHUVLW\ RI $EHUWD\ 6FKRRO RI &RPSXWLQJ DQG (QJLQHHULQJ 6\VWHPV 5HVHDUFK (WKLFV 6XE&RPPLWWHH  7KH VWXG\ KDG D ZLWKLQ VXEMHFWV GHVLJQ ZLWK HDFK SDUWLFLSDQW EHLQJ VXEMHFWHG WR WKH VDPH FRQGLWLRQV 3DUWLFLSDQWV LQWHUDFWHGZLWKDFRPSXWHUSURJUDPZKLFKUHTXLUHG WKHP WR FRPSOHWH D VKRSSLQJ WDVN VLPLODU WR PRVW RQOLQHVKRSSLQJVFHQDULRVZKLFKXWLOLVHD³VKRSSLQJ EDVNHW´ 3DUWLFLSDQWV UHFHLYHG RQVFUHHQ LQVWUXFWLRQV WR DGG YDULRXV LWHPV IURP D OLVW LQWR D YLUWXDOVKRSSLQJEDVNHW$'DQLPDWHGDJHQW ZDV SUHVHQWRQWKHVFUHHQGXULQJWKHWDVNGLVSOD\LQJD ORRSHG DQLPDWLRQ RI HPRWLRQDOO\ QHXWUDO EHKDYLRXU 3DUWLFLSDQWV ZHUH DVNHG WR DWWHPSW WR FRPSOHWH DOO WDVNVZLWKWKHRSWLRQRIFOLFNLQJDQRQVFUHHQ³TXLW´ EXWWRQLIWKH\IHOWWKH\FRXOGQRWFRPSOHWHDWDVN  ,Q RUGHU WR VLPXODWH D UHDOZRUOG 667 LQWHUDFWLRQ ZKHUH DQ HYHQW PDNHV LW LPSRVVLEOH IRU D XVHU WR DFKLHYH WKHLU JRDO ZLWKRXW WKH DVVLVWDQFH RI D PHPEHU RI VWDII WKH SURJUDP SXUSRVHIXOO\ REVWUXFWHGWKHXVHUPDNLQJWKHWDVNLPSRVVLEOHWR FRPSOHWH  7KLV ZDV DFKLHYHG E\ FDXVLQJ WKH SURJUDP WR DGG LQFRUUHFW LWHPV WR WKH VKRSSLQJ EDVNHW DIWHU D SUHVHW QXPEHU RI DFWLRQV KDG EHHQ FDUULHG RXW E\ WKH XVHU  ,I SDUWLFLSDQWV DVNHG WKH UHVHDUFKHU IRU DVVLVWDQFH WKH\ ZHUH DGYLVHG WR FOLFN WKH ³TXLW´ EXWWRQ LI WKH IHOW WKDW WKH\ ZHUH XQDEOH WR FRPSOHWH WKH WDVN  $IWHU WKH TXLW EXWWRQ

ZDV SUHVVHG SDUWLFLSDQWV ZHUH SUHVHQWHG ZLWK D VFUHHQ GLVSOD\LQJ DQ DUUD\ RI VHYHQ DQLPDWHG IDFHV HDFK GLVSOD\LQJ HLWKHU DQJHU GLVJXVW IHDU KDSSLQHVV VDGQHVV VXUSULVH RU WKH HPRWLRQDOO\ QHXWUDO EHKDYLRXU H[SUHVVHG GXULQJ WKH WDVN  7KH IDFHV ZHUH DUUDQJHG UDQGRPO\ DQG DQLPDWLRQV ODVWHGIRUIRXUVHFRQGVEHIRUHEHLQJUHSHDWHG  3DUWLFLSDQWVXVHGRQVFUHHQSRLQW/LNHUWVFDOHVWR UHFRUG KRZ DSSURSULDWH WKH\ IHOW HDFK HPRWLRQDO H[SUHVVLRQZDVLQWKHFXUUHQWVLWXDWLRQ7KHSRLQWV RI HDFK /LNHUW VFDOH ZHUH ODEHOOHG LQ RUGHU DV ³FRPSOHWHO\ LQDSSURSULDWH´ ³VOLJKWO\ LQDSSURSULDWH´ ³QHLWKHU DSSURSULDWH QRU LQDSSURSULDWH´ ³VOLJKWO\ DSSURSULDWH´ RU ³FRPSOHWHO\ DSSURSULDWH´ 3DUWLFLSDQWVZHUHWKHQSUHVHQWHGZLWKDQRQVFUHHQ TXHVWLRQQDLUH JDWKHULQJ GHWDLOV RI WKHLU DJH DQG JHQGHUDVZHOODVDIUHHIRUPZULWLQJVHFWLRQZKHUH WKH\FRXOGVXJJHVWFKDQJHVWKH\ZRXOGOLNHWRVHH PDGH WR WKH DJHQW LQ RUGHU WR PDNH LW PRUH DFFHSWDEOH  7KH HPRWLRQDOO\ H[SUHVVLYH DJHQW XVHG LQ WKLV VWXG\ ZDV GHYHORSHG E\ 6ORDQ HW DO    ,WV HPRWLRQDO EHKDYLRXUV FRQVLVWHG RI DQLPDWLRQV RI WKH VL[ EDVLF XQLYHUVDO HPRWLRQV (NPDQ   7KHVH DQLPDWLRQV KDG SUHYLRXVO\ EHHQ YDOLGDWHG 6ORDQ HW DO   WR FRQILUP WKDW HDFK UHOLDEO\ SRUWUD\HG WKH HPRWLRQ LW SXUSRUWHG WR  7KH EDVLF XQLYHUVDO HPRWLRQV ZHUH FKRVHQ IRU WKLV VWXG\ DV HDFK KDV D FRUUHVSRQGLQJ IDFLDO H[SUHVVLRQ ZKLFK LV XQLYHUVDOO\ SURGXFHG DQG UHFRJQLVHG E\ SHRSOH RI DOO FXOWXUHV (NPDQ    +XPDQV DUJXDEO\ KDYH D PXFK ZLGHU UDQJH RI HPRWLRQV HJ UDJH IUXVWUDWLRQDQGFRQWHQWPHQW KRZHYHUWKHVHGRQRW KDYHXQLYHUVDOO\UHFRJQLVHGIDFLDOH[SUHVVLRQVDQG ZHUHWKHUHIRUHXQVXLWDEOHIRUWKLVVWXG\ 4. RESULTS ,QWKHVWDJHUHVHDUFKLQWHUDFWLRQVZHUHILOPHG DQG DQDO\VHG ZLWK DQ DYHUDJH OHQJWK RI  VHFRQGV  7KH DFWLYDWLRQ RI  GLIIHUHQW $8V ZDV REVHUYHG LQ WKH YLGHR FOLSV  $ FKLVTXDUH JRRGQHVVRIILW WHVW ZDV FDUULHG RXW WR GHWHUPLQH ZKHWKHU WKH IUHTXHQFLHV RI RFFXUUHQFHV IRU HDFK $8 GHYLDWHG VLJQLILFDQWO\ IURP WKH H[SHFWHG GLVWULEXWLRQ7KLVZDVIRXQGWREHWKHFDVH ;  Q   S ZLWK$8DQG$8  RFFXUUHG DW OHDVW WZLFH DV IUHTXHQWO\ DV DQ\ RWKHU$8 )LJXUH   ,Q WKH VWDJH  VWXG\ UHVXOWV IURP DJHQW DSSURSULDWHQHVV /LNHUW VFDOHV ZHUH FRPELQHG IURP  WR  FDWHJRULHV ,QDSSURSULDWH 1HLWKHU $SSURSULDWH 1RU ,QDSSURSULDWH DQG $SSURSULDWH )LJXUH    &KL6TXDUH JRRGQHVVRIILW WHVWV ZHUH FDUULHG RXW WR DVVHVV UHVSRQVHV WR HDFK RI WKH DQLPDWHGH[SUHVVLRQV$SSURSULDWHQHVVUDWLQJVIRU WKH µGLVJXVW¶ DQG HPRWLRQDOO\ QHXWUDO IDFLDO

353



Towards the improvement of self-service systems via emotional virtual agents C. Martin, L. Ball, J. Archibald, L. Carson

H[SUHVVLRQV ZHUH IRXQG WR GHYLDWH VLJQLILFDQWO\ IURPWKHH[SHFWHGGLVWULEXWLRQ 

3DUWLFLSDQWV PDGH VHYHUDO VXJJHVWLRQV DERXW LPSURYHPHQWV ZKLFK FRXOG EH PDGH WR WKH DJHQW WKH PDMRULW\ RI ZKLFK UHODWHG WR WKH DJHQW¶V DSSHDUDQFH LQFOXGLQJ FRPPHQWV VXJJHVWLQJ WKH DJHQWVKRXOGFKDQJHLWVDJHDQGJHQGHUEDVHGRQ WKH XVHU¶V DJH DQG JHQGHU DOWKRXJK LW ZDV QRW VSHFLILHG ZKHWKHU WKHVH VKRXOG EH FKDQJHG WR PDWFKRUFRQWUDVWWKHXVHU¶VJHQGHUDQGDJH 



Occurrences







5. DISCUSSION





,QVWDJHDFROOHFWLRQRIYLGHRFOLSVZDVJDWKHULQJ VKRZLQJ SHRSOH VSRQWDQHRXVO\ H[SUHVVLQJ IDFLDO H[SUHVVLRQVGXULQJD66,$8DQG$8ZHUH REVHUYHG ZLWK SDUWLFXODU IUHTXHQF\  :KHQ GLVSOD\HG VLPXOWDQHRXVO\ WKHVH $8V KDYH EHHQ OLQNHG WR WKH H[SHULHQFH RI DQJHU .RWVLD  3LWDV  /XFH\ HW DO  5R]LQ HW DO   ZKLOH $8  DORQH LV DOVR DVVRFLDWHG ZLWK WKLV HPRWLRQ &DUURO 5XVVHOO6FKPLGWHWDO   :H SURSRVH WKDW WKH DSSHDUDQFH RI WKHVH $8V LQ WKH FROOHFWHG IRRWDJH LQGLFDWHV D QHJDWLYH HPRWLRQDO VWDWH RFFXUULQJ LQ WKHVH 667 XVHUV 7KHVH $8V ZHUH XVHG WR WUDLQ D +DDUOLNH IHDWXUH GHWHFWLRQ V\VWHP KRZHYHU WKH SHUIRUPDQFH RI WKH V\VWHPPDGHLWXQVXLWDEOHIRUXVH  7KHUHVXOWVRIVWDJHLQGLFDWHWKDWIRUDQDIIHFWLYH DJHQW LQ D 66, GLVJXVW ZDV UDWHG DV DQ LQDSSURSULDWH IDFLDO H[SUHVVLRQ ZLWK VLJQLILFDQWO\ KLJKHU IUHTXHQF\ WKDQ H[SHFWHG ZKLOH DQ HPRWLRQDOO\ QHXWUDO H[SUHVVLRQ ZDV UDWHG DV DSSURSULDWHZLWKVLJQLILFDQWO\KLJKHUIUHTXHQF\WKDQ H[SHFWHG  7KLV VXJJHVWV WKDW GLVJXVW LV DQ XQVXLWDEOH HPRWLRQ WR EH H[SUHVVHG GXULQJ D SUREOHPDWLF 667 LQWHUDFWLRQ ZKLOH DQ HPRWLRQDOO\ QHXWUDOH[SUHVVLRQPD\EHSUHIHUDEOHWRRWKHUEDVLF HPRWLRQV  ,W ZDV IRXQG WKDW IHPDOHV UDWHG VXUSULVH DV LQDSSURSULDWH ZLWK VLJQLILFDQWO\ KLJKHU IUHTXHQF\ WKDQ H[SHFWHG  6LJQLILFDQW GLIIHUHQFHV ZHUH IRXQG EHWZHHQ PDOHV¶ DQG IHPDOHV¶ DSSURSULDWHQHVV UDWLQJV IRU IHDU KDSSLQHVV VDGQHVV DQG VXUSULVH VXJJHVWLQJ WKDW DQ DJHQW¶V EHKDYLRXU VKRXOG YDU\ GHSHQGLQJ RQ WKH XVHU¶V JHQGHU LQ RUGHU WR EH SHUFHLYHG DV DSSURSULDWH  3DUWLFLSDQWV DOVR VXJJHVWHGWKDWWKHDJHQWVKRXOGEHDEOHWRFKDQJH LWV DJH DQG JHQGHU GHSHQGLQJ RQ WKDW RI WKH XVHU DOWKRXJK SDUWLFLSDQWV GLG QRW VSHFLI\ ZKHWKHU WKH\ ZRXOG SUHIHU DQ DJHQW RI WKHLU RZQ RU RSSRVLWH JHQGHURUWKHLURZQRUDGLIIHUHQWDJH  7KH UHVXOWV VXJJHVW WKDW QRQH RI WKH EDVLF HPRWLRQV ZHUH FRQVLGHUHG DSSURSULDWH E\ DOO SDUWLFLSDQWV  7KLV FRXOG EH GXH WR SDUWLFLSDQWV QRW ZDQWLQJWRVHHPRUHFRPSOH[HPRWLRQV IUXVWUDWLRQ LPSDWLHQFH FRQFHUQ HWF  SDUWLFLSDQWV ZDQWLQJ WR VHH OHVV H[WUHPH PDQLIHVWDWLRQV RI WKH EDVLF



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Facial Action Unit

 Figure1: Occurrences of AUs observed in SST users

(PRWLRQV ZHUH UDWHG DV 1HLWKHU $SSURSULDWH 1RU ,QDSSURSULDWH E\ D PLQRULW\ RI SDUWLFLSDQWV 7KH PDMRULW\ RI SDUWLFLSDQWV UDWHG HPRWLRQV DV HLWKHU ,QDSSURSULDWH RU $SSURSULDWH :KHQ UHVSRQVHV IRU PDOHV DQG IHPDOHV ZHUH H[DPLQHG LQGLYLGXDOO\ LW ZDV IRXQG WKDW WKH GLVWULEXWLRQ IRU VXUSULVH LQ IHPDOHV YDULHG VLJQLILFDQWO\ IURP WKH H[SHFWHG GLVWULEXWLRQ ;  Q      S    LQGLFDWLQJ WKDW IHPDOHV IRXQG VXUSULVH LQDSSURSULDWH Participant Ratings

Response Frequency

16 14 12 10 8 6 4 2 0

Anger

Disgust

Fear

Happiness

Neutral

Sadness

Surprise

Figure2: Combined appropriateness responses to emotional expressions. Black = inappropriate, grey = neither, white = appropriate

7KH IUHTXHQF\ RI HDFK UDWLQJ ZDV FRPSDUHG EHWZHHQPDOHVDQGIHPDOHV6LJQLILFDQWYDULDWLRQV ZHUH IRXQG LQ WKH GLVWULEXWLRQV RI HDFK UDWLQJ EHWZHHQPDOHVDQGIHPDOHVLQGLFDWLQJWKDW x 0RUH IHPDOHV UDWHG VXUSULVH DV LQDSSURSULDWH x 0RUHPDOHVUDWHGVXUSULVHDVDSSURSULDWH x 0RUHPDOHVUDWHGIHDUDVLQDSSURSULDWH x 0RUHIHPDOHVUDWHGIHDUDVQHLWKHU x 0RUHIHPDOHVUDWHGKDSSLQHVVDVQHLWKHU x 0RUHIHPDOHVUDWHGVDGQHVVDVQHLWKHU

354



Towards the improvement of self-service systems via emotional virtual agents C. Martin, L. Ball, J. Archibald, L. Carson

HPRWLRQV RU SDUWLFLSDQWV QRW ZDQWLQJ WR EH SUHVHQWHG ZLWK DQ\ DJHQW HPRWLRQV GXULQJ DQ 66, &RPSOH[HPRWLRQVPD\EHGLIILFXOWWRLPSOHPHQWDV WKHUH LV D ODFN RI OLWHUDWXUH RQ UHOLDEO\ UHFRJQLVHG IDFLDOH[SUHVVLRQVIRUWKHVHHPRWLRQV

1DUD-DSDQ±0DUFKSS,(((3UHVV 3LVFDWDZD\1-86$  (NPDQ 3 HW DO   8QLYHUVDOV DQG FXOWXUDO GLIIHUHQFHV LQ WKH MXGJPHQWV RI IDFLDO H[SUHVVLRQV RI HPRWLRQ -RXUQDO RI 3HUVRQDOLW\ DQG 6RFLDO 3V\FKRORJ\  SS±  (VVD , $ 3HQWODQG $ 3   &RGLQJ $QDO\VLV ,QWHUSUHWDWLRQ DQG 5HFRJQLWLRQ RI )DFLDO ([SUHVVLRQV ,((( 7UDQVDFWLRQV 2Q 3DWWHUQ $QDO\VLV$QG0DFKLQH,QWHOOLJHQFH  SS±   -DNVLF 1 %UDQFR 3 6WHSKHQVRQ 3 (QFDUQDomR / 0   7KH HIIHFWLYHQHVV RI VRFLDO DJHQWV LQ UHGXFLQJ XVHU IUXVWUDWLRQ &+, ¶ ([WHQGHG $EVWUDFWV RQ +XPDQ )DFWRUV LQ &RPSXWLQJ 6\VWHPV 0RQWUpDO &DQDGD ± $SULOSS$&01HZ