Multimedia stimuli databases usage patterns: a ...

3 downloads 17910 Views 608KB Size Report
1. Stimulus generator (StimGen). 2. Emotionally annotated image database heuristic builder. Motivation: Development of custom software tools for multimodal ...
Multimedia stimuli databases usage patterns: a survey report

Dissemination of the survey results Marko Horvat Department of Computer Science and Information Technology University of Applied Sciences Zagreb, Croatia 1

Motivation: Shortcomings of contemporary emotionally annotated databases (1) • Emotionally annotated databases are often used in research of emotional reactions, affective states, attention, anxiety, phobias, stress resilience, cognitive capacities, etc. yet little effort has been made to facilitate and expand their use. • Even today stimuli from these databases are extracted manually by laborious visual examination of each stimulus and accompanying manuals. Equally so, stimuli sequence construction is often demanding and timeconsuming task.

2

Motivation: Shortcomings of contemporary emotionally annotated databases (2) • To complicate their usage even further, emotionally annotated databases have diverse structures, describe emotional and semantic data differently and contain various media formats and stimuli modalities. There is no consensus among researchers in an optimal format or implementation of a database. For these reasons a database supervisor has to be a well-trained domain expert proficient in emotion elicitation and in totally separate technology-related tasks such as multimedia annotation and retrieval, or sequence construction and delivery. Such heterogeneous fusion of skills is difficult to master and can only be applied to a single specific database. 3

Motivation: Intelligent generators of stimuli sequences • All this outlines the need for an intelligent stimuli generator computer system which can assist experts in finding the most appropriate stimulus in a number of different applications and to do it in the shortest time possible. Such system must be as universal as possible towards various database and media formats, efficient and user-friendly. Furthermore, the system should have built in heuristic empirically derived rules for decision support and automatic generation of stimuli sequences. Also, the intelligent generator should support personalized stimulation individually tailored to specific emotional and semantic parameters. 4

Motivation: Development of custom software tools for multimodal stimulation • The author of the survey has developed a number of afore described software tools designed to assist experts in many exposure strategies ranging from CCBT, MRT, SIT to Cognitive Bias Modification and other novel procedures. These tools allow practitioners to focus more on the subject/trainee and the elicitation procedure itself thereby increasing the quality and control of the process. – Some tools in the context of this survey (briefly presented in the next two slides) are: 1. Stimulus generator (StimGen) 2. Emotionally annotated image database heuristic builder 5

Stimulus Generator (StimGen) •Generates audio-visual stimuli sequences using IAPS and IADS •Simple point-and-click interface with history allows fast stimuli discovery •Rich visualization: adaptable sequence time graphs, val/ar plane trajectories, SAMs, etc. •Supports congruent and concurrent stimuli •Image retrieval along various data dimensions: semantics, emotion, picture set, etc. •Independent management of IAPS and IADS semantics •Converts subsets of IAPS/IADS to ARFF format for data mining in Weka •Supports management of custom tag clouds of IAPS and IADS keywords allowing grouping of semantically similar stimuli •Full data backup and restore functionality •Multiple monitors for operator and subject HMD support •Deploys on a single desktop or mobile computer 6

Emotionally annotated image database heuristic builder •Batch creation of customized IAPS-compliant emotionally annotated databases •Uses predefined semantics and/or emotion constraints •Supports image annotation with formal ntologies and RDF triplets •Randomized selection and sorting of selected stimuli •Heuristic rules for filtering of unwished stimuli •Simple interface •MATLAB visualization •Useful for large-scale experiments

7

Method (1) • Between 19 March and 19 April 2012 an invitation to a survey has been sent to 120 e-mail addresses of different researchers, psychologists, psychiatrists, neuroscientists and MDs that have been published papers in which they have used at least one emotionally annotated database. The targeted databases were predominantly International Affective Picture System (IAPS), NimStim Face Stimulus Set, although some invitees have been using Karolinska Directed Emotional Faces (KDEF), Pictures of Facial Affect (POFA), Japanese Female Facial Expression (JAFFE) Database and International Affective Digital Sounds (IADS). 8

Method (2) • Out of 120 different invitees 29 have completed the survey (24.16%). Three (3) have filled out the form in Word format and other 26 have completed the survey online. – Some researchers represented groups or laboratories. Therefore, in actuality a larger number of participants have taken part in the survey but their answers were aggregated and averaged through one participant.

• The answers were sent in period 26 March – 19 July 2012. • All participants finished the survey in under 5 minutes, with 2min 40sec on average. • The anonymity of the participants was guaranteed. Only their IP addresses, the time of access and responses were recorded. Duplicate answers were not allowed.

9

Method (3) • The survey was short and consisted of 10 question with predefined possible answers (i.e. values). • Possible answers were in a range 1–5 or 1–7. • All questions also had “Not sure / Not applicable” (“N / A”) option as one of the possible answers. • Any question could be left unanswered (blank). Such responses were later processed as “N / A”.

10

Method: Questions (1) •

The questions: 1.

2. 3.

4.

5.

How difficult or easy is the image retrieval process from an emotionallyannotated database (e.g. International Affective Picture System (IAPS), NimStim Face Stimulus Set, International Affective Digital Sounds (IADS), Karolinska Directed Emotional Faces (KDEF), Pictures of Facial Affect (POFA), The Yale Face Database, Cohn-Kanade AU Coded Facial Expression Database, Japanese Female Facial Expression (JAFFE) Database, Geneva Affective PicturE Database (GAPED), etc.)? How satisfied are you with the above mentioned level of difficulty in the image retrieval process from the database? How much time was necessary to effectively search the database and construct one picture sequence that was used in your research? Have you at any time felt that the picture set you were using is missing images with particular emotion or semantic content that would be useful for your stimuli sequence? On a scale of 1 to 7, with one being extremely inadequate and 7 being extremely adequate, please rate how inadequate, ambiguous or insufficient did you find the predefined descriptions of the images you used? 11

Method: Questions (2) •

The questions (cont.): 6.

On a scale 1 to 7, with one being extremely useless and 7 being extremely useful, please rate how useful, helpful or beneficial would a user-friendly software tool for intelligent retrieval of emotionally-annotated images be to your research? 7. Have you at any time during your research wanted to find the most appropriate emotionally-annotate image faster and more efficiently? 8. Did you construct the sequence manually or with a help of a software tool? 9. Skip this question if you did not use a software tool for retrieval of emotionallyannotated images. Did your group actually develop the tool used in your experiment or acquired it elsewhere? 10. How useful or useless a stimuli database with real-life video-clips, instead of just still images, would be to your work?

12

Results: Question 1 •

How difficult or easy is the image retrieval process from an emotionally-annotated database (e.g. International Affective Picture System (IAPS), NimStim Face Stimulus Set, International Affective Digital Sounds (IADS), Karolinska Directed Emotional Faces (KDEF), Pictures of Facial Affect (POFA), The Yale Face Database, CohnKanade AU Coded Facial Expression Database, Japanese Female Facial Expression (JAFFE) Database, Geneva Affective PicturE Database (GAPED), etc.)?

Count

%

Very difficult

0

0%

Difficult

5

17.24%

10

34.48%

Easy

6

20.69%

Very easy

3

10.34%

N/A

5

17.24%

Answer

Neither difficult nor easy

Very difficult; 0 N / A; 5

Difficult; 5

Very easy; 3

Easy; 6

Neither difficult nor easy; 10

13

Results: Question 2 •

How satisfied are you with the above mentioned level of difficulty in the image retrieval process from the database?

Count

%

Very dissatisfied

1

3.45%

Dissatisfied

4

13.79%

Answer

Neither satisfied nor dissatisfied

10

34.48%

Satisfied

8

27.59%

Very satisfied

2

6.9%

N/A

4

13.79%

Very dissatisfied; 1

Q2 Very satisfied; 2

Satisfied; 8

N / A; 4

Dissatisfied; 4

Neither satisfied nor dissatisfied; 10

14

Results: Question 3 •

How much time was necessary to effectively search the database and construct one picture sequence that was used in your research? Count

%

Less than 5 mins

0

0%

5-15 mins

1

3.45%

Answer

15-30 mins

3

10.34%

30-60 mins

3

10.34%

1-2 hrs

8

27.59%

Q3

Less than 5 mins; 0

5-15 mins; 1

N / A; 2 More than 12 hrs; 6

15-30 mins; 3 30-60 mins; 3

6-12 hrs; 3

2-6 hrs

3

10.34%

6-12 hrs

3

10.34%

More than 12 hrs

6

20.69%

N/A

2

6.9%

1-2 hrs; 8 2-6 hrs; 3

15

Results: Question 4 •

Have you at any time felt that the picture set you were using is missing images with particular emotion or semantic content that would be useful for your stimuli sequence?

Count

%

Yes

21

75%

No

5

17.86%

N/A

2

7.14%

Answer

Q4 N / A; 2 No; 5

Yes; 21

16

Results: Question 5 •

On a scale of 1 to 7, with one being extremely inadequate and 7 being extremely adequate, please rate how inadequate, ambiguous or insufficient did you find the predefined descriptions of the images you used?

Count

%

Value 1

0

0%

Value 2

2

6.9%

Answer

Q5 Value 3

9

Value 7; 0 Value 6; 3

31.03%

Value 1; 0

Value 8; 2

Value 2; 2 Value 3; 9

Value 4

6

20.69%

Value 5

7

24.14%

Value 6

3

10.34%

Value 7

0

0%

Value 8

2

6.9%

Value 5; 7

Value 4; 6

17

Results: Question 6 •

On a scale 1 to 7, with one being extremely useless and 7 being extremely useful, please rate how useful, helpful or beneficial would a user-friendly software tool for intelligent retrieval of emotionally-annotated images be to your research?

Count

%

Value 1

0

0%

Value 2

0

0%

Answer

Q5 Value 3

0

Value 7; 0 Value 6; 3

0%

Value 1; 0

Value 8; 2

Value 2; 2 Value 3; 9

Value 4

3

10.34%

Value 5

7

24.14%

Value 6

4

13.79%

Value 7

14

48.28%

Value 8

1

3.45%

Value 5; 7

Value 4; 6

18

Results: Question 7 •

Have you at any time during your research wanted to find the most appropriate emotionally-annotate image faster and more efficiently?

Count

%

Yes

25

86.21%

No

3

10.34%

N/A

1

3.45%

Answer

Q7

N / A; 1 No; 3

Yes; 25

19

Results: Question 8 •

Did you construct the sequence manually or with a help of a software tool?

Answer Manually

Count

%

13

44.83%

Q8 N / A; 2 Both; 7

With a software tool

7

24.14%

Both

7

24.14%

N/A

2

6.9%

Manually; 13

With a software tool; 7

20

Results: Question 9 •

Skip this question if you did not use a software tool for retrieval of emotionallyannotated images. Did your group actually develop the tool used in your experiment or acquired it elsewhere?

Count

%

The tool was acquired

6

20.69%

The tool is own development

6

20.69%

17

58.62%

Answer

N/A

Q9 The tool was acquired; 6

N/A; 17

The tool is own development; 6

21

Results: Question 10 •

How useful or useless a stimuli database with real-life video-clips, instead of just still images, would be to your work?

Count

%

Completely useless

0

0%

A little bit useful

1

3.45%

Neither useful nor useless

0

0%

Useful

6

20.69%

21

72.41%

1

3.45%

Answer

Very useful N/A

Q10

Completely useless; 0

A little bit useful; 1

Neither useful nor useless; 0

N / A; 1 Useful; 6

Very useful; 21

22

Discussion (1) • The survey has clearly indicated that intelligent, efficient and user-friendly generators of emotionally and semantically annotated multimedia stimuli are clearly desired by the professionally community. • Almost all researchers (25 of 29; 86,21%) consider such software tools to be useful, helpful and beneficial in the intelligent retrieval of emotionally-annotated images.

23

Discussion (2) • Furthermore, the results show that stimuli sequences are predominantly constructed by hand – scouring through IAPS and other databases – over a period of 1-2 hours. Several researchers use general purpose tools like Matlab and SPSS, or to a lesser extent, custom-made software applications. • Most researchers are ambiguous towards practical difficulties in construction of stimuli sequences, yet a great majority of them (86.21%) would like to accomplish this task even faster and more efficiently.

24

Discussion (3) • A clear majority of the participants (75%) think that current emotionally annotated databases lack at least some stimuli with a particular semantic or emotional content that could be useful in their work. • Most participants (58.62%) deem stimuli descriptors to be inadequate, ambiguous or insufficient in conveying the true semantic and emotional content of stimuli in emotionally annotated databases. • Finally, 72.41% of the participants would like to use reallife video sequences in their elicitation procedures. 25

Conclusion • Although not large by the complexity of questions or by the number of participants, this survey may be considered a success. Answers follow Gaussian distribution and have statistically significant indicators of the targeted population’s opinions. • This survey was initially conceived as a test for validity of the intelligent stimuli sequence generator concept, but apart from succeeding in its verification the survey provided an insight in the needs and preferences of the professional community which may be used in potential future surveys and the associated research. 26

Reference • If you want to cite the results of this survey please use the following reference: • Horvat, M., Popović, S., Ćosić, K.: Multimedia stimuli databases usage patterns: a survey report. In: Proc. of the 36nd International Convention MIPRO 2013: Computers in Technical Systems, Intelligent Systems, Opatija, pp. 993−997 (2013)

12:39:33

27

Contact Dr Marko Horvat Lecturer

Department of Computer Science and Information Technology University of Applied Sciences Vrbik 8, HR-10000 Zagreb Croatia Phone: +385 (0)1 56 03 944 Fax: +385 (0)1 56 03 999 E-mail: [email protected] Web (work): http://www.tvz.hr Web (pers.): http://www.marko-horvat.name

28