Measurement Educational and Psychological - Iowa ASCD

4 downloads 0 Views 253KB Size Report
Jun 29, 2011 - Roberts, 2004; Wigelsworth, Humphrey, Kalambouka, & Lendrum, 2010). There seems ...... Measure–Revised [PTM-R], Assessment of Children's Emotion Skills [ACES], DANVA, .... Fairfax, VA: George Mason University.
Educational and Psychological Measurement http://epm.sagepub.com/

Measures of Social and Emotional Skills for Children and Young People : A Systematic Review Neil Humphrey, Afroditi Kalambouka, Michael Wigelsworth, Ann Lendrum, Jessica Deighton and Miranda Wolpert Educational and Psychological Measurement 2011 71: 617 DOI: 10.1177/0013164410382896 The online version of this article can be found at: http://epm.sagepub.com/content/71/4/617

Published by: http://www.sagepublications.com

Additional services and information for Educational and Psychological Measurement can be found at: Email Alerts: http://epm.sagepub.com/cgi/alerts Subscriptions: http://epm.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations: http://epm.sagepub.com/content/71/4/617.refs.html

>> Version of Record - Jun 29, 2011 What is This?

Downloaded from epm.sagepub.com by guest on January 1, 2012

Measures of Social and Emotional Skills for Children and Young People: A Systematic Review

Educational and Psychological Measurement 71(4) 617­–637 © The Author(s) 2011 Reprints and permission: sagepub.com/journalsPermissions.nav DOI: 10.1177/0013164410382896 http://epm.sagepub.com

Neil Humphrey1, Afroditi Kalambouka1, Michael Wigelsworth1, Ann Lendrum1, Jessica Deighton2, and Miranda Wolpert2

Abstract This study presents the findings of a systematic review of measures of social and emotional skills for children and young people. The growing attention to this area in recent years has resulted in the development of a large number of measures to aid in the assessment of children and young people. These measures vary on a number of variables relating to implementation characteristics and psychometric properties.The methodology of the review followed the general principles of systematic reviewing, such as systematic search of databases, the adoption of predetermined set of inclusion and exclusion criteria, and a multistage filtering process. The review process resulted in the retention of 12 measures, which are presented and discussed in relation to key issues in this area, including difficulties with the underlying theory and frameworks for social and emotional skills, inconsistent terminology, the scope and distinctiveness of available measures, and more practical issues such as the type of respondent, location, and purpose of measurement. Keywords social and emotional skills, measurement, children

1

University of Manchester, Manchester, UK The Anna Freud Centre, London, UK

2

Corresponding Author: Neil Humphrey, School of Education, University of Manchester, Oxford Road, Manchester, M13 9PL, UK Email: [email protected]

Downloaded from epm.sagepub.com by guest on January 1, 2012

618

Educational and Psychological Measurement 71(4)

Introduction This article reports on the findings of a systematic review of measures of social and emotional skills for children and young people. Its intended contributions are (a) to provide an overview of the state of the field in this important area of psychological measurement, (b) to highlight key issues pertaining to the definition and measurement of social and emotional skills in children and young people, and (c) to outline and discuss the implementation characteristics and psychometric properties of the 12 measures of social and emotional skills that passed our systematic review screening process and are firmly established in the academic literature in this area. The article is the first (to our best knowledge) truly systematic and rigorous review of measures of social and emotional skills for children and young people, and as such it may provide a unique contribution to knowledge in this field. It is intended to be of use to researchers and practitioners with an interest in children’s social and emotional skills around the world. To this end, we have included information pertaining to the availability of norms and translated versions of the measures identified during the review process. This is of particular importance given the increasing interest in children’s social and emotional skills in countries other than the United States and the United Kingdom (Fundacion Marcelino Botin, 2008).

Defining Social and Emotional Skills In recent years, educational policies in the United Kingdom, the United States, and elsewhere have reflected a growing interest and investment in the development of social and emotional skills as a means to promote children’s well-being, adjustment, and academic achievement. The issue of the mixed and inconclusive evidence on the effectiveness of these approaches is a matter of ongoing debate that has generated a great deal of academic and professional interest and will continue to do so for the foreseeable future (Humphrey, Curran, Morris, Farrell, & Woods, 2007; Humphrey et al., 2008; Zeidner, Roberts, & Matthews, 2002). However, more fundamental issues—such as inconsistencies in the definition, measurement, and utility of the concept of social and emotional skills—also need to be acknowledged and explored (Matthews, Zeidner, & Roberts, 2004; Wigelsworth, Humphrey, Kalambouka, & Lendrum, 2010). There seems to be little common consensus as regards to what is meant by social and emotional skills, and how they are best measured. As Humphrey et al. (2007) have argued, clear definition is a basic scientific requirement—yet one is still lacking in this area. “Social and emotional skills” is referred to in this article as it is the currently preferred term in English government policy documentation (e.g., Department for Children, Schools and Families [DCSF], 2007a, 2007b). We use it to incorporate other terms, such as “social and emotional intelligence,” “emotional literacy,” and “social and emotional competence,” among others. Although some authors (e.g., Weare & Gray, 2003) have argued for the differentiation of these terms, reviews of the literature in this area (e.g., Humphrey et al., 2007; Wigelsworth et al., 2010) have led us to the

Downloaded from epm.sagepub.com by guest on January 1, 2012

619

Humphrey et al. Table 1. Denham’s (2005a) Framework for Social–Emotional Competence Emotional competence skills

Self-awareness Self-management Social awareness

Understanding self emotions Emotional and behavioral regulation Understanding emotions Empathy/sympathy

Relational/prosocial skills

Social problem solving Relationship skills Cooperation Listening skills Turn-taking Seeking help

conclusion that they do not describe qualitatively different concepts. In defining social and emotional skills, we refer readers to the framework of social–emotional competence proposed by Denham (2005a). Drawing on the work of Rose-Krasnor (1997) and Payton et al. (2000), Denham proposes a delineation of relational/prosocial skills and emotional competence skills as given in Table 1. This model benefits from reflecting the domains outlined in major theoretical models (e.g., Bar-On, 1997; Goleman, 1995; Salovey & Mayer, 1990) in this area, without suffering from the nebulous, imprecise nature of the more wide-ranging definitions (e.g., Petrides & Furnham, 2001) that have attracted criticism from skeptics (see Matthews, Zeidner, et al., 2004). A number of key issues in reviewing measures of social and emotional skills are suggested in the literature. In brief, these include (a) the distinction between capturing typical and maximal behavior (Wilhelm, 2005), (b) the extent to which measures of social and emotional skills provide information that is distinct or unique among existing constructs (e.g., personality), (c) the scope and specificity of measures (e.g., single, unidimensional vs. complex, multidimensional), and (d) who provides information (e.g., children, teacher, parent, peers; see Wigelsworth et al., 2010).

Rationale and Context for the Current Study The accurate assessment of social and emotional skills in children and young people has crucial implications for public health because of the associations with mental health, academic performance, and other key outcomes (Denham, Wyatt, Bassett, Echeverria, & Knox, 2009). In late 2008, the authors were commissioned by the DCSF to conduct a systematic review of measures of social and emotional skills for children and young people (Humphrey et al., 2009) as companion to a systematic review of measures of child mental health in children (Wolpert et al., 2009). The aim of the review was to identify a range of measures suitable for assessing social and emotional skills in children and young people for possible future inclusion in a population survey relating to progress toward the “2020 vision” in the Children’s Plan (DCSF, 2007b). This policy document

Downloaded from epm.sagepub.com by guest on January 1, 2012

620

Educational and Psychological Measurement 71(4)

sets out an ambitious set of goals for children in England to be achieved by the year 2020, including a focus on improving social and emotional skills. Beyond this, we felt that a review of this kind was extremely timely. Although there are a handful of discussion papers (e.g., Wigelsworth et al., 2010) and reviews (e.g., Denham, 2005a; StewartBrown & Edmunds, 2003) in the literature, they are all selective and none have applied a rigorous and systematic approach. Furthermore, given the emphasis on monitoring and evaluation in currently popular initiatives such as the Social and Emotional Aspects of Learning (SEAL) program, a review of this kind should also be useful to practitioners. Although the DCSF have provided a guidance document on this topic (see DCSF, 2007a), it contains no substantive references to issues such as reliability and validity, and only one of the recommended measures has undergone any kind of appropriate validation process. Finally, given the high level of academic interest and debate around the concept of emotional intelligence, a review of this kind is crucial: “Determining whether EI [emotional intelligence] is a measurable quality is central to its development as a scientific construct” (Matthews, Roberts, & Zeidner, 2004, p. 182).

Review Method The present review followed the general paradigm of a parallel systematic review of mental health measures for children and young people (Wolpert et al., 2009). The review followed a systematic process across six consecutive stages, summarized in the flow diagram given in Figure 1 and described below.

Stages 1: Identification of Measures Setting the Parameters of the Review and Search of Key Databases. Following the parallel child mental health review (Wolpert et al., 2009), “outcome measures” were defined as any assessment or evaluation tools (e.g., questionnaires, checklists, or scales) that aim to measure the social and emotional skills of children and young people. The terms of the “measures of social and emotional skills” review were divided into three categories (see below) and their related terms (see Table 2): a. terms relating to social and emotional skills, b. terms relating to measurement, and c. terms relating to the population under study (children and young people) Initial database searches focused on the main psychology, sociology, education, and medical databases. However, searches in the main medical databases produced limited relevant hits and a lot of irrelevant hits (e.g., those that were within the child mental health review remit), so it was decided that the medical databases would be dropped in favor of more extensive searches in the following educational, social and psychology databases: PsychInfo, Education Resources Information Centre (ERIC), British Education Index (BEI), Australian Education Index (AEI), and Applied Social Sciences Indexes

Downloaded from epm.sagepub.com by guest on January 1, 2012

621

Humphrey et al.

Identification of measures

Stage 1: Literature searches in 5 databases: Psychinfo, ERIC, BEI, AEI, and ASSIA

Stage 2: Secondary searches and collective knowledge tranche Identification of 189 measures Sifting of measures

Stage 3: Filtering of measures according to inclusion and exclusion criteria Filtering down to 52 measures

Collecting more Information on retained measures and further sifting

Stage 4: Secondary searches; requests to measure developers for more information Filtering down to 23 measures Stage 5: Filtering according to academic presence - number of articles using measure Filtering down to 12 measures

Detailed examination of final measures

Stage 6: Presentation and description – implementation characteristics, and psychometric properties

Figure 1. Stages in the systematic review process

and Abstracts (ASSIA). All related terms for each of the three categories of interest (see Table 1) were then entered in the thesaurus facility within each database in order to ensure that the key words under which articles were catalogued would be identified through using the appropriate terms within each specific database. The thesaurus searches produced additional relevant key words and/or headings, which were then incorporated into subsequent searches. For example, the terms “emotional adjustment” and “emotional development” were identified within the ERIC thesaurus facility search and were later used to capture relevant articles. Any additional terms produced through the thesaurus/ heading search that were too broad and produced too many irrelevant hits were later

Downloaded from epm.sagepub.com by guest on January 1, 2012

622

Educational and Psychological Measurement 71(4)

Table 2. Categories of Interests and Related Terms for the Systematic Review Category

Related Terms

1. Social and emotional skills

2. Measurement 3. Population of interest

Social/emotional Social, emotional, socioemotional, socio-emotional, effect, affective, interpersonal Skills Skills, intelligence, competence, literacy, learning, awareness, ability, regulation Measure, measurement, questionnaire, survey, checklist, check list, tool, rating scale, scale, inventory, assessment, evaluation, screening, instrument, test Children, adolescents, pupils, students

discarded. This stage generated a list of final search terms, which were then used to run searches across each of the databases (see Table 3). A total of 2,870 hits were transferred and saved into bibliographic applications (EndNote 9.0.1 for the ASSIA and PsychInfo databases; Reference Manager v11 for the ERIC, BEI, and AEI databases). Basic filtering was then applied through reading the title (and/or abstract in cases where the title was not sufficient) to discard the obviously irrelevant hits (i.e., papers not related to the measurement of children’s social and emotional skills). Basic filtering resulted in a total of 542 articles. The details of these articles (title, abstract, and publication details) were transferred into Excel spreadsheets, where further filtering took place involving the application of the following exclusion criteria: (a) no social and/or emotional skills measure mentioned in the abstract, (b) the measure mentioned was too narrow to provide a broad assessment of social and emotional skills (e.g., focused exclusively on just one specific aspect of social and emotional skills such as empathy), (c) the article referred to a measure that was not used with child or adolescent populations, (d) the article was not in English, and/or (e) the article was a duplicate of a previous hit within the database. All remaining hits after this filtering procedure were organized according to the measures they referred to, resulting to a total of 137 measures.

Stage 2: Secondary Searches and Collective Knowledge Tranche The list of 137 measures was complemented by a number of measures from a “collective knowledge tranche.” These were measures that were known to members of the research team through previous research. Additional measures were identified through reviewing papers and compendiums (e.g., Denham, 2005a) and/or bibliographic references. The authors also contacted a number of relevant groups and organizations (e.g., Collaborative for Academic, Social and Emotional Learning; Antidote) via email, but

Downloaded from epm.sagepub.com by guest on January 1, 2012

623

Humphrey et al. Table 3. Final Search Terms for the Systematic Review Database ASSIA

ERIC, BEI, and AEI

PsychInfo

Search Terms for Subject Heading

Initial Hits

Hits After Basic Filtering

DE = (emotion* or social) and DE = (skills or intelligence or competence or learning or awareness or regulation or behavior or interaction) and DE = (measure* or questionnaire* or assessment or test* or survey* or scale* or evaluation or instrument*) and DE = (child* or (young people) or adolescent* or pupil* or student*) (emotional skills OR socioemotional skills OR social intelligence OR social competence OR emotional literacy OR emotional development OR social development OR interpersonal competence OR social cognition OR affective behavior OR social behavior OR social skills OR emotional intelligence OR emotional competence OR socioemotional competence OR social literacy OR socioemotional literacy OR social learning OR emotional learning OR social awareness OR emotional awareness OR emotional adjustment OR emotionalresponse OR social ability OR emotional ability OR socioemotional ability) DE AND (Measurement OR Questionnaire OR Rating-Scale OR surveys OR checklist OR psychometric OR assessment OR screening) DE AND (children OR adolescent OR pupil OR student) DE (emotion* or social* or socio*) AND (measure* or question* or survey or check* or tool or rating or scale* or inventor* or instrument* or test* or screening or repository or assessment* or evaluation*) AND (child* or adolscen* or youth* or young* or pupil* or student*)

172

80

1820

390

788

72

Note. ASSIA = Applied Social Sciences Indexes and Abstracts; ERIC = Education Resources Information Centre; BEI = British Education Index; AEI = Australian Education Index. “Hits” refers to the number of articles identified in each database.

this did not result in the procurement of any measures not already identified elsewhere. A total of 52 measures identified in this stage were added to the existing list for further sorting, increasing the total number of measures to 189.

Downloaded from epm.sagepub.com by guest on January 1, 2012

624

Educational and Psychological Measurement 71(4)

Stage 3: Sifting of Measures Filtering of Measures According to Inclusion and Exclusion Criteria. This stage involved reading of the relevant abstract and/or article and applying a list of inclusion/ exclusion criteria. A measure was included if it met all of the inclusion criteria and excluded if it met one or more of the exclusion criteria: Inclusion criteria. To include a measure of social and emotional skills for children and adolescents, the measure should 1. seek to provide measurement of generic social and emotional skills in children and young people (up to 18); 2. be either multidimensional or unidimensional; 3. refer to a measure that can be completed by child or parents/carers with the possible addition of professionals such as teachers (but not by teachers only); 4. have been validated in a child or adolescent context, even if not originally developed for this purpose; 5. be available in English language; and 6. be able to be used with a reasonably wide age range (e.g., not just for preschoolers). Exclusion criteria. A measure would be excluded if it: 1. is not available in English; 2. does not measure social and/or emotional skills; 3. does not cover a broad range of social and/or emotional skills (e.g., focused exclusively on just one specific aspect of social and emotional skills such as empathy—see Table 1); 4. is not used with children and/or adolescents; 5. is based on professional report only (e.g., teacher, educational psychologist, or other practitioner); 6. takes more than 30 minutes to complete; 7. provides open-ended responses that have to be manually coded; 8. the age range is too narrow (e.g., preschool version of the measure only); 9. has not been used with a variety of populations (e.g., only used with specialist groups such as children with autism). Following application of the above criteria, the list of 187 measures was reduced to 52 measures (for a list of the 52 measures see Humphrey et al., 2009). Measures for which information from the abstract/article did not provide adequate evidence to apply the above inclusion/exclusion criteria at this stage were still retained until further evidence was available to enable a decision on retention.

Downloaded from epm.sagepub.com by guest on January 1, 2012

625

Humphrey et al.

Stage 4: Collecting More Information on Retained Measures Secondary Searches and Requests to Measure Developers for More Information. The above list of the 52 potentially suitable measures was organized in an Excel database, where information was entered on the following variables: a brief description of the measure, different versions available (e.g., child, parent, teacher), number of items and completion time, scales and subscales are included in the measure, response format (e.g., 4-point Likert-type scale), information on reliability (internal consistency, test–retest) and validity (e.g., construct validity), sample items, cost, and key references. To provide as complete information as possible on these variables, several procedures were followed: (a) an attempt was made to obtain as many articles as possible on each measure and (b) measure developers were contacted through email/telephone to fill in gaps in our database and to request manuals and articles. As we found more detailed information on each measure it became obvious that some did not meet the inclusion/ exclusion criteria set out in the previous stage (for instance, if we found that a given measure could only be completed by teachers). By the end of this stage a total of 23 measures were retained.

Stage 5: Filtering According to Academic Presence: Number of Articles Using Measure This stage involved a further filtering process in which only measures that had been used in four or more articles in peer-reviewed academic journals were retained. This was done in an effort to retain only measures with an established and sustained presence in the academic literature, whilst also ensuring a workable number of measures for detailed examination in the final stage. Four or more articles was considered to be a useful benchmark since it would typically signify that the measure has gone beyond basic development and validation and is being used by the academic community for program evaluation, theory development, and so on. This process resulted in 12 measures being retained.

Stage 6: Detailed Examination of Final Measures Presentation and Description: Implementation Characteristics and Psychometric Properties of Final 12 Measures. The final 12 measures were reviewed in depth in relation to their implementation characteristics and psychometric properties. These are presented in Tables 4 and 5, respectively.

Discussion The primary purpose of our systematic review was to was to identify a range of measures suitable for assessing social and emotional skills in children and young people for possible future inclusion in a population survey relating to progress toward the “2020

Downloaded from epm.sagepub.com by guest on January 1, 2012

626

Downloaded from epm.sagepub.com by guest on January 1, 2012

PKBS-2

PTM-R

Preschool and Kindergarten Behaviour Scales-2 (Merrell, 1996)

Prosocial Tendencies Measure–Revised (Carlo, Hausmann, Christiansen, & Randall, 2003)

SCBE

CABS, CABS-SR

EQI:YV, EQI:YV(S)

Bar-On Emotional Quotient Inventory:Youth Version (Bar-On & Parker, 2008)

Child Assertive Behaviour Scale (Michelson & Wood, 1982) Social Competence and Behavior Evaluation Scale (LaFreniere & Dumas, 1996)

Acronym

Measure

Typical

Typical

Maximal (child), typical (other) Typical

Typical

Maximal or Typical

Social– emotional

Social

Social

Social

Social– emotional

Scope

11-18

3-6

2.5-6.5

8-12

7-18

Age (Years)

Child

Teacher, parent

Teacher, parent

Child, parent, teacher

Child

Versions

Table 4. Implementation Characteristics of the Final 12 Measures

25

76

80, 30(S)

27

60, 30(S)

Length Scales and Subscales

30

12

15

Multiple choice 1-5

Likert 1-4

Response Scales

Depressive–joyful, anxious– Likert 1-6 secure, angry–tolerant, isolated–integrated, aggressive–calm, egotistical–prosocial, oppositional–cooperative, dependent–autonomous and 4 summary scales: social competence, externalizing problems, internalizing problems, and general adaptation Social cooperation, Likert 1-4 social interaction, social independence, externalizing problems and internalizing problems Public, anonymous, dire, Likert 1-5 emotional, compliant and altruistic prosocial behavior

20-25, Total emotional intelligence, 10-15 (S) interpersonal, intrapersonal, adaptability, stress management, general mood, positive impression, inconsistency index 10-15 Total assertiveness, passivity, and aggressiveness

Completion Time (Minutes)

(continued)

Spanish

FrenchCanadian, Spanish, Japanese, Russian, Chinese, Portuguese, Italian, AustrianGerman

Spanish, Dutch

Pedi (South Africa)

Other Languages



627

Downloaded from epm.sagepub.com by guest on January 1, 2012

CRS

SSIS, SSRS

ACES

Social Skills Improvement System (Gresham & Elliot, 1990)

Assessment of Children’s Emotion Skills (Schultz, Izard, & Bear, 2004)

Acronym

Child Rating Scale (Hightower, Cowen, Spinell, & Lotyczewski, 1987)

Measure

Table 4. (continued)

Maximal

Typical

Typical

Maximal or Typical

Emotional

Social– emotional

Social

Scope

4-8

3-18

5-13

Age (Years)

Child

Child, parent, teacher

Child, parent, teacher

Versions

56

79 (parent), 75 (child), 83 (teacher)

24, 38 (teacher), 18 (parent)

Length

10-25

10-25

10-20

Completion Time (Minutes) Rule compliance/acting out, anxiety/withdrawal, interpersonal social skills, self-confidence (child); acting out, shy-anxious, learning, frustration tolerance, assertive social skills, task orientation, peer sociability (teacher); Acting out, frustration tolerance, shy-anxious and peer sociability (parent) Social skills (communication, cooperation, assertion, responsibility, empathy, engagement, self-control), competing problem behaviors (externalizing, bullying, hyperactivity/ inattention, internalizing, autism spectrum), and academic competence (reading achievement, math achievement, motivation to learn) Emotional attribution accuracy, anger attribution tendencies, happiness attribution tendencies, sadness attribution tendencies

Scales and Subscales

Multiple choice 1-5

Likert 1-4

Likert 1-3

Response Scales

(continued)

Spanish, Iranian

Other Languages

628

Downloaded from epm.sagepub.com by guest on January 1, 2012

Diagnostic Analysis of Nonverbal Accuracy (Nowicki & Duke, 1989) Differential Emotions Scale (Izard, Dougherty, Bloxom, & Kotsch, 1974)

Emotion Regulation Checklist (Shields & Cicchetti, 1997) Matson Evaluation of Social Skills with Youngsters (Matson, Rotari, & Helsel, 1983)

Measure

Maximal

Typical

DES-IV

Typical

Typical

Maximal or Typical

DANVA

MESSY

ERC

Acronym

Table 4. (continued)

Emotional

Emotional

Social– emotional

Emotional

Scope

7-adult

4-adult

4-18

6-12

Age (Years) 24

Length

Child

Child

36

16-32

Child, 64 (teacher), teacher 62 (child)

Parent, teacher

Versions

10-15

30

10-25

10

Completion Time (Minutes)

Positive emotionality (interest, joy, surprise), negative emotionality (shyness, sadness, anger, disgust, contempt, selfhostility, fear, guilt and shame)

Appropriate social skills, inappropriate assertiveness, impulsiveness, overconfident behavior, jealousy Interpretation of child and adult faces, paralanguage and posture

Negativity, emotional regulation

Scales and Subscales

Likert 1-5

Multiple choice 1-4

Likert 1-5

Likert 1-4

Response Scales

Japanese, Spanish, Turkish, Chinese, Dutch, Portuguese

Other Languages



629

Downloaded from epm.sagepub.com by guest on January 1, 2012

U.S. Norms

Yes

No

Yes

Yes

Measure

EQI:YV

CABS

SCBE

PKBS-2

No

No

No

Yes

U.K. Norms

.84-.97

.80-.89

.79 (total scale)

.67-.90

Internal Standard error of measurement and prediction available by age and gender

Other

.74-.87 (2 weeks), .59-.70 (6 months) .58-.86 (3 Standard error of weeks), measurement .69-.78 (3 available; months) interrater reliability: .36-.63

0.66-0.78 (4 weeks)

.77-.89

Test–Retest

Reliability

Table 5. Psychometric Properties of the Final 12 Measures

Yes

Yes (for both single and dual structure) Yes

Yes

Factorial Distinguishes between gifted and nongifted students

Discriminative

Predicts problem gambling in adolescence and academic achievement in high school

Predictive

Distinguishes between children with high and low sociometric status Correlates moderately Differentiated a Predicts social with internalizing smaller clinical interaction/ and externalizing sample from the rejection and peer problems overall sample victimization Correlated with other Discriminates established measures between typically of social skills, while developing problem behavior children, scores correlated children at risk with externalizing of developing and internalizing social–behavioral problems problems, children with internalizing– externalizing problems, and children with ADHD symptoms

Correlates with adult emotional intelligence, personality, internalizing and externalizing problems Correlates with social problem solving

Convergent

Validity

(continued)

Other

630

Downloaded from epm.sagepub.com by guest on January 1, 2012

Yes

No

No

Yes

SSIS

ACES

ERC

MESSY

PTM-R

?

No

Measure

CRS

U.S. Norms

No

No

No

No

No

No

U.K. Norms

Table 5. (continued)

.54-.82 (2 weeks)

Test–Retest

Split-half a: .78 (child), .87 (teacher)

.84-.92

.46-.70

.72-.97 (parent), .73-.97 (teacher), .73-.95 (child)

Other

.60-.90

.72-.87 Interrater (parent), reliability: .48.68-.92 .69 (teacher), (teacher), .38-.69 (parent); .59-.81 standard error (child) of measurement by age and gender available

.49-.80 .49-.79 (child), (child), .61-.91 .89-.95 (teacher), (teacher) .91 (parent)

.59-.86

Internal

Reliability Convergent

Discriminative

Yes

Yes Correlates with similar measures in normal and learning disabled samples

Predictive

Predicts academic competence and attentional competence Distinguishes between bullies and victims

Subscales correlate with several theoretically related variables (from cognitive variables to emotive and behavior measures) Yes Correlates with Discriminates classroom between stressadjustment, health resilient and stressresources, and selfaffected children esteem Yes (including Correlates with other Discriminates with ADHD similar measures between controls subsample) and children with ADHD

Yes

Factorial

Validity

(continued)

Development of tool included item response theory

Other



631

Downloaded from epm.sagepub.com by guest on January 1, 2012

No

No

Measure

DANVA

DES-IV

No

No

U.K. Norms

.50-.85

.69-.81

Internal

.30-.75

.66

Test–Retest

Reliability Other

Factorial Correlates with previous version of measure Subscales correlate with depression

Convergent

Discriminates between abused and nonabused female samples, and depressed and nondepressed subjects

Discriminative

Validity Predictive

Other

Note. EQI:YV = Bar-On Emotional Quotient Inventory:Youth Version; CABS = Child Assertive Behavior Scale; SCBE = Social Competence and Behaviour Evaluation Scale; PKBS-2; Preschool and Kindergarten Behaviour Scales–2; PTM-R = Prosocial Tendencies Measure–Revised; CRS = Child Rating Scale; SSIS = Social Skills Improvement System; ACES = Assessment of Children’s Emotion Skills; ERC = Emotion Regulation Checklist; MESSY = Matson Evaluation of Social Skills with Youngsters; DANVA = Diagnostic Analysis of Nonverbal Accuracy; DES-IV = Differential Emotions Scale–IV; ADHD = attention deficit hyperactivity disorder.

U.S. Norms

Table 5. (continued)

632

Educational and Psychological Measurement 71(4)

vision” in the Children’s Plan (DCSF, 2007b). At a secondary level, we sought to present an overview of the current state of this field. The review process resulted in the retention of 12 measures that vary greatly in both their implementation characteristics and psychometric properties. In this final section of the current article we present some key issues that have arisen during the course of our review, providing illustrative examples from the measures identified. In relation to the establishment and sustained presence of measures in the academic literature, very few measures appear to have a good “shelf life.” Although 12 measures have yielded 4 or more publications in peer-reviewed journals, only a very small handful (notably the Diagnostic Analysis of Nonverbal Accuracy [DANVA], Social Competence and Behaviour Evaluation Scale [SCBE], and Social Skills Improvement System [SSIS]) have been used with great frequency (e.g., more than 10 articles). Considering that 189 measures were originally identified, it is clear that many are short-lived. This is not uncommon though—Butler and Gasson (2005) identified a similar issue in their review of measures of self-concept and self-esteem. One must also consider that the upsurge in interest in social and emotional learning (and therefore measurement of social and emotional skills) among children and adolescents is relatively recent (although interest in social skills can be traced back to Thorndike, a standard watershed in the literature is Salovey and Mayer’s (1990) article on emotional intelligence). Furthermore, the rather protean nature of the construct in question provides a fundamental problem for measure developers. Indeed, it has led some critics to state that it is “bereft of any conceptual meaning” (Zeidner et al., 2002, p. 215). A further observation we have made is the imbalance among the scope and type of the measures identified. There appear to be many more well-established measures that tap social skills, as opposed to emotional skills or both. Furthermore, there are relatively few maximal behavior measures. This has important implications in terms of recommending a truly “direct” measure, since there is the danger that “typical behavior” social and emotional skills measures may simply tap aspects of personality (Gannon & Ranzijn, 2005; Schulte, Ree, & Carretta, 2004; Van Rooy, Viswesvaran, & Pluta, 2005). However, this has to be balanced against the practical advantages (e.g., time) of typical behavior measures, and also the emergent pockets of evidence regarding their validity (e.g., Petrides et al., 2006, found that scores on the Trait Emotional Intelligence Questionnaire were associated with peer nominations of behavioral descriptions, such as “cooperative” and “disruptive”). There are also concerns relating to the practicalities of capturing certain domains—such as self-awareness—in a maximal behavior paradigm. The majority of measures reviewed here have been developed and standardized with American populations. For some of these measures, evidence of research carried out with other populations has concluded that they can be translated and adopted in different cultures while still retaining acceptable reliability and validity. However, of the 12 final measures produced by this review, only one (the Bar-On Emotional Quotient Inventory: Youth Version [EQI:YV]) has U.K. norms; several have no norms at all, and none have norms for countries other than the United States or the United Kingdom.

Downloaded from epm.sagepub.com by guest on January 1, 2012

633

Humphrey et al.

This is a particular concern given that cultural transferability between the United States, the United Kingdom, and other countries cannot be automatically assumed. In terms of psychometrics, our retained measures largely demonstrated acceptable properties using standard techniques (e.g., internal consistency, test–retest reliability, factorial validity, construct validity, discriminative validity), but evidence pertaining to more advanced analysis (such as those based on item response theory—as recommended by Terwee et al., 2007) was lacking for most measures (notable exceptions include the Matson Evaluation of Social Skills with Youngsters [MESSY] and SSIS). This may again reflect the emergent state of the field. Beyond the aforementioned issues, there appears to have been little analysis of the applicability of the measures to different groups of children—such as different ethnic groups—although the SSIS is an exception in this regard (Van Horn, Atkins-Burnett, Karlin, & Snyder, 2007). This kind of work is of course essential to ensuring that a given measure is appropriate for use with diverse groups of children and young people (e.g., in the aforementioned population survey), but just as importantly it can lead to development and improvement of the instrument itself. For instance, in Van Horn et al.’s (2007) recent psychometric analysis of the SSIS, an alternative factor structure with a stronger data fit to that proposed in the manual for the measure emerged. The range of possible respondents in our final 12 measures also varied greatly. Only three (Child Assertive Behavior Scale [CABS], Child Rating Scale [CRS], and SSIS) have child, parent, and teacher versions, and five (EQI:YV, Prosocial Tendencies Measure–Revised [PTM-R], Assessment of Children’s Emotion Skills [ACES], DANVA, and Differential Emotions Scale [DES]) can only be completed by the child/young person. Consideration of the range of respondents for a given measure is important because each has access to unique information. Through introspection, the child has access to the most detailed information about himself or herself of any of the possible respondents. Furthermore, recent policy and legislation has placed increasing emphasis on the importance of the child’s perspective (e.g., Every Child Matters—Department for Education and Skills, 2003). However, as self-awareness/perception follows a developmental trajectory, older children and adolescents are likely to be more accurate responders than younger children (Denham, 2005b). The limited number of measures that incorporate child, teacher, and parent responses creates difficulties for researchers and/or practitioners attempting to create a triangulated profile of the social and emotional skills of a child or group of children. The advantages to triangulating measures of social and emotional skills are clear, especially in response to overcoming difficulties in self-report (see above). Of course, one must factor in the increased time and expense in collecting data from numerous perspectives, especially given typically low response rates of certain groups (such as parents). However, given the relatively low concurrence between teacher, parent, and child responses in this area (e.g., Humphrey et al., 2008, found correlations as low as .25, indicating only 6% shared variance), consideration needs to be given as to whose perspective is prioritized.

Downloaded from epm.sagepub.com by guest on January 1, 2012

634

Educational and Psychological Measurement 71(4)

Conclusion In concluding, it is also important to note the limitations imposed on our review process. Although we have conducted an entirely systematic review, the protean nature of the area under investigation caused some difficulties, and other authors’ interpretations of “social and emotional skills” may differ from our own (meaning that other authors may have used different definitions—leading to different results!). The fact that the precise keywords used and/or the precise question under investigation may determine the result of a systematic review has been acknowledged as one of the integral characteristics of systematic reviews (Hannes, Claes, & The Belgian Campbell Group, 2007; Nind, 2006). Furthermore, the systematic review process itself is not without criticism. For example, publication bias (the tendency for a greater proportion of studies with statistically significant positive results to be published—Torgerson, 2006) might have limited the published articles for a given measure and therefore the measure’s inclusion in the later stages of our review. While we acknowledge this, we felt that this criterion was an important way to assess the “academic presence” of the measures. Furthermore, all criteria were negotiated with users from the project steering group. Such close consultation with users not only contributes to the significance of the review and its fitness for purpose but also helps in addressing bias relating to the interpretation of findings (Harlen & Crick, 2004). The systematic review reported in this article identified 12 measures with varying implementation characteristics and psychometric properties. We feel that this variability— particularly in relation to psychometric properties—reflects the emergent state of the field. Although there has been an interest in social and emotional learning over the last two decades, measure development has struggled to keep pace. This is undoubtedly reflected in the fact that many reports of social and emotional learning program evaluations have not contained any social and emotional skills outcome measures, instead using proxy indicators of success like reductions in mental health problems or increases in attendance (Humphrey et al., 2007). Declaration of Conflicting Interests The authors declared no potential conflicts of interests with respect to the authorship and/or publication of this article.

Funding The research reported in this article was funded by a grant from the Department for Children, Schools and Families (REF: 2008/056).

References *References marked with an asterisk indicate key article/document for final 12 measures. *Bar-On, R. (1997). Bar-On Emotional Quotient Inventory (EQ-i): Technical manual. Toronto, Ontario, Canada: Multi-Health Systems.

Downloaded from epm.sagepub.com by guest on January 1, 2012

635

Humphrey et al.

Bar-On, R., & Parker, J. (2008). Bar-On Emotional Quotient Inventory: Youth Version (EQ-i:YV): Technical manual. Toronto, Ontario, Canada: Multi-Health Systems. Butler, R. J., & Gasson, S. L. (2005). Self-esteem/self-concept scales for children and adolescents: A review. Child and Adolescent Mental Health, 10, 190-201. *Carlo, G., Haussman, A., Christiansen, S., & Randall, B. A. (2003). Sociocognitive and behavioural correlates of a measure of pro-social tendencies for adolescents. Journal of Early Adolescence, 23, 107-134. Denham, S. A. (2005a). Assessing social-emotional development in children from a longitudinal perspective for the National Children’s Study: Social-emotional compendium of measures. Fairfax, VA: George Mason University. Denham, S. A. (2005b). Assessing social-emotional development in children from a longitudinal perspective for the National Children’s Study. Fairfax, VA: George Mason University. Denham, S. A., Wyatt, T. M., Bassett, H. H., Echeverria, D., & Knox, S. S. (2009). Assessing social-emotional development in children from a longitudinal perspective. Journal of Epidemiology and Community Health, 63(Suppl.), 37-52. Department for Children, Schools and Families. (2007a). Secondary Social and Emotional Aspects of Learning (SEAL) programme: Guidance. Nottingham, England: Author. Department for Children, Schools and Families. (2007b). The children’s plan: Building brighter futures. Nottingham, England: Author. Gannon, N., & Ranzijn, R. (2005). Does emotional intelligence predict unique variance in life satisfaction beyond IQ and personality? Personality and Individual Differences, 38, 1353-1364. Fundacion Marcelino Botin. (2008). Social and emotional education: An international analysis. Santander, Spain: Author. Goleman, D. (1995). Emotional intelligence. New York, NY: Bantam Books. *Gresham, D. C., & Elliott, S. N. (1990). Social skills rating system: Manual. Circle Pines, MN: American Guidance Service. Hannes, K., Claes, L., & The Belgian Campbell Group. (2007). Learn to read and write systematic reviews: The Belgian Campbell Group. Research on Social Work Practice, 17, 748-753. Harlen, W., & Crick, R. D. (2004). Opportunities and challenges of using systematic reviews of research for evidence based policy in education. Evaluation & Research in Education, 18, 54-71. *Hightower, A. D., Cowen, E. L., Spinell, A. P., & Lotyczewski, B. S. (1987). The Child Rating Scale: The development of a socio-emotional self-rating scale for elementary school children. School Psychology Review, 16, 239-255. Humphrey, N., Curran, A., Morris, E., Farrell, P., & Woods, K. (2007). Emotional intelligence and education: A critical review. Educational Psychology, 27, 235-254. Humphrey, N., Kalambouka, A., Bolton, J., Lendrum, A., Wigelsworth, M., Lennie, C., & Farrell, P. (2008). Primary social and emotional aspects of learning: evaluation of small group work (Research Report RR064). Nottingham, England: DCSF Publications. Humphrey, N., Kalambouka, A., Lendrum, A., Wigelsworth, M., Samad, M., Farrelly, N., . . . Croudace, T. (2009). A systematic review of social and emotional skills measures for children and young people (DCSF Research Report 2008/056B). Nottingham, England: DCSF Publications.

Downloaded from epm.sagepub.com by guest on January 1, 2012

636

Educational and Psychological Measurement 71(4)

*Izard, C. E., Dougherty, B. M., Bloxom, B. M., & Kotsch, W. E. (1974). The Differential Emotion Scale: A method of measuring the subjective experience of discrete emotions. Unpublished manuscript, Vanderbilt University, Nashville, TN. *LaFreniere, P. J., & Dumas, J. E. (1996). Social competence and behavior evaluation in children ages 3 to 6 years: The short form (SCBE-30). Psychological Assessment, 8, 369-377. *Matson, J. L., Rotatori, A. F., & Helsel, W. J. (1983). Development of a rating scale to measure social skills in children: The Matson Evaluation of Social Skills with Youngsters (MESSY). Behavior Research and Therapy, 21, 335-340. Matthews, G., Roberts, R. D., & Zeidner, M. (2004). Seven myths about emotional intelligence. Psychological Inquiry, 15, 179-196. Matthews, G., Zeidner, M., & Roberts, R. D. (2004). Emotional intelligence: Science and myth. Cambridge: MIT Press. *Merrell, K. W. (1996). Social-emotional assessment in early childhood: The Preschool and Kindergarten Behavior Scales. Journal of Early Intervention, 20, 132-145. *Michelson, L., & Wood, R. (1982). Development and psychometric properties of the Children’s Assertive Behavior Scale. Journal of Behavioral Assessment, 4, 3-13. Nind, M. (2006). Conducting systematic review in education: A reflexive narrative. London Review of Education, 4, 183-195. *Nowicki, S., Jr., & Duke, M. P. (1989). A measure of nonverbal social processing ability in children between the ages of 6 and 10. Paper presented as part of a symposium at the American Psychological Society, Alexandria, VA. Payton, J. W., Wardlaw, D. M., Graczyk, P. A., Bloodworth, M. R., Tompsett, C. J., & Weissberg, R. P. (2000). Social and emotional learning: A framework for promoting mental health and reducing risk behaviours in children and youth. Journal of School Health, 70, 179-185. Petrides, K. V., & Furnham, A. (2001). Trait emotional intelligence: Psychometric investigation with reference to established trait taxonomies. European Journal of Personality, 15, 425-448. Petrides, K. V., Sangareau, Y., Furnham, A. and Frederickson, N. (2006). Trait emotional intelligence and children’s peer relations at school. Social Development, 15, 537–547. Rose-Krasnor, L. (1997). The nature of social competence: A theoretical review. Social Development, 6, 111-135. Salovey, P., & Mayer, J. D. (1990). Emotional intelligence. Imagination, Cognition and Personality, 9, 185-211. Schulte, M. J., Ree, M. J., & Carretta, T. R. (2004). Emotional intelligence: Not much more than g and personality. Personality and Individual Differences, 37, 1059-1068. *Schultz, D., Izard, C. E., & Bear, G. A. (2004). Children’s emotion processing: Relations to emotionality and aggression. Development & Psychopathology, 16, 371-387. *Shields, A., & Cicchetti, D. (1997). Emotion regulation among school-age children: The development and validation of a new criterion Q-sort scale. Developmental Psychology, 33, 906-916. Stewart-Brown, S., & Edmunds, L. (2003). Assessing social and emotional competence in primary-school and early years settings (DfES EOR/SBU/2002/042). Nottingham, England: DfES Publications. Terwee, C. B., Bot, S. D., de Boer, M. R., van der Windt, D. A., Knol, D. L., Dekker, J., . . . de Vet, H. C. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60, 34-42.

Downloaded from epm.sagepub.com by guest on January 1, 2012

637

Humphrey et al.

Torgerson, C. J. (2006). Publication bias: The Achilles’ heel of systematic reviews? British Journal of Educational Studies, 54, 89-102. Van Horn, M., Atkins-Burnett, S., Karlin, E., & Snyder, S. (2007). Parent ratings of children’s social skills: Longitudinal psychometric analysis of the social skills rating system. School Psychology Quarterly, 22, 142-199. Van Rooy, D. L., Viswesvaran, C., & Pluta, P. (2005). An evaluation of construct validity: What is this thing called emotional intelligence? Human Performance, 18, 445-462. Weare, K., & Gray, G. (2003). What works in developing children’s emotional and social competence and well-being? (RR456). Nottingham, England: DfES Publications. Wigelsworth, M., Humphrey, N., Kalambouka, A., & Lendrum, A. (2010). A review of key issues in the measurement of children’s social and emotional skills. Educational Psychology in Practice, 26, 173-186. Wilhelm, O. (2005). Measures of emotional intelligence: Practice and standards. In R. Schulze & R. D. Roberts (Eds.), Emotional intelligence: An international handbook (pp. 131-154). Cambridge, MA: Hogrefe & Huber. Wolpert, M., Aitken, J. Syrad, H., Munroe, M., Saddington, C., Trustam, E., . . . Croudace, T. (2009). Review and recommendations for national policy for England for the use of mental health outcome measures with children and young people (DCSF Research Report 2008/56). Nottingham, England: DCSF Publications. Zeidner, M., Roberts, R. D., & Matthews, G. (2002). Can emotional intelligence be schooled? A critical review. Educational Psychologist, 37, 215-231.

Downloaded from epm.sagepub.com by guest on January 1, 2012