Coding Joint Engagement Live in School-Based Research: Reliability and Psychometric Considerations Jessica R. Dykstra, Brian A. Boyd, Linda R. Watson, Caroline McCarty, Grace T. Baranek, & Betsy R. Crais University of North Carolina at Chapel Hill
BACKGROUND • Active engagement is a critical component of interventions for children with autism (NRC, 2001) • Engagement is concurrently and predictively related to other developmental abilities and skills • Language (Adamson et al., 2004; 2009) • Academic outcomes (Greenwood et al., 1991, Logan et al., 1997) • Joint engagement focuses on the social nature of engagement and may be a particularly relevant measure for individuals with autism given their social deficits
OBJECTIVES
3. Describe and examine the process of measuring and attaining reliability
RESEARCH PROJECTS Two different studies needed a tool to assess joint engagement in live classroom settings during typical classroom activities or routines. Study
Study 1: ASAP Project
Type of study Multi-site, randomized control trial
Study 2: Student Engagement Descriptive study for dissertation
Ages
3 – 5 year olds
8 – 12 year olds
Setting
Self-contained preschool classrooms
Self-contained elementary and middle classrooms
Observations Three 5-min. segments during typical classroom activities
Six 5-min. segments during classroom instruction (1:1, small group and large group)
• Child exhibits no apparent engagement with a specific person or object • Examples: wandering, maladaptive behaviors, rapidly flipping pages in a book
Unengaged
• Child watches another person who is engaged in an activity, intently observes the person or objects the person is manipulating • Examples: watching a peer play with blocks, watching the teacher read a book
Onlooking Object only
• Child explores or plays with object(s) on his/her own, other people are not influencing the play • Examples: pretending to make cookies, stacking blocks, putting together a puzzle
Person only
• Child interacts with another person, objects are not a part of the interaction • Examples: playing chase, singing a song with a teacher, engaging in a conversation
Supported joint engagement
• Child and other person engage in same activity, person influences the activity, but the child does not acknowledge the other person • Examples:
Coordinated joint engagement
• Child and other person engage in same activity, child coordinates attention to objects and people in an alternating/integrated manner • Examples:
PSYCHOMETRIC CHARACTERISTICS
1. Describe the joint engagement coding system adapted for live coding 2. Examine the psychometric characteristics of the coding system
ADAPTATIONS
CODING SCHEME (Adapted from Adamson et al., 1998)
STUDY 1 Unengaged Onlooking Object Person Supported Joint Coordinated Joint
Mean .39 .11 .33 .05 .02 .10
SD .28 .17 .32 .10 .07 .18
Min .01 .00 .00 .00 .00 .00
Max .92 .67 .99 .65 .44 .74
STUDY 2 Unengaged Onlooking Object Person Supported Joint Coordinated Joint
Mean .42 .17 .12 .04 .04 .21
SD .29 .20 .18 .06 .08 .24
Min .02 .00 .00 .00 .00 .00
Max .97 .76 .75 .32 .47 .90
Study 2
Study 1
Single Engagement 38%
Joint Engagement 25%
Single Engagement 16%
CONCLUSIONS • JE coding system is feasible for use in live classroom settings in preschools, as well as elementary and middle school classrooms • JE coding captures range of performance, but may not result in normal distributions
Joint Engagement 12%
Non/Passive Engagement 50%
Adaptations for Live Coding • Coding multiple 5-minute segments per child • Revise and add examples and non-examples relevant to classroom settings • 3 second rule: coders count to 3 before changing states • Create cheat sheets for live coding Training Procedure • Practice coding using 5-minute video segments • Practice coding in live classrooms • Final reliability using 5-minute video segments
Non/Passive Engagement 59%
• Differences in engagement across the studies may be reflective of: • Methodological differences: Study 2 coded instructional time with teachers, while Study 1 coded more varied activities
• Skewness and kurtosis indicate normal distributions for combined categories (i.e., NPE, SE, JE)
• Setting differences: Focus of preschool differs from elementary and middle school
• However, the distributions for some individual categories were non-normal in each study
• Population differences: Engagement may look different over the course of development or may differ based on other child characteristics
• Study 1: OL, PER, SJE, & CJE • Study 2: PER & SJE
RELIABILITY • Overall reliability estimates were .83 for Study 1 and .80 for Study 2 with 6-second error window • Average percent agreement across both studies >.80 for each engagement state category • Intraclass correlations for the proportion of time spent in engagement states • Study 1: >.90 except for SJE (.33) • Study 2: >.80 except for OL (.75), SJE (.36)
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant #R324A110256 to the University of North Carolina at Chapel Hill. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
• Overall reliability is acceptable, but patterns across two studies suggest coders have limited agreement on proportions of time spent in SJE
REFERENCES Adamson, L. B., Bakeman, R., & Deckner, D. F. (2004). The development of symbol-infused joint engagement. Child Development, 75(4), 1171-1187. Adamson, L. B., Bakeman, R., Deckner, D.F., & Romski, M. (2009). Joint engagement and the emergence of language in children with autism and Down syndrome. Journal of Autism and Developmental Disorders, 39(1), 84-96. Adamson,L. B., Bakeman, R., Russell, C. L., & Deckner, D. F. (1998). Coding Symbol-Infused Engagement States, Technical Report 9. Atlanta, GA: Developmental Laboratory at Georgia State University. Greenwood, C. R., Carta, J. J., & Atwater, J. (1991). Ecobehavioral analysis in the classroom: Review and implications. Journal of Behavioral Education, 1(1), 59-77. Logan, K. R., Bakeman, R., & Keefe, E. B. (1997). Effects of instructional variables on engaged behavior of students with disabilities in general education classrooms. Exceptional Children, 63(4), 481-497. National Research Council (2001). Educating children with autism. Washington, DC: National Academy Press.
For additional information, email Jessica Dykstra at
[email protected] or the research team at
[email protected] 