Prev Sci (2014) 15:165–176 DOI 10.1007/s11121-012-0359-1
Engagement Matters: Lessons from Assessing Classroom Implementation of Steps to Respect: A Bullying Prevention Program Over a One-year Period Sabina Low & Mark J. Van Ryzin & Eric C. Brown & Brian H. Smith & Kevin P. Haggerty Published online: 2 March 2013 # Society for Prevention Research 2013
Abstract Steps to Respect: A Bullying Prevention Program (STR) relies on a social–ecological model of prevention to increase school staff awareness and responsiveness, foster socially responsible beliefs among students, and teach social–emotional skills to students to reduce bullying behavior. As part of a school-randomized controlled trial of STR, we examined predictors and outcomes associated with classroom curriculum implementation in intervention schools. Data on classroom implementation (adherence and engagement) were collected from a sample of teachers using a weekly on-line Teacher Implementation Checklist system. Pre-post data related to school bullying-related outcomes were collected from 1,424 students and archival school demographic data were obtained from the National Center for Education Statistics. Results of multilevel analyses indicated that higher levels of program engagement were influenced by school-level percentage of students receiving free/ reduced lunch, as well as classroom-level climate indicators. Results also suggest that higher levels of program engagement were related to lower levels of school bullying problems, enhanced school climate and attitudes less supportive of bullying. Predictors and outcomes related to program fidelity (i.e., adherence) were largely nonsignificant. Results suggest that student engagement is a key element S. Low (*) T. Denny Sanford School of Social and Family Dynamics, Arizona State University, Tempe, AZ, USA e-mail:
[email protected] M. J. Van Ryzin Wichita State, Wichita, USA E. C. Brown : K. P. Haggerty Social Development Research Group, University of Washington, Seattle, WA, USA B. H. Smith Committee for Children, Seattle, WA, USA
of program impact, though implementation is influenced by both school-level demographics and classroom contexts. Keywords School bullying . Program implementation . Fidelity . Prevention . Steps to Respect
Introduction Given the high prevalence of bullying in schools and its strong relationship to adverse mental health outcomes, there has been considerable growth in the number and dissemination of school-based bullying prevention programs over the past 20 years. Part of this growth is spurred by states requiring school districts to adopt policies and practices to address bullying and harassment among students. However, as with other K-12 prevention curricula, the development, testing, and dissemination of interventions do not automatically lead to strong public health impacts. Indeed, several recent meta-analyses reveal that school-wide bullying prevention evaluations have demonstrated negligible to non-significant results (Merrell et al. 2008; Smith et al. 2004), with the more promising studies being based in Europe (Farrington and Ttofi 2009). It is becoming increasingly accepted that implementation (i.e., how well a program is put into practice) strongly influences whether programs are maximally effective (Durlak and DuPre 2008; Dusenbury et al. 2003). Optimal implementation may be particularly challenging for bullying prevention programs, which often entail whole-school approaches with required coordination among different social–ecological levels (e.g., teachers and administrators). Implementation not only serves as a bridge between science and practice, but knowledge of how a program was conducted in research studies has implications for internal validity (linking program model to outcomes), external validity (i.e., replication), program theory (i.e., how does a program work), and our ability to accurately
166
determine program impact (i.e., minimize Type I or II errors; Botvin 2004; Durlak and DuPre 2008). Implementation and Bullying Prevention Efforts There is now ample evidence that implementation influences outcomes of school-based prevention programs and may have a dramatic impact on effect sizes, though the majority of studies have derived from drug abuse prevention research (Durlak and DuPre 2008). Relatively fewer bullying prevention programs have measured and examined the role of program implementation, especially in the context of a rigorously designed study (Ryan and Smith 2009). Similar to the general prevention science literature, higher implementation fidelity has enhanced program impact (see Olweus 1991; Smith et al. 2004), and in other cases, effects were only found in highimplementation schools (Salmivalli et al. 2005; noteably, implementation was aggregated at school level). Hirschstein et al. (2007) studied the implementation of Steps to Respect: A Bullying Prevention Program (STR; Committee for Children 2005) in a previous quasi-experimental trial and were one of the first to measure both implementation fidelity (adherence) and lesson delivery quality on anti-bullying program effects. Implementation measures included adherence (or fidelity) to essential lesson components, quality of lesson instruction, teacher support for the generalization of students’ social– emotional and bullying prevention skills, and teacher coaching of students involved in bullying incidents (Hirschstein et al. 2007). Hirschstein et al. (2007) found that STR is most effective when teachers adhere more closely to lesson content, support student use of program skills beyond the lessons, and coach students involved in peer victimization. Taken together, these studies highlight the importance of implementation features in bullying prevention, as well as the need to consider the multi-dimensional nature of program implmentation. Multiple Dimensions of Classroom Implementation Five features most commonly cited as important aspects of prevention program implementation are: fidelity (i.e., adherence, replication), dosage (i.e., quantity), quality of delivery (enthusiasm, preparedness), participant responsiveness (i.e., student attitudes and adoption of program), and program differentiation (i.e., program uniqueness; Dane and Schneider 1998; Durlak and DuPre 2008). Despite the potential importance of these different dimensions, the majority of implementation research has focused on fidelity and, to a lesser extent, dosage. Specifically, the meta-analysis by Durlak and DuPre (2008) found 59 published studies that examined links between implementation and program outcomes for child/adolescent health promotion programs. Of these studies, 37 (63 %) evaluated fidelity, while 29 (50 %) assessed dosage, and only 2 looked at quality of delivery
Prev Sci (2014) 15:165–176
(3.3 %); overall, 69 % measured only one dimension of implementation and 31 % measured fidelity and dosage. These figures highlight the predominantly narrow focus of implementation research, and the need to consider other important dimensions of implementation. There is a clear need for more research attention to participant responsiveness to prevention interventions more generally (Dusenbury et al. 2003; Dusenbury et al. 2005; Ringwalt et al. 2009), and bullying prevention in particular. For example, Durlak and Dupre (2008) found participant responsiveness had not received sufficient research attention to be included in their meta-analysis. Participant responsiveness refers typically to student attitudes toward a program (i.e., motivation, adoption), and interest and participation in program lessons (Carroll et al. 2007; Dusenbury et al. 2003; Rohrbach et al. 2010). A related construct, student engagement, has been studied widely in education research and has been shown consistently to be related to school achievement (Fredericks et al. 2004; Marks 2000). Some research in the prevention field has indicated that student engagement with a prevention program increases positive outcomes (Ringwalt et al. 2009), suggesting the importance of further research. Thus, one aim of the current study is to examine and contrast the influence of adherence (i.e., fidelity) and student engagement (i.e., active participation, cooperativeness) of classroom implementation on effects of a school-randomized trial of a bullying prevention program. Factors Influencing Program Implementation The second aim of the current study is to examine predictors of curriculum adherence and classroom engagement, as there are important implications for designing and delivering effective interventions and supports (Fagan and Mihalic 2003). Much as bullying prevention interventions are typically informed by a social–ecological perspective, implementation of prevention programs can also be affected by factors at varying levels of the ecology of schools and communities (Altschuld et al. 1999; Durlak and DuPre 2008; Ozer 2006; Shediac-Rizkallah and Bone 1998; Wandersman 2003). Feasibility of implementation is negatively impacted at the program level by factors such as greater program complexity, and at the school level by instability due to issues such as budget difficulties or high student turnover (Thaker et al. 2008). Botvin (2004) identifies several impediments to proper substance abuse prevention program execution, including school level factors (e.g., resource limitations, unmanageable class sizes and lack of time for adequate preparation and full implementation of the program due to increased pressure for success in core academic areas) and teacher level factors (e.g., insufficient training and support, issues with classroom
Prev Sci (2014) 15:165–176
management and discipline, and teacher apathy). Research in bullying prevention has identified other teacher level factors that impact implementation. A teacher who has read program materials, recognizes and values his or her ability to affect the problem of peer victimization, and has the ability to empathize with victims of bullying is likely to have a higher degree of adherence (Olweus 2004). Furthermore, an instructor’s acknowledgement of bullying as a problem generally, and in his or her school specifically, may also influence the percentage of the curriculum that he or she utilizes (Olweus 2004). Classroom Context Although most prevention curricula are delivered in classrooms, characteristics of this milieu are surprisingly neglected in investigations of program implementation. After all, teacher and student characteristics are dynamically inter-related (transactional), which implies that classroom climate and student characteristics (e.g., proportion of aggressive youth), seem inherently interdependent with key components of implementation. There are a few studies that have examined how classroom context affect program outcomes. For example, Aber et al. (1998) found that classrooms with more aggressive norms undermined program effects, yet Grossman et al. (1997) found no effect of classroom climate on effects of a universal violence prevention program. Though, there is little to no a priori data on the interaction between classroom context and teacher-led implementation. Drawing on the framework proposed by Ozer (2006) one might reasonably speculate that factors influencing classroom climate and management (e.g., relationships, norms, aggression) would potentially influence both teachers’ adoption of specified curricula, as well as student engagement. That is, features of classroom climate, such as the level of trust and quality of relations among students, seem not only necessary for programs to be delivered as intended, but they also allow for better organization/management which would facilitate engagement. In addition, norms around bystander involvement and attitudes among students (toward intervention and violence), as well as their own experience with bullying victimization (prior to intervention), may modify students’ orientation toward and motivation to participate in programming. Conversely, having a higher density of children who are themselves aggressive may undermine the conditions of organization and trust needed for optimal implementation. Given this, we hypothesized that more prosocial student attitudes, perceived student and staff intervention in bullying, and more positive student/school climate at baseline would be positively related to engagement and adherence, while bullying involvement would be negatively related to both these implementation factors. Victimization experience at baseline was hypothesized to relate to more engagement, but no relation with adherence was expected.
167
School-Level Context The broader community in which schools are located, or the communities represented in the student population (i.e., characteristics of communities from which most children are being drawn) can impact prevention program implementation, but much of the discussion of community factors has focused on funding and aspects of the prevention system (Durlak and DuPre 2008). However, higher levels of community factors such as community disorganization, student poverty and urbanicity appear to affect the functioning of schools through negatively impacting the academic attitudes and achievement of students (Gottfredson 2001; Laub and Lauritsen 1998; Ozer 2006). It is not clear how these risk factors, and student demographic characteristics in particular, influence violence prevention implementation. One study found that communities with higher concentrations of poverty and AfricanAmericans had greater success at some aspects of prevention program implementation (Payne et al. 2006). On the other hand, student engagement has been found to influence the success of prevention programs, and at least in terms of schooling in general, research has found that levels of engagement are lower for students from lower social–economic levels (Lee and Smith 1993, 1994; Yazzie-Mintz 2007). Purpose of Current Study The current study assesses classroom- and school-level predictors of and outcomes associated with classroom implementation (i.e., adherence and engagement) of Steps to Respect: A Bullying Prevention Program© (STR) over a 1-year period. The STR program is based on a social–ecological model of bullying which views youth behavior as shaped by multiple factors within nested contextual systems at the school, peer, and individual levels (Swearer et al. 2010). School-wide components are intended to foster a positive school climate and positive norms through teacher and staff training focused on the creation of effective disciplinary policies, improved monitoring of students and instruction on how to effectively intervene with students involved in bullying situations. Classroom curricula are intended for the upper three elementary grades and seek to promote socially responsible norms and behavior and increase social–emotional skills. Lessons help students recognize bullying, increase empathy for students that are bullied, build friendship skills to increase protective social connections, improve assertiveness and communication skills to help students deter and report bullying and teach appropriate bystander responses to bullying. A randomized controlled trial of Steps to Respect (Brown et al. 2011), indicated positive effects on student attitudes, bullying related behavior (e.g., social competence, physical bullying perpetration, positive bystander behavior) and school climate. Analyses for this trial used hierarchical linear modeling, allowing for estimation of the influence of the nested data
168
(e.g. teachers within classrooms, classrooms within schools). In keeping with a social–ecological model, the present study uses multilevel analysis to enhance our understanding of what predicts strong classrrom implementation, as well as how classroom implementation influences student outcomes in these three domains (behaviors, attitudes, climate) in the context of school community demographics. Specifically, we hypothesized that prosocial attitudes (i.e., attitudes against bullying and toward bullying intervention) and behaviors (i.e., bystander behavior, intervention), and more positively perceived climate of trust and support at the classroom level (i.e., aggregated student perceptions) would relate to higher levels of engagement and adherence. Second, we hypothesized that higher proportions of aggressive youth involved in bullying prior to intervention would undermine engagement and adherence. Third, higher levels of prior victimization were expected to relate to higher levels of engagement in the program, but no effect on adherence was anticipated. Fourth, staff perceptions of positive school climate, staff bullying intervention and perceived levels of school bullying problems were expected to relate to higher levels of adherence and engagement, to the extent they represent acknowledgement of bullying as a problem, attitudes favorable toward bullying prevention effotrs, as well as empathy for students (in the context of good leadership/morale). Fifth, school policies/procedures regarding bullying were expected to relate to higher adherence to curriculum (see Olweus 2004; Ozer 2006), while schools with a higher proportion of students from low income households were expected to have compromised classroom implementation. Lastly, we hypothesized, similar to previous bullying prevention trials, that enhanced classroom implementation would be associated with more positive program effects on attitudes, behaviors, and school climate.
Methods Participants Schools The STR efficacy trial was conducted in 33 elementary schools in north-central California. Twenty-five percent of the schools were from rural areas, 10 % was from small towns, 50 % was from suburban areas, and 15 % was located in midsized cities. On average, 40 % of students received free or reduced-price lunch (SD= 29 %). The mean number of students per school was 479 (SD=177, range=77 to 749 students) and the mean number of teachers per school was 24. Since this study focused on program implementation, only the 17 schools in the intervention condition were included.
Prev Sci (2014) 15:165–176
Teachers We randomly selected four classrooms from third to fifth grade for data collection from participating intervention schools, for a total of 128 classrooms (n=52 thirdgrade, n=62 fourth-grade, n= 11 fifth-grade, n=2 thirdand fourth-grade split classroom, and n = 1 fourth- and fifth-grade split classroom). Average classroom size was 23.36 (SD=6.49, range=14–34). Teachers from each of these classrooms consented actively to participate in the study (100 % participation). Teachers were asked to complete individual online assessments of student behavior and teachers were asked to record program implementation progress. Teachers were reimbursed $5 per student for their time in completing the online assessment of student behavior at pretest (fall) and posttest (spring), with an additional bonus of $25 if all assessments were completed within 2 weeks of the initial announcement of the survey. Teachers were reimbursed an additional $75 if implementation data were given on all 11 STR classroom lessons by the end of the study. Students All students in each of the selected classrooms were included in the target sample for completion of the student survey, yielding a final analysis sample of 1,424 students (94 % of the target population of students). Approximately half (51 %) of the student sample was male, 52 % was White, 7 % was African American, 6 % was Asian American, 35 % was of other or mixed race. Fortytwo percent of students identified as of Hispanic origin. Students ranged in age from 7 to 11 years. Students received a small gift worth about $5 for each interview they completed in the fall (pretest) and spring (posttest).
Program Components Staff Training Committee for children trainers provided the standard on-site, 1-day training for all participating teachers and staff. The training included an overview of program goals and key features of program content (e.g., a definition of bullying, a model for responding to bullying reports). Teachers, counselors, and administrators received additional training in how to coach students involved in bullying. It is recommended that all schools using the STR program must have an anti-bullying policy, and the training provides examples as well as guidelines for important components of those policies. For the purposes of this intervention, the only requirement for those randomized to intervention condition was that they have a bullying policy. Third- through fifth-grade teachers also received an overview of classroom materials and program-specific instructional strategies. Classroom Curriculum Eleven semi-scripted skills lessons focusing on social–emotional skills for positive peer
Prev Sci (2014) 15:165–176
relations; emotion management; and recognizing, refusing, and reporting of bullying behavior were delivered by teachers. Lesson topics included joining groups, distinguishing reporting from tattling, and being a responsible bystander. Instructional strategies included direct instruction, largeand small-group discussions, skills practice, and games. Weekly lessons, totaling about 1 h, were taught over 2 to 3 days. Upon completion of skill lessons, teachers implemented a grade-appropriate literature unit, based on children's books that provided deeper opportunities to explore bullying-related themes. Measures Program Implementation Checklist Various aspects of implementing the classroom curriculum portion of the STR program were assessed using web-based Program Implementation Checklists (PIC; Low, Brown, & Smith, 2008). These were developed for each of the 11 lessons in each of the 3 years or levels of the curriculum. Items on the PIC asked teachers to report the percentage of students that were present for the lesson, the amount of time teachers spent preparing for and delivering the lesson, the degree to which students were engaged in the lesson, modifications teachers may have made to the lesson, and activities that teachers may have added to reinforce a lesson. Additionally, teachers were asked to report on whether they completed each of the five to eight specific activities outlined in each lesson. The student lesson engagement construct consisted of the average of four questions: (a) “To what extent were your students engaged by this lesson (e.g., asking questions, volunteering)?” (b) “To what extent were students distracting other students during this lesson?” (c) “To what extent do you feel your students could demonstrate the objectives of this lesson?,” and (d) “How difficult was it to manage students during this lesson?” Response options for these questions were coded as 0 = Not at all, 1 = A little, 2 = Some, and 3=A lot. Each of these four questions were averaged first across all lessons, then across the four questions (M=2.34, SD =.40) . Coefficient alpha of the four questions (across teachers and lessons) was .86. The second measure of program implementation assessed teacher lesson adherence . This measure was calculated as the percentage of a sample (Md=6, range=5 to 9) of activities for a lesson (e.g., Reviewed the definition of “respect,” Showed the video “Connect with Respect—Finding Friends,” Had students read Student Handout 1: Identify the Problem, Gave student Family Handout 2: Conflict between Friends), with percentages for each lesson averaged over lessons 4 through 11 (during electronic data transmission back to the University of Washington, fidelity data from lessons 1 through 3 were corrupted and unrecoverable). On average, 92 % of sampled lesson activities was
169
completed and 67 % of teachers did all sampled lesson activities. Across all lessons, 18 % of teachers reported omitting one or more elements from a lesson. Student Survey Student pre- and posttest survey data were collected using a revised version of the Colorado Trust’s Bullying Prevention Initiative Student Survey (Csuti 2008). The 11 scales contained in the survey, number of items per scale, scale coefficient alphas, and sample items for the posttest administrations of the survey are presented in Table 1. Scale scores for Student Survey measures were constructed as the mean of all nonmissing items. If more than 2/3 of items in a scale were missing, that scale score was set to missing. School Environment Survey Pretest and posttest data were collected from school staff during school staff training sessions (in intervention schools only) or during in-service meetings using the School Environment Survey (SES). The SES is a brief (10-min), anonymous, paper-and-pencil survey, which was adapted for the current study from the Colorado Trust’s 3-year statewide Bullying Prevention Initiative (Csuti 2008). The SES was designed to parallel several of the measures collected from the Student Surveys to provide an alternative source of information on the social–ecological context of the school environment. School staff were asked about their perceptions of their school’s climate, staff and student willingness to intervene in observed bullying, bullying-related problems in their school, their school’s anti-bullying policies and strategies, and background demographic information (age, gender, race/ethnicity, how many years they worked at the school, and their position at the school). Scale scores for outcome measures were created as the mean of all nonmissing items on the scale and were considered missing if more than 67 % of items in that scale were missing. The four SES measures used to predict implementation were: School Anti-bullying Policies and Strategies was measured by eight items that asked school staff about how much their school was doing with regard to policies and strategies to prevent bullying (e.g., Demonstrating administrator commitment and leadership to address bullies, bullied, and bystanders). Responses were recorded on a four-point Likert-type scale ranging from 1=Not at all to 4=A lot (M=2.83, SD=.76). Coefficient alpha for this measure was .93. School Climate was measured by seven items that asked school staff how much they were connected or bonded, or shared the same values with each other, with students, and with parents (e.g., …Staff in this school can be trusted.). Responses were recorded on a fourpoint Likert-type scale ranging from 1 = Strongly
170
Prev Sci (2014) 15:165–176
Table 1 Characteristics of student survey outcome measures Outcome
Number of items
Coefficient alpha
Sample item/response options
7
.88
4
.79
How okay is it when students tease weaker students in front of others?(1=Extremely wrong to 5=Very okay) How okay is it when students defend others who are being shoved around by strong students? (1=Extremely wrong to 5=Very okay)
4
.77
Teacher/Staff Bullying Intervention
4
.75
Positive Bystander Behavior
5
.71
Bullying Perpetration
7
.87
Bullying Victimization
4
.75
A particular student or group of students pushed, shoved, tripped, or picked a fight with me.(1=Never to 5=A lot)
School Climate Student Support
6
.76
Student Climate
4
.69
School Connectedness
5
.76
Staff Climate
7
.82
Students my age can be trusted a lot (1=Not true at all to 4=Completely true) Students in my school are willing to help each other. (1=Strongly disagree to 4=Strongly agree) This is a pretty close-knit school where everyone looks out for each other. (1=Strongly disagree to 4=Strongly agree) Teachers/staff in my school can be trusted. (1=Strongly disagree to 4=Strongly agree)
Student Attitudes Toward Bullying Student Attitudes Against Bullying Student Attitudes Toward Intervention In Bullying Incidents Student Prosocial and Bullying Behavior Student Bullying Intervention
disagree to 4 = Strongly agree (M = 3.46, SD = .47). Coefficient alpha for this measure was .91. Staff Bullying Intervention was measured by five items that asked school staff to report how often it was that staff at their school would intervene in different bullyingrelated behaviors (e.g., A student or group of students is trying to pick a fight with a weaker student.). Responses were recorded on a seven-point Likert-type scale ranging from 1=Never to 7=Every Time (M=5.99, SD=.91). Coefficient alpha for this measure was .95. School Bullying-Related Problems consisted of seven items that asked school staff how much of a problem certain issues were in their school (e.g., Students spreading rumors or lies about students they are mad at or don’t like.). Responses were recorded on a fourpoint Likert-type scale ranging from 1=Not a problem to 4=A huge problem (M=3.60, SD=1.05). Coefficient alpha for this measure was .82. School Characteristics Data on the size of intervention schools and the demographic characteristics of their students
Students in your school would help out if a student spread rumors or lies about another student behind their back. (1=Never to 4=Always) Teachers and staff in your school would help out if a student or group of students pushed, shoved or tried to pick a fight with a weaker student. (1=Never to 4=Always) I ignored rumors or lies that I heard about other students. (1=Never to 5=A lot) I teased or said mean things to certain students.(1=Never to 5=A lot)
were obtained from the National Center for Educational Statistics (NCES; http://www.ed-data.k12.ca.us). The following variables were used as school-level predictors of program implementation: total student enrollment (M=479, SD=180), percentage of Hispanic students (M=33, SD= 30), percentage of White students (M=51, SD=30, and percentage eligible for the free or reduced-price lunch program (FRPL; M=40, SD=29). Statistical Analyses Prediction of Classroom Implementation Statistical analyses for the examination of predictors of student lesson engagement and lesson adherence consisted of twolevel (i.e., classrooms nested within schools) hierarchical linear models (HLM; Raudenbush and Bryk 2002), with the two continuously distributed implementation measures regressed, respectively, on pretest student survey measures (at level 1, averaged by classroom) and staff survey data and school characteristics (at level 2).
Prev Sci (2014) 15:165–176
For example, classroom levels of students’ lesson engagement regressed upon (a) classroom levels of studentreported perceived School Climate (SCLIMATE) measured at pretest and (b) school-level percentage of students eligible for free or reduced-price lunch (FRPL) is illustrated in the following model: Level 1 (Classroom) ENGAGEMENT ¼ b 0 þ b1 ðSCLIMATE1 Þ þ e Level 2 (School) b 0 ¼ g 00 þ g 01 ðFRPLÞ þ r0 Classroom-level intercepts (β0) were modeled as random effects to assess the variability in levels of engagement across schools. Because of the large number of predictors, we first assessed each predictor’s zero-order relationship with each outcome and subsequently entered significant (p < .05) predictors into a final multivariable model. Intraclass correlation coefficients (ICCs) were calculated as the proportion of total variability attributed to differences in program implementation measures across schools. Outcomes of Classroom Implementation Program implementation outcomes were examined using three-level HLMs (i.e., students nested within classrooms nested within schools), with teachers’ program implementation measures at level 2 predicting student posttest outcomes controlling for pretest levels of the student outcome in question and student characteristics (i.e., age, gender, race and ethnicity) at level 1, and school demographic characteristics and staff perceptions (at level 3). An example of this model is shown as follows: Level 1 (Student) BULLYPERP2 ¼ b 0 þ b1 ðAGE Þ þ b2 ðSEX Þ þb 3 ðWHITE Þ þ b4 ðHISPÞ þ b 5 ðBULLYPERP1 Þ þ e Level 2 (Classroom) b 0 ¼ g 00 þ g 01 ðENGAGEMENT Þ þ r0 Level 3 (School) g 00 ¼ x000 þ x001 ðFRPLÞ þ u00 where posttest Bullying Perpetration (BULLYPERP2) is regressed on students’ age, sex, race/ethnicity (WHITE, coded 1=White and 0=Other; and HISP, coded 1=Hispanic and 0=Other), and pretest Bullying Perpetration (BULLYPERP1) with random effects across classrooms and schools for the prediction of teacher-level engagement scores and school-level percentages of students eligible for FRPL.
171
indicated that 11 % and less than 1 % of the variability in program engagement and adherence measures, respectively, were across schools; however, variance components associated with these ICCs were non-significant. Across teachers, the correlation between engagement and adherence was not significant, r=.11 (p>.05). Zero-order correlations at the classroom-level (level 2) are presented in Table 2. Prediction of Classroom Implementation As shown in Table 3, significant positive associations with students’ engagement in the STR lessons were found for classroomaverage levels of student support, student attitudes against bullying, student climate and school connectedness, which each accounted for between 2 % and 7 % of the unexplained variance. For lesson adherence, student attitudes against bullying exhibited a significant positive association, explaining approximately 2 % of unexplained variance. Among school-level predictors, the perceived school-level bullying problems and percentage of students receiving FRPL were significantly related to lower levels of student lesson engagement. Results of the final conditional model with all significant zero-order associations with program implementation outcomes indicated that student climate and the percentage of students receiving FRPL remained as predictors in the final conditional model predicting student lesson engagement (see Table 4). Cumulatively, these predictors accounted for 20 % of variance in student engagement. No final model was run for lesson adherence, given that only one predictor was found to be significant (i.e., student attitudes against bullying). Outcomes of Classroom Implementation As shown in Table 5, higher levels of student lesson engagement were significantly related to a variety of outcomes, including higher levels of student support, student attitudes against bullying (marginally significant), student attitudes toward bullying intervention, student climate, school connectedness (marginal), and staff climate (marginal), and lower levels of bullying perpetration (marginal) and bullying victimization. Among these outcomes, engagement accounted for 4 % (student attitudes against bullying) to 32 % (student support) of classroom- and school-level variation. Teacher lesson adherence was not associated significantly with any of the posttest student outcomes.
Discussion Results Unconditional ICCs (i.e., without predictor variables) in HLMs for the two program implementation measures
This study examined the predictors and outcomes associated with implementation of a universal school-based bullying prevention program. A previous investigation of STR found positive impact on student attitudes, bullying-related
Prev Sci (2014) 15:165–176 Table 3 Regression coefficients (and standard errors) for prediction of program implementation
– .92 .07
Lesson adher
172
– .11 2.34 .40 – .76*** .27* .08 3.00 .29 – .69*** .58*** .40** .00 2.60 .22
– .22 .08 3.17 .22
Classroom Level (Student Report) Student Attitudes Against Bullying Student Attitudes Toward Bullying Int Student Bullying Intervention Teacher/Staff Bullying Int Positive Bystander Behavior Bullying Perpetration Bullying Victimization Student Support Student Climate School Connectedness Staff Climate School Level School Policies/Procedures (Staff Report) School Climate (Staff ) Staff Bullying Intervention (Staff) School-Level Bullying Problems (Staff) Percentage free or reduced lunch Percentage Hispanic Percentage White Number enrolled students
– −.48*** −.24 −.28* −.31* −.02 .01 2.16 .36 – .42** −.58*** −.33** −.50*** −.55*** −.15 −.19 1.31 .76 – .15 −.06 .08 −.06 −.20 −.16 −.13 .04 2.35 .26
– .60*** .66*** .62*** .24 .02 2.94 .23
School conn Bully victim Bully perp Bystnd beh
Studnt suppt
Studnt climte
Staff climte
Studnt engag
Predictor (pretest)
Zero-order predictions Engagement
Adherence
.24 (.09)* ns ns ns ns ns ns .47 (.23)* .72 (.22)**
.04 (.02)* ns ns ns ns ns ns ns ns
.45 (.19)* ns
ns ns
ns
ns
ns ns −.46 (.15)**
ns ns ns
−.01 (.00)** ns ns ns
ns ns ns ns
ns nonsignificant
– −.30* −.46*** −.38** .31* .06 .26* .22 .09 .25† 2.88 .38
*p
