Chapter 1

PROGRESS IN EDUCATION

PROGRESS IN EDUCATION VOLUME 35

No part of this digital document may be reproduced, stored in a retrieval system or transmitted in any form or by any means. The publisher has taken reasonable care in the preparation of this digital document, but makes no expressed or implied warranty of any kind and assumes no responsibility for any errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of information contained herein. This digital document is sold with the clear understanding that the publisher is not engaged in rendering legal, medical or any other professional services.

PROGRESS IN EDUCATION Additional books in this series can be found on Nova‘s website under the Series tab.

Additional e-books in this series can be found on Nova‘s website under the e-book tab.

PROGRESS IN EDUCATION

PROGRESS IN EDUCATION VOLUME 35

ROBERTA V. NATA EDITOR

New York

Copyright © 2015 by Nova Science Publishers, Inc. All rights reserved. No part of this book may be reproduced, stored in a retrieval system or transmitted in any form or by any means: electronic, electrostatic, magnetic, tape, mechanical photocopying, recording or otherwise without the written permission of the Publisher. We have partnered with Copyright Clearance Center to make it easy for you to obtain permissions to reuse content from this publication. Simply navigate to this publication‘s page on Nova‘s website and locate the ―Get Permission‖ button below the title description. This button is linked directly to the title‘s permission page on copyright.com. Alternatively, you can visit copyright.com and search by title, ISBN, or ISSN. For further questions about using the service on copyright.com, please contact: Copyright Clearance Center Phone: +1-(978) 750-8400 Fax: +1-(978) 750-4470 E-mail: [email protected]. NOTICE TO THE READER The Publisher has taken reasonable care in the preparation of this book, but makes no expressed or implied warranty of any kind and assumes no responsibility for any errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of information contained in this book. The Publisher shall not be liable for any special, consequential, or exemplary damages resulting, in whole or in part, from the readers‘ use of, or reliance upon, this material. Any parts of this book based on government reports are so indicated and copyright is claimed for those parts to the extent applicable to compilations of such works. Independent verification should be sought for any data, advice or recommendations contained in this book. In addition, no responsibility is assumed by the publisher for any injury and/or damage to persons or property arising from any methods, products, instructions, ideas or otherwise contained in this publication. This publication is designed to provide accurate and authoritative information with regard to the subject matter covered herein. It is sold with the clear understanding that the Publisher is not engaged in rendering legal or any other professional services. If legal or any other expert assistance is required, the services of a competent person should be sought. FROM A DECLARATION OF PARTICIPANTS JOINTLY ADOPTED BY A COMMITTEE OF THE AMERICAN BAR ASSOCIATION AND A COMMITTEE OF PUBLISHERS. Additional color graphics may be available in the e-book version of this book.

Library of Congress Cataloging-in-Publication Data ISSN: 1535-4806 ISBN: (eBook)

Published by Nova Science Publishers, Inc. † New York

CONTENTS Preface Chapter 1

Chapter 2

Chapter 3

Chapter 4

vii Assessing the Quality of Research on Youth Mentoring: Implications for Schools, Programs and Educational Research Susan P. Farruggia, Pat Bullen, Ann Dunphy, Frank Solomon, Efeso Collins and Victoria Burney Update on Response-to-Intervention in Preschool: Preliminary Findings for Response-to-Intervention in Emergent Literacy Skills, Social-Emotional Skills, and Challenging Behaviors Cathy L. Grist and Lori A. Caudle Types of Parent Involvement As Predictors of the Postsecondary Educational Plans and Future Educational Aspirations of 7th and 9th Grade Students Kristin Skells, Lee Shumow and Jennifer A. Schmidt Underestimation Bias in Monitoring Accuracy Using the Gamma Coefficient: A Monte Carlo Study Gregory Schraw, Fred Kuch and Robin Roberts

Chapter 5

Training Future Physicians in Legislative Advocacy Kimberly D. Northrip and L. Curtis Cary

Chapter 6

Assessment of the Teaching Quality of Teachers of Primary and Special Education África Borges and Manuela Rodríguez-Dorta

Chapter 7

Orientating Pedagogy towards Hybrid Learning Spaces Guglielmo Trentin

Chapter 8

Learning, Teaching and Assessing in Portugal: Are There Differences to Other European Countries? Patrícia Albergaria-Almeida, Betina da Silva Lopes and Mariana Martinho

1

21

39

57 71

83 105

125

vi Chapter 9 Index

Contents Corpus Informed Foreign Language Vocabulary Instruction İhsan Ünaldi

141 179

PREFACE This series presents substantial results from around the globe in selected areas of educational research. The field of education is consistently on the top of priority lists of every country in the world, yet few educators are aware of the progress elsewhere. Many techniques, programs and methods are directly applicable across borders. Topics discussed herein include the assessment of the quality of research on youth mentoring; an update on response-to-intervention in preschools; types of parent involvement as predictors of the postsecondary educational plans and future educational aspirations of 7th and 9th grade students; underestimation bias in monitoring accuracy using the gamma coefficient; training future physicians in legislative advocacy; assessment of the teaching quality of teachers of primary and special education; orientating pedagogy towards hybrid learning spaces; learning, teaching and assessing in Portugal; and corpus informed foreign language vocabulary instruction. Chapter 1 - There has been an increasing awareness at the policy level, nationally and internationally, of the importance of evidence-based programing. In the United States (US), evidence-based registries, where programs are rated on their effectiveness based on the quality of the evidence is common place. Although such registries are not common in New Zealand, there is increasing discourse there and elsewhere on what constitutes quality evidence as programs are being funded on their ability to demonstrate effectiveness. To gain an understanding of the nature and quality of the evidence relating to program effectiveness, a systematic review was conducted assessing the effectiveness of youth mentoring programs in New Zealand, many of which were school-based and had educational goals (96%). The review included quantitative, qualitative and mixed-methods studies with varying research design, including correlational, quasi-experimental and experimental. Overall, 88% of the programs included in this review showed some level of effectiveness. There was, however, wide variation in quality of the research raising concerns about the accuracy of some of the research. The present study assessed the quality of the research of studies included in that systematic review to determine if the differences in the quality of the research were systematic or random. Specifically, the research had three aims: The first was to assess the overall quality of the research on youth mentoring included in the review. The second was to examine associations between research characteristics and qualities with program effectiveness. Finally, the third was to develop evidence-informed recommendations to programs and schools which utilize youth mentoring programs to support their students. The results suggest that much of the

viii

Roberta V. Nata

research had low levels of quality, thus reducing the confidence in the findings. Further, there was an association between quality and effectiveness such that studies using weaker designs showed greater effects. Finally, when looking at research method, qualitative studies were more likely to report program effects as compared to quantitative designs. Evaluation research is meant to assist programs in improving their effectiveness, yet this cannot be achieved if the research is questionable. Implications of this research for programs and schools, and how these findings inform the implementation of program registries are discussed. Chapter 2 - There is considerable research on RTI in the school age years and while application of RTI is increasing in the preschool years, there still seems to be issues with widespread implementation. The progression of RTI from the school-age years to RTI in the preschool years is reasonable considering the evidence of early intervention. RTI was first introduced and developed to provide identification and intervention to children who are at risk or have a learning disability (IDEA 2004). Researchers have demonstrated the effectiveness of Response-to-Intervention (RTI) in most models of the intervention while ensuring quality implementation. (VanDerHeyden, n.d.) There are typically two goals of RTI with the first goal being to provide evidence-based interventions and the second goal is to progress monitor the intervention based on the student‘s response to the intervention (East, 2006). The following chapter will discuss the methods used for implementing RTI in preschool in general as well as in a particular preschool project. Implementation of evidencebased interventions with regards to social-emotional skills, challenging behaviors, and preliteracy skills in preschool will be discussed. Chapter 3 - This study examines the impact of different types of parent involvement on the postsecondary educational plans and future educational aspirations of 298 students in seventh or ninth grade. Parent involvement in school, parent involvement at home, and mothers‘ and fathers‘ expectations for their children were measured to assess how parent involvement predicts postsecondary educational plans (level of education the student plans to pursue immediately after high school) and future educational aspirations (goal for total years of education). Analyses tested whether these relationships differ by student gender. Results of OLS regressions indicated that, while mothers‘ academic expectations did not predict postsecondary educational plans or future educational aspirations for either grade level, fathers‘ academic expectations showed different patterns of association for seventh vs. ninth graders. For example, fathers‘ academic expectations significantly predicted both students‘ postsecondary plans and future educational aspirations for ninth graders, but was only predictive of future educational aspirations among seventh graders. Parent involvement in school also predicted future educational aspirations for ninth graders but not seventh graders. Parent involvement in homework was not predictive of postsecondary educational plans or outcomes at any grade level. Further analyses suggest that fathers‘ academic expectations are particularly predictive for female students. Implications are discussed for promoting subtle, yet impactful forms of parent involvement. Chapter 4 - Researchers studying metacognitive monitoring commonly use the gamma coefficient to estimate monitoring accuracy. Gamma provides a measure of non-parametric association between a performance variable and a corresponding judgment of performance over a set of items for individual examinees. However, gamma may be inappropriate when a test is short (e.g., 20 or fewer items) due to undefined scores and empty cells, which cause underestimation bias. Three Monte Carlo Simulations examined the effect of test length and

Preface

ix

difficulty on distributional parameters and bias. Simulation 1 found that tests with 20 or fewer items significantly underestimated expected population values of gamma and contained a large proportion of empty cells in the 2 X 2 data array. These problems increased as the test became easier. Simulation 2 found that using a replacement strategy to seed an empty cell with one observation led to significant overestimation of gamma on short tests. Simulation 3 showed that bias remained even when using a 2 x n rather than 2 x 2 data array. Overall, tests with 20 or fewer items produced significantly biased underestimates of gamma; whereas longer, more difficult tests based on larger data arrays reduced underestimation because they eliminated empty cells or reduced restricted range in scores. Implications for research and impact on the collective literature were discussed. Chapter 5 - Many medical professional organizations emphasize the importance of physician involvement in the legislative process. These organizations encourage their members to advocate for policies that promote population health and to help shape legislation related to the practice of medicine. Some of the Residency Review Committees (RRC) of the Accreditation Council for Graduate Medical Education (ACGME) have also begun to encourage training in advocacy. Pediatric Milestone SB7 and Family Medicine Milestone SB3, used by the respective RRCs, evaluate a trainee‘s ability to advocate for community health. Several medical schools and residency programs have developed advocacy curricula that include legislative advocacy. These curricula have received positive responses from medical trainees and increased subjective comfort with the process. Professional organizations such as the American Medical Association, the American College of Physicians, and the American Academy of Pediatrics have both Legislative Advocacy opportunities for trainees and recommendations for effective advocacy. Legislative advocacy curricula can be enhanced through legislative visits, educational collaboration with other training programs, and partnerships with professional organizations. Chapter 6 - Providing a quality education is a challenge for today's society, because teaching is not enough and it is also requires demonstrating excellence and rigor. Therefore, it is essential to study and deepen the process of teaching and learning that takes place in the classroom. Within this context, the teacher plays a fundamental role, as they are responsible for teaching and getting your students to learn. This is why the evaluation of teaching in an aspect of great interest to guarantee quality education. When the evaluation process focuses on professional performance in the classroom, the methodology of choice is observational. It is useful and flexible for information in natural contexts. This methodology requires a rigorous, valid and reliable observational instrument. This chapter tests an observation instrument aimed at evaluating the performance of teachers in primary and special education in the classroom (Observational Protocol in the Teaching Functions in Primary School and Special Education, PROFUNDO-EPE). The instrument's reliability is checked using Cohen's Kappa index and the Generalizability Theory (GT) following a uni-facet design OxC, where the generalized facet is observers (O) as well as homogeneity, also through TG using the same design as for the calculation of inter-observer reliability, but in this case the generalized facet is the code (C). The instrument is tested by observing four teachers, two of them teach Primary Education, and two of them teach Special Education. One of the Special Education teachers works in Aula Enclave (Self-Contained Unit) and the other one in Therapeutic Education

x

Roberta V. Nata

(PT) with students of Preschool and Primary Education. These teachers are recorded during their professional performance in the classroom. Seven trained observers participated in the coding of the recordings obtained by the video cameras. Since the greatest strength of observational methodology does not reside in the study of independent behaviors, but behavioral patterns by determining which conduct follows another with a higher probability than chance, the analysis was performed with sequential lag analysis. The results show that both the inter-observer reliability and the homogeneity of the instrument are suitable. On the other hand, the sequential analysis shows the behavioral patterns developed by each of the teachers, describing their performance in the classroom and interaction with their students. This paper shows that the observational instrument evaluates the behavior of teachers in their professional performance in the classroom, while providing recommendations for continuous improvement. This way, the strengths and weaknesses of teachers can be operationalized, highlighting how useful observational methodology is in assessing teachers‘ performance in the classroom and helping to maintain the quality of teaching. Chapter 7 - The huge spread of network and mobile technology offers new dimensions and spaces for interpersonal interaction. The present-day ―always-on‖ condition erases any clear distinction between physical and digital spaces, introducing a new, so-called ―hybrid‖, conception of space. Hybrid spaces are dynamic and characterised by constant connectedness, whereby remote contexts are integrated with the space/time dimensions of the here and now. The aim of this contribution is to illustrate how these spaces have gained increasing importance in pedagogy, and to examine the risks of an over-simplistic, reductive interpretation of the Bring Your Own Device (BYOD) approach. Thus there will be a discussion of the pedagogical, teaching and instructional design aspects of an educational process which is destined to develop more and more in hybrid learning spaces, and where the real and virtual blend together, losing their separate connotations. Examples from university experiences will be presented to illustrate the close interdependence of these aspects. Chapter 8 - The OECD Teaching and Learning International Survey (TALIS) is the largest international survey of teachers. This survey looks at the working conditions of teachers and at the features that influence effective teaching, such as teachers‘ initial training and their professional development, the feedback they receive on their teaching, the climate in the classrooms and schools, teachers‘ satisfaction with their job, as well as the teaching, learning and assessment strategies they use in the classroom. TALIS was conducted for the first time in 2008 in 24 participating countries and economic entities and surveyed lower secondary teachers (ISCED level 2, as classified by the International Standard Classification of Education in 1997). In 2013, the second cycle of TALIS was implemented in 34 countries from America, Asia, Europe and Oceania. In this chapter the authors propose to (i) analyze the TALIS 2013 data regarding the teaching and the assessment strategies used in the classroom by low secondary (ISCED level 2) Portuguese teachers; (ii) compare the teaching strategies adopted by Portuguese teachers with those used by the other European teachers; (iii) compare the use of the assessment strategies made by low secondary Portuguese

Preface

xi

teachers‘ with those used by other European teachers; (iv) characterize Portuguese teachers‘ teaching and learning beliefs, (v) compare the teaching and learning beliefs held by Portuguese teachers with those held by the other European teachers. Chapter 9 - This experimental study explores the effectiveness of corpus-based language learning activities to help the learners overcome lexicon related problems. In the quasiexperimental phase, 37 intermediate and upper-intermediate level Turkish EFL learners participated in the study. The experimental group was composed of 18 learners, and there were 19 learners in the control group. Before the treatment, both groups were given vocabulary recognition and essay writing tests. The experimental treatment included corpusbased concordancing activities. This treatment lasted for about 10 weeks, and at the end, both groups were given the same vocabulary recognition and writing test again. In order to test the delayed effects of the treatment, the same written test was given to both groups after about two weeks. Analysis results revealed that the experimental group obtained significantly better scores compared to the control group in terms of vocabulary recognition and production. As the last stage of the study, the participants were interviewed about the use of concordancing activities in language instruction. The results indicated that the majority of the participants improved positive attitudes towards these activities and found them beneficial.

In: Progress in Education. Volume 35 Editor: Roberta V. Nata

ISBN: 978-1-63482-503-0 © 2015 Nova Science Publishers, Inc.

Chapter 1

ASSESSING THE QUALITY OF RESEARCH ON YOUTH MENTORING: IMPLICATIONS FOR SCHOOLS, PROGRAMS AND EDUCATIONAL RESEARCH Susan P. Farruggia1,, Pat Bullen2, Ann Dunphy3, Frank Solomon3, Efeso Collins3 and Victoria Burney2 1

University of Illinois at Chicago, Chicago, IL, US The University of Auckland, Auckland, New Zealand 3 The Youth Mentoring Network

2

ABSTRACT There has been an increasing awareness at the policy level, nationally and internationally, of the importance of evidence-based programing. In the United States (US), evidence-based registries, where programs are rated on their effectiveness based on the quality of the evidence is common place. Although such registries are not common in New Zealand, there is increasing discourse there and elsewhere on what constitutes quality evidence as programs are being funded on their ability to demonstrate effectiveness. To gain an understanding of the nature and quality of the evidence relating to program effectiveness, a systematic review (Farruggia et al., 2011) was conducted assessing the effectiveness of youth mentoring programs in New Zealand, many of which were school-based and had educational goals (96%). The review included quantitative, qualitative and mixed-methods studies with varying research design, including correlational, quasi-experimental and experimental. Overall, 88% of the programs included in this review showed some level of effectiveness. There was, however, wide variation in quality of the research raising concerns about the accuracy of some of the research. The present study assessed the quality of the research of studies included in that systematic review to determine if the differences in the quality of the research were systematic or random. 

Corresponding author: Dr. Susan P. Farruggia, University of Illinois at Chicago, UH2704, M/C103, Chicago, IL 60607. Telephone: 312-996-8115, e-mail: [email protected].

2

Susan P. Farruggia, Pat Bullen, Ann Dunphy et al. Specifically, the research had three aims: The first was to assess the overall quality of the research on youth mentoring included in the review. The second was to examine associations between research characteristics and qualities with program effectiveness. Finally, the third was to develop evidence-informed recommendations to programs and schools which utilize youth mentoring programs to support their students. The results suggest that much of the research had low levels of quality, thus reducing the confidence in the findings. Further, there was an association between quality and effectiveness such that studies using weaker designs showed greater effects. Finally, when looking at research method, qualitative studies were more likely to report program effects as compared to quantitative designs. Evaluation research is meant to assist programs in improving their effectiveness, yet this cannot be achieved if the research is questionable. Implications of this research for programs and schools, and how these findings inform the implementation of program registries are discussed.

Keywords: Youth mentoring, evaluation, research bias, systematic review

INTRODUCTION Like elsewhere around the world, youth mentoring has flourished in New Zealand in the past two decades as a means of providing structured social support for young people, particularly those at risk of not achieving their potential in academic domains (Brooker, Ellis, Parkhill, and Bates, 2010; Farruggia, Bullen, Davidson, Dunphy, Solomon, and Collins, 2011a), as well as those who could benefit from increased feelings of well-being within school settings (Swain-Campbell and Quinlan, 2009). A recent systematic review (Farruggia, et al., 2011a) on youth mentoring in New Zealand found that a high proportion of the programs demonstrated some level of effectiveness (88%). In addition, the New Zealand programs have a strong emphasis on education with the majority of included studies had educational goals (96%). Further, school was the most common location for programs. Results of both New Zealand and international research demonstrate that youth mentoring is effective in supporting the achievement and academic motivation of young people (e.g., DuBois, Portillo, Rhodes, Silverthorne, and Valentine, 2011), although programs may be more effective for psychological and social goals as compared to educational goals (Farruggia, et al., 2011a). Researchers have conducted studies to look at the impact of mentoring on at-risk populations and involvement in youth mentoring programs has been found to be associated with less school absenteeism, more positive attitudes toward school, greater well-being, less likelihood to start using illegal drugs and alcohol, less engagement in aggressive behavior, and decreases in skipping school (for a review see DuBois et al., 2011; Tolan, Henry, Schoeny, and Bass, 2008). In examining the New Zealand context, the recent systematic review found that youth mentoring bears a number of similarities to international mentoring literature, such as most New Zealand mentoring programs utilize a one-to-one mentoring relationship (Farruggia, et al., 2011a). However, some differences were identified in terms of the location of programs, with a majority of New Zealand mentoring programs delivered within schools and more likely being independent, localized programs, rather than community-based and large-scale nationwide mentoring organizations established overseas, such as Big Brothers/Big Sisters in the United States.

Research on Youth Mentoring

3

Unlike the above mentioned US-based research, the systematic review included both quantitative and qualitative methodologies. Importantly, including both types of studies provided the opportunity to see if the effectiveness varied as a function of methodology. While findings from qualitative studies are limited in their ability to determine causality, there are certain criteria that can be used to determine the quality of qualitative research (e.g., Flick, 2008), such as utilizing rigorous and systematic analysis techniques. Like quantitative studies, poor quality can impact the validity of qualitative findings. Due to limited research on youth mentoring in New Zealand, all evaluation studies were included in the systematic review regardless of the quality. However, the quality of the research was assessed to highlight potential threats to the validity of the findings (Farruggia, et al., 2011a). This is particularly noteworthy given the increased awareness at the policy level, nationally and internationally, of the importance of quality evidence and evidencebased programing (Gluckman, 2011; Burkhardt, Schroter, Magura, Means, and Coryn, 2015). It is important to note that a key purpose of program evaluation is to help programs increase effectiveness. Yet, this cannot be achieved if the quality of the evaluation research is low and the validity of the findings is questionable. While the assumption is that lower quality research may artificially inflate the effectiveness of the programs, some studies have found greater effects for research with more rigorous designs (e.g., Anderson et al., 2003, study on the influence of media violence). For both of these reasons, it is important to examine the quality of the research and determine if there is, indeed, an association with reported effectiveness.

Aims of the Study The current study has three aims. The first is to assess the quality of the research on youth mentoring in New Zealand. The second is to examine associations between research quality and program effectiveness, and the third is to develop evidence-informed recommendations to programs and schools which utilize youth mentoring programs to support their students.

METHODS Criteria for Considering Studies for This Review Only studies that met all the inclusion criteria were included in the review (see Farruggia, et al., 2011a for a full description of methodology). Adapted from Littell, Corcoran and Pillai (2008) and Tolan et al. (2008), criteria included: 1) Program effectiveness for the mentees was examined and address outcomes in at least one of the following areas: emotional/ psychological, problem/high risk behavior, academic/educational, career/employment, and social competence. 2) Participants were NZ youth over the age of 6 years and under the age of 24 years, with the mean age for the study not being over 19 years, reflecting the Ministry of Youth Development‘s definition of young people (MYD, 2003). 3) Studies evaluated a formal mentoring program, including one-to-one, group, team, peer or e-mentoring, but not natural

4

Susan P. Farruggia, Pat Bullen, Ann Dunphy et al.

mentoring. It should be noted that due to the limited research on youth mentoring in New Zealand, studies with less rigorous methodologies were included, but bias was identified. In addition, both qualitative and quantitative studies were included, as long as they met the following criteria. For qualitative studies to be included, they needed at least an indicator of effectiveness reflecting change; post-test only was acceptable if change was measured. For quantitative studies to be included, there needed at least to be one indicator of effectiveness including an indication of change or difference (e.g., pre-test/post-test change or the use of a comparison group).

Search Strategy Search Strategy for Identifying Relevant Studies The search strategy for relevant literature was conducted in four ways. Firstly, a contact at the Youth Mentoring Network (YMN), a charitable trust that supports mentoring organizations that work with young people, approached all youth mentoring organizations that were part of the Network to request copies of any evaluation reports on their particular program. It is believed that all of the active youth mentoring at the time of the search were under the umbrella of the YMN. Secondly, an extensive database search was conducted, including: education databases (i.e., ERIC, A Plus Education, Education Sage, Professional Development Collection, and Proquest Education Journals); psychological and medical databases (i.e., PsycInfo, MEDLINE, Psychological and Behaviour Sciences Collection, Web of Science, and Science Direct); social science databases (i.e., FAMILY, Proquest Social Science Journals, Social Services Abstracts, and SAGE Sociology); New Zealand databases (i.e., Index New Zealand and TePuna); and other databases (i.e., Proquest Dissertations and Theses, all New Zealand university theses and dissertation databases, and the Cochrane Library). The list of search terms developed with the assistance of a subject librarian included: mentor*, role model, youth, young*, child*, teen*, adolescent*, juvenile, program*, evaluate*, and intervention*. ‗Zealand‘ was added as a term to all searches. Thirdly, an internet search was conducted which covered national research sites, Ministerial websites, Google and Google Scholar. Forth and lastly, reference lists of retained studies were checked for further research that had not been identified by the above methods.

Data Collection and Analysis Methods Selection of Studies A total of 13,292 studies (unduplicated citations) were identified during the search. Of the studies identified, two were unobtainable and a further two on-going studies were not included in this review. A total of 74 were deemed to be relevant to the review based on the citation and abstract. All relevant, full-text reports were coded using an inclusion code sheet adapted from Littell et al. (2008) and Tolan et al. (2008). This inclusion coding was done by two independent coders and the inter-rater agreement was 83%.


5

When there was disagreement, the first author of this chapter met with the coders and came to a consensus. A total of 26 studies met the inclusion criteria for the review and are included in this study. The list of included studies is found in Table 1. Excluded studies (48) did not meet criteria for the following non-mutually exclusive reasons: 36 did not test for program effects, 12 were not a formal mentoring program, 7 had mentees outside the age range, 4 had the same data presented in another included study, 2 examined mentor, but not mentee, effects, and 1 was not based in New Zealand. Table 1. Research characteristics for included studies Study Adams (2004) Afeaki-Mafile'o (2007) Ave, et al., (1999) Ballinger, et al., (2009)

Design Qualitative

Publication Masters thesis

Evaluator(s) Measures External Interviews

Qualitative

Book chapter

Internal

Single case study

Qualitative

Technical report

External

Mixed methods Technical report

External

Deane andHarre (2008)

Quantitative

Technical report

External

Enkey (2001)

Quantitative

Technical report

External

Postgraduate Diploma project

Internal

Multiple case study Questionnaire (mentees), interview Questionnaire (teachers, parents, mentees, mentors, control students) Questionnaire (teachers, parents, students, control students), school records, physical fitness tests Questionnaire (teacher), interview, focus group Questionnaire (mentees), interview Questionnaire (students), online survey (mentors), school records, mentee session attendance, interview, focus group Questionnaire (mentees), exam results

Hammond (2007) Mixed methods Heke (2005)

Mixed methods Masters Thesis

Internal

Hill (2008)


External

Irving, et al. (2003) Quantitative

Journal article

External

Kostuk-Warren (2005)

Mixed methods

Doctoral dissertation

External

Lennan (2006)

Qualitative

Technical report

External

Litchfield (2006)


Internal

Lyon (1992)


Internal

Mclean (2007) Qualitative McInerny (2005) Qualitative Milne et al. (2002) Quantitative

Masters Thesis Masters Thesis Journal article

External Internal External

Questionnaire, interview Multiple case study, interview Questionnaire (mentees), interview Questionnaire (mentees), interview Focus group Interview Questionnaire (mentees)

6

Susan P. Farruggia, Pat Bullen, Ann Dunphy et al. Table 1. (Continued)

Study Design Ministry of Quantitative Education (2009) QiaoandMcNaught Quantitative (2007)

Publication

Evaluator(s) Measures

Technical report

External

Technical report

External

Selwood (2005)

Quantitative

Masters thesis

External

Starpath (2006)


External

Starpath (2007)


External

Stevenson, L. (2008)

Qualitative

Masters thesis

External

Tasi, B. S. (2009) Qualitative

Masters Thesis

External

Wilson, S. (2006) Quantitative

Journal article

Internal

Youth at Risk of Offending Team (2001)


External

School records Questionnaire (mentees, control students) Questionnaire (teachers, parents, mentees) Questionnaire (teachers, parents, mentees, mentors), interview, multiple case study Questionnaire (mentees, mentors), interview, multiple case studies Observation, interview, document analysis Multiple case study, interview Questionnaire (teacher, mentees) Questionnaire (parents, mentees), multiple case study

Data Extraction and Management A data extraction coding sheet was developed for the purposes of extracting relevant information for the review from the included studies using established documents (i.e., DuBois, Holloway, Valentine, and Cooper, 2002; Littell, et al., 2008; Tolan et al., 2008) and adapted to fit the New Zealand context. The data extraction coding sheet covered aspects of report/research characteristics and methodologies, program features, mentee characteristics, outcome goals and measures (i.e., educational, psychological) adverse effects, timing of intervention, and quantitative and qualitative outcomes. For a number of studies, there was limited information on program characteristics and qualities. In cases where the study was evaluating a current program, the Youth Mentoring Network contacted the individual program to request missing information. Alternatively, program websites were searched to identify missing information. All of the information was coded by two independent researchers. Goals were coded into six categories, including educational, psychological, behavioral, interpersonal, vocational, and cultural goals. Programs could include more than one goal category and were coded on each goal where 1 = having that goal or 0 = not having that goal. Farruggia et al. (2011a) provides a detailed description of the coding process. Below provides details related to the research-related key variables included in this study. The interrater agreement was 80%; again, when there was disagreement, the first author met with the two coders and came to consensus.


7

Assessment of Research Characteristics of Included Studies Studies were coded on a number of research characteristics which included design type, publication type, the nature of evaluator‘s involvement in the research, and measures used to assess outcomes. For design type, studies were categorized as qualitative, quantitative or mixed-methods. Publication type coded where studies were published, including journal articles, technical reports or studies completed as part of a thesis or university project. The evaluator code looked at whether the researcher(s) independent of (external to) or formed part of (internal to) the intervention project. Studies were coded on the measures used for data analysis. For qualitative studies, these were coded as either 1) a single case study, 2) a multiple case study, with data from at least two participants, 3) interview design, or 4) a focus group design, or any combination of the above design types. For quantitative studies, measures included 1) participant self-report 2) third party report, such as data from participant‘s parents and teachers, 3) use of archival data or records, including school reports, 4) observation, including direct observations made by researchers, and 5) the use of standardized assessments, such as questionnaires or achievement tests. Across all studies, the timing of assessment was recorded by coding whether assessment had occurred: 1) pre-test, before the initiation of the program, 2) mid-intervention, 3) immediately after the intervention had finished, and at various lengths of follow up: short term (within 2-8 weeks of program completion), moderate (between 2-6 months after intervention had ended), and long term follow up (6-12 months after the program or more than 12 months post program). Assessment of Methodological Quality of Included Studies Quality coding involved assessing whether a study met standards for thorough and systematic research and, therefore, provided the ability to gauge the quality of the research, and, consequently, the validity of the findings being presented. Each quality variable was coded as met or unmet and a total percentage was then calculated for each study. Quality coding was conducted for each study on 16 items relating to methods, intervention, participation, and analysis. All studies were assessed for dissemination quality, specifically whether the study was subject to peer review. The peer-review standard was set high and thus included publications in peer-reviewed journals and conferences, but did not include theses or reports. Studies were also coded in aspects of study design including the use of a control group, or a pre-test/posttest design for data collection. Studies were coded for program qualities that would influence the effectiveness of the program. This quality category included items such as program participation – all eligible participants were offered a place in the program and at least 80% of those eligible enrolled in the program; no performance bias (intervention group)– intervention group did not receive any other services beyond mentoring; no performance bias (control group) – control group did not have any negative experiences/services while intervention group was being mentored; attrition – program dropout rate was 20% or less and equal with comparison group, if applicable. Researcher quality was coded by assessing conflict of interest where researchers or data collectors would not benefit from favorable results of program, and detection bias – where the assessor was unaware of the assigned treatment when collecting outcome measures. Due to the varied nature of the research procedures, detection bias was coded separately for quantitative and qualitative studies.

8


Research participant quality indicators were rated separately for qualitative and quantitative study methodologies due to varying procedures. Participation indicated that least 80% of those eligible and selected to participate in the research did participate. Selection meant that inclusion of participants in research was randomly allocated, or all were included (e.g., not targeted). Intention to treat was met when those who started the program were possible participants (e.g., included those who dropped out of the program as potential participants). Both qualitative and quantitative studies were assessed for quality data analysis. Qualitative studies were coded as having: a clearly stated and followed research paradigm, researcher acknowledgement of the social context, researcher acknowledgement of reflexivity, an adequate description of data analysis techniques, data analysis techniques that were thorough and systematic, and data analysis techniques guided by a clear, theoretical framework. Quantitative studies were coded for: standardized observation periods where follow-up data were collected from each case at a fixed point in time, use of measures with demonstrated reliability and validity, and use of inferential statistical tests to assess effectiveness.

Measures of Treatment Effect Treatment effects were measured in a number of ways as both quantitative and qualitative data were used in this review. Further, many of the studies that utilized quantitative data did not conduct statistical tests to determine effect and/or significance. As such, a variety of approaches were taken to determine the treatment effects. Individual effect indicators were determined for each outcome reported. See Farruggia et al. (2011a) for a complete description of the calculation of treatment effects. For quantitative studies that provided statistical results, an effect size (Cohen‘s d) for each measure was calculated. Effect sizes of below .20 were seen as signifying unsuccessful outcomes; those with effect sizes between .20 and .35 were seen as indicating moderately successful outcomes, and effect sizes above .35 indicated successful outcomes. Once effect sizes were calculated, results were recoded for each goal domain (i.e., educational, psychological, behavioral, interpersonal, vocational, and cultural) as not effective, mixed or moderately effective, or effective. Mixed effects reflected multiple indicators within the same goal domain, but with inconsistent results. For qualitative studies, outcomes were coded in the data extraction code sheet for success, using the response choices: not effective, mixed results, and effective. To be coded as effective, all or most of the qualitative results needed to have indicated a positive effect. To be coded as mixed, some of the results needed to effective. To be coded as not effective, none or very few of the results were effective. Once individual outcomes were assessed for effectiveness, these data were aggregated by domain. At the end of these processes, both quantitative and qualitative results were on the same scale. Quantitative and qualitative data were merged and programs were coded for overall effectiveness in the following categories: not effective (not effective in any domain or using either methodology; very few effects found), mixed/moderately effective (effects found in some domains or had moderate effects across domains), effective (effective in many domains, possibly some minor variation by research methodology), or very effective (consistent, strong effects across domains and methodology).


9

RESULTS The results are divided into three sections. The first section is a description of the research on youth mentoring programs in New Zealand included in this review. The second section is an analysis of the quality of these studies. The third and final section is an examination of the associations between study design and quality with program effectiveness.

Description of the Research Methodology Overall, 31% (n = 8) of studies were purely quantitative studies, 31% (n = 8) were purely qualitative studies, and 38% (n = 10) were mixed-methods studies. Most of the studies were a thesis or project to earn an education qualification (42%, n = 11) or a technical report (38%, n = 10). Studies were typically conducted by an external evaluator (73%, n = 19) and no studies utilized an action research design. In terms of measures, of the 18 studies with a quantitative approach, 17 used questionnaires whereas only three used school records and one used program records (not mutually-exclusive). For the 18 studies with a qualitative approach, interview was the most common technique (n = 13), followed by case study (n = 7), focus group (n = 3), observation (n = 1), and document analysis (n = 1). Regarding the features that reflect quality of the research, few studies utilized a control group (31%, n = 8). Those that did were either quantitative-only studies (n=5) or mixed methods studies (n=3). Similarly, studies did not typically utilize a pre-test, post-test design (31%, n = 8); those that did were equally split between quantitative-only (n=4) and mixed methods (n=4) studies. Combining these features, only one study used an experimental design (a quantitative-only study). Seven studies used a quasi-experimental with a control group design (n = 4 for quantitative-only; n = 3 for mixed methods), four studies used a quasi-experimental without a control group design (n = 1 for quantitative-only; n = 3 for mixed methods), and 14 studies used a correlational design (n = 2 quantitative-only; n = 4 for mixed methods; n = 8 for qualitative-only).

Study Quality Quality was assessed for all studies. However, it is important to note that some studies lacked adequate description; in these instances, a quality assessment could not be made. As seen in Table 2, some quality features were difficult to assess such as performance bias (77% missing), program participation (54% missing), attrition (38% missing) and participation (23% missing). Of the studies included in this review, only one was classified as high quality (more than 90% quality indicators present), 31% were classified as having moderate quality (more than 50% quality indicators present), and 65% were classified as having low quality (50% or less quality indicators present). There were a number of particularly low areas of research quality seen across the majority of studies in this review. Indicators of research quality below are found in Table 2.

10

Susan P. Farruggia, Pat Bullen, Ann Dunphy et al. Table 2. Percentage of included studies that met quality indicators

Quality Indicator Dissemination Quality Peer reviewed (n=26) Study Design Quality Used control group (n=26) Used pre-test methods (n=26) Research Program Quality No program participation bias (n=12) No performance bias (intervention group) (n=26) No performance bias (control group) (n=6) No attrition bias (n=16) Researcher Quality No conflict of interest (n=26) Total no detection bias (n=21) No detection bias – Qualitative (n=17) No detection bias – Quantitative (n=13) Research Participant Quality No participation bias (n=20) Total no selection bias (n=24) No selection bias – Qualitative (n=14) No selection bias – Quantitative (n=18) Total no intention to treat bias (n=23) No intention to treat bias – Qualitative (n=18) No intention to treat bias – Quantitative (n=15) Qualitative procedures and analysis Clearly stated and followed research paradigm (n=16) Acknowledged social context (n=10) Acknowledged reflexivity (n=15) Adequate description of data analysis (n=18) Thorough and systematic data analysis techniques (n=18) Data analysis techniques guided by a clear, theoretical framework (n=18) Quantitative procedures and analysis Used standardized observation periods (n=17) Used measures with demonstrated reliability and validity (n=17) Used statistical tests (n=18)

Percentage 12 31 31 75 42 100 50 73 29 65 23 65 21 64 17 30 44 27 56 90 33 39 50 44 82 76 61

Dissemination Quality Only 12% of the included studies were peer reviewed. All of those were quantitative-only studies; no mixed-methods or qualitative-only studies were peer-reviewed. Program Quality One area of relatively high consistency of quality was for program participation. Most programs included in the review (75%) offered a place in the mentoring program to all eligible candidates, avoiding bias associated with targeting particular candidates for participation.


11

Across all 26 studies reviewed, 42% demonstrated no performance bias, meaning that the intervention group did not receive any other services beyond mentoring. For the majority of studies that did show evidence of performance bias, understanding the effectiveness of mentoring is likely to be more difficult, as it is hard to know whether outcomes are the result of mentoring, or due to the other services that were received by the intervention group. Performance bias was not found for any study which used a control group design (n=6); no study indicated that control participants were subject to any negative experiences or services throughout the duration of the study. Half of the studies showed no attrition bias where more than 20% of the participants dropped out of the program. This also can be viewed as an indicator of program success, as it is likely that the majority of participants found the program helpful, and consequently remained engaged in the mentoring program. High levels of attrition in the other 50% of reviewed studies may reflect a poor fit between participants and the mentoring program. This could be problematic as those who left may not have found the program helpful or engaging. In addition, the effects reported in these studies may be artificially high.

Researcher Quality The majority of studies in this review (73%) reported no conflicts of interest, a quality which strengthens conclusions based on the findings of these studies. When considering detection bias, 65% of qualitative studies did not show any evidence of selection based on favorable results (of the mentoring program). However, this quality was not as apparent in quantitative studies, where only 23% of studies ensured the assessor was not aware of the assigned treatment group of participants when collecting data. High levels of detection bias for the other reviewed quantitative studies could be seen as a problem as there is a risk of the assessor unconsciously eliciting certain responses from the participants, particularly if the assessor has an investment in the program, as was the case for a number of studies in this review. Research Participant Quality Most of studies in this review (65%) were found to have avoided participation bias, with at least 80% of eligible and selected individuals opting to take part in research assessing the mentoring programs. This quality allows for more reliable and robust findings to be made from the studies, as evaluation results are more likely to be representative of the target population under consideration. No selection bias was typical for qualitative studies (64%), meaning they randomly selected or selected all possible participants. However, only 17% of quantitative studies were found to have no selection bias, with the majority of quantitative studies biased by not randomly assigning participants to treatment or control groups, or not matching the treatment and control group in the analysis. This is important as random assignment or matched-control accounts for a large amount of variance between groups based on group attributes; in cases where this is not met, differences between groups could be due to differing group attributes, rather than the effect of the intervention. For both quantitative and qualitative studies, there were very low levels of quality regarding intention to treat, with only 27% of quantitative studies and 44% of qualitative studies including everyone who started the program as potential participants in the research. In total, only approximately one third of studies in this review successfully negated intention to treat bias in all components of their evaluation. For the majority of studies that did not

12


demonstrate this research quality, the implication of not involving those who had left or dropped out of the program in the research means that results may be inflated, as views and outcomes for those still in the program and those who had left may be different, and possibly less effective.

Qualitative Procedures and Analysis Half (50%) of the studies clearly stated the research paradigm used in the collection and interpretation of data. Most studies (67%) included statements indicating an understanding of the social context of the research participants. However, only five studies (27%) acknowledged reflexivity, an awareness of the effect the researcher may have on the construction and interpretation of results. While most studies (67%) provided an adequate description of the methods and procedures used regarding data collection, less than half included information on the data analysis process (39%). Further, only 39% of the studies showed evidence that their data analysis techniques were thorough and systematic. In terms of the interpretation and presentation of results, for most of the studies included in this review, an adequate proportion of the data were taken into account (65%), the interpretation of results followed logically from the analysis (83%), and the findings appeared to match derived data (78%). Quantitative Procedures and Analysis Of the quality measures specific to quantitative research approaches, the use of standardized measurement periods was the feature incorporated by most of the studies (82%). Most quantitative studies in this review also used measures with demonstrated reliability and validity (76%), and more than half (61%) utilized statistical tests in reporting their findings. Ratings of quality in this review were highest for these features of quantitative studies. However, while all of these quality indicators add to the confidence that can be placed in the findings of some studies with quantitative components, not all studies reviewed achieved these high levels of quality, indicating that a level of bias still remains in research of this type.

Associations between Study Design and Quality with Study Effectiveness This final section presents the associations between study design and quality with effectiveness. First, study characteristics are examined for variation in bias and effectiveness. Then, bias and effectiveness are examined to see if they are associated. The first set of analyses in this section looks at associations between study characteristics including methodology, publication type (technical report versus thesis/project for qualification), and evaluator with effectiveness. For methodology, qualitative studies had a greater proportion of effective programs (75%) as compared to mixed methods (50%) or quantitative studies (38%). Of mixed-methods studies only, allowing for differing goal domains to be assessed independently, qualitative results showed more effective outcomes 60% of the time, quantitative results showed more effective outcomes for 7% of the time, and quantitative and qualitative methods reflected similar effectiveness 33% of the time. For evaluators, there appeared to be no differences between internal and external evaluators for overall effectiveness and quantitative effects; however, there was a difference found for qualitative studies where external evaluators (83%) were more likely to find


13

programs effective as compared to internal evaluators (50%). Next, associations between study qualities for all studies (control group, use of a pre-test, and experimental design), quantitative studies (statistical test), and qualitative (stated research paradigm, social context, reflexivity, data collection description, data collection analysis description, adequate proportion of data, systematic data analysis, and findings match data) with effectiveness were assessed. It should be noted that only when there were enough studies included in the assessment was a determination made (e.g., peer-review was not included in this section as only 3 studies had been through the peer-review process). Greater effectiveness was found for studies that did not utilize a control group (overall effectiveness: 61% versus 38%; no difference apparent for quantitative only), did not have a pre-test (overall effectiveness: 61% versus 38%; no difference apparent for quantitative only; qualitative: 79% versus 50%), and, relatedly, had a less rigorous experimental designs (overall effectiveness: correlational 86% versus quasi-experimental/experimental 33%). For quantitative studies only, studies that did not utilize a statistical test also appeared to show more effects than those that did (quantitative: 50% versus 27%). Associations between qualitative research qualities were examined in association with qualitative data effectiveness. Studies showed greater effects when there was not adequate description of the data collection (adequate 58% versus not adequate 100%) and when the findings did not match the derived data (did not match 50% versus matched 79%). No differences were apparent for stated research paradigm, reflexivity, description of data analysis, and using an adequate proportion of the data. Finally, associations between study effectiveness and bias were assessed. To examine this, the overall effectiveness, as well as the effectiveness broken down by quantitative and qualitative studies, was checked for variation in the amount of bias. There appeared to be no difference in the amount of bias as a function of effectiveness overall, as well as for quantitative and qualitative studies. Studies of more effective programs were just as likely to have high levels of bias as studies of less effective programs.

DISCUSSION The current study had three aims. The first was to assess the quality of the research on youth mentoring. The second was to examine associations between research quality and program effectiveness, and the third was to develop evidence-informed recommendations for programs and schools which utilize youth mentoring programs to support their students. Included studies only evaluate 35% of current mentoring programs at the time of this review. Those programs that had undergone and continue to undergo the evaluation process should be commended as it is a principle of good practice (DuBois, 2014) and was associated with effectiveness (Farruggia et al., 2011a).

The Quality of Research on Youth Mentoring and Program Effectiveness Overall, there was great variability in the quality of the research on youth mentoring. In general, a fair amount of the research was high in bias and did not meet an appropriate standard of evaluation. For example, the peer-review process is considered an important

14


indicator of research quality, as it allows for any potential flaws in study methodology or the analysis and interpretation of results to be identified and resolved, with the aim of ensuring disseminated research is of the highest standard (Danick, 1991; Littell et al., 2008). Yet, only a small proportion of studies were peer-reviewed, none of which were qualitative studies. While the peer-review process does help to ensure research quality, it is not necessary for programs and may be out of the scope of what is feasible. However, beyond this higher standard, many of the evaluations still fail to address clear areas of bias that can cast uncertainty about the accuracy of the results. One example of where this was seen was the large proportion of studies with high intent to treat bias (70%), indicating that most evaluations did not include participants who did not complete the program, potentially inflating effectiveness. As noted by Grossman (2005), this form of bias can threaten validity, the extent to which the conclusions made are accurate or ‗truthful‘. Given that the outcomes of program evaluations are often tied to funding, ensuring effects can accurately be attributed to program participation is vital. In addition, if programs are incorrectly informed that their effectiveness is higher than it really is, they will not make required changes to improve the quality of their programs. Another important area of consideration of the research is that very few studies (13%) included assessment of effectiveness in the long-term (i.e., more than 6 months postintervention), with most (52%) measuring effectiveness in the short-term (i.e., up to two months post-intervention). This makes it difficult to know if there were any lasting effects of the program. One interesting finding was that the effectiveness of programs varied as a function of methodology with qualitative studies being associated with greater effectiveness. This difference has two potential explanations. First, it could be said that qualitative approaches are more sensitive to effects, particularly for studies with small sample sizes. However, it could also mean that the qualitative methods used were more biased and reported effects may not have been accurate. Indeed, there may be some support for this explanation as qualitative studies where the interpretation did not match the findings were more likely to report effectiveness, meaning that findings were overstated. However, given that external evaluators were more likely to find programs effective as compared to internal evaluators, it could be argued that there should be less bias in reported effects. One important final point related to this difference is that very few qualitative studies acknowledged reflexivity. While reflexivity was not associated with effectiveness, the critical scrutiny associated with this practice can be viewed as a mechanism to increase research quality (Elliott, Fischer, and Rennie, 1999). Future research should clarify associations between methodology effectiveness. The final point of discussion regarding research on NZ youth mentoring is that this review focused on the effectiveness of youth mentoring, not the cost-effectiveness of youth mentoring. Cost effectiveness analysis reflects how much a unit of achieved outcome costs so that comparisons can be made about the efficiency of programs. Research has indicated that youth mentoring programs, and in particular school-based programs, can be a cost effective means of promoting positive development (Herrera, Sipe, and McClanahan, 2000; Yates, 2005) compared with other interventions. While this is a very important assessment, it is beyond the scope of this review. However, adequate assessment of cost-effectiveness can only be achieved if programs‘ effects are valid and reliable; highlighting the importance of quality research in this sector. In addition, while these programs may be cost-effective, there


15

is still a cost involved. Encouraging quality evaluations helps ensure money in this sector is spent wisely.

Implications for Schools and Programs Given that a large proportion of youth mentoring programs in New Zealand have educational goals and many are school-based (Farruggia et al., 2011a), youth mentoring is an important tool for promoting positive growth within the education sector. Indeed, previous research has demonstrated that mentoring, in general, can facilitate positive change in education-related outcomes such as more positive attitudes toward school and decreases in skipping school (Herrera, Grossman, Kauh, and McMaken, 2011; Karcher, 2008; Tolan et al., 2008), and encourage key educational competencies (Noonan, Bullenand Farruggia, 2012). To further validate effectiveness within the New Zealand context, it is vital that youth mentoring programs are supported and encouraged to implement quality evaluations. Although school-based mentoring has the potential to impact a variety of school-related outcomes, to facilitate effectiveness, programs need to be well designed and implemented over longer periods of time for optimal impact (Herrera and Karcher, 2014; Wood and MayoWilson, 2012).

Implications for Program Registries This research contributes importantly to the current dialogue regarding evidence-based programming and program registries (see for example Schröter and Coryn, 2015). More specifically, there is a need for clear and consistent guidelines on what constitutes quality evidence – regardless of methodology. This is particularly pertinent given that practice and funding decisions are based on research outcomes. The current review highlights at least two key points worth noting. First, the variability in quality and high level of bias suggest that strategies are needed to help communicate to the sector what constitutes quality evidence. Second, although previous reviews have not included both quantitative and qualitative methodologies, we wanted to demonstrate that both methods contribute importantly to our understanding of the effectiveness of youth mentoring. We acknowledge that both methodologies have limitations; however the focus should be on issues that impact research quality (e.g., rigor and cultural appropriateness; Farruggia, Bullen, Solomon, Collins, Dunphy, 2011b) rather than methodology.

Recommendations for Researchers Researchers need to eliminate the high levels of bias found in most of the research currently conducted on youth mentoring in New Zealand. Highlighted recommendations include:

16

Susan P. Farruggia, Pat Bullen, Ann Dunphy et al.     

using control groups, preferably with random assignment or at least matched comparison; including youth who leave the program in the evaluation; conducting pre-tests; conducting post-tests with adequate time between program close and the assessment (e.g., 6 months or more); and submitting the results of their evaluations to peer reviewed outlets.

Further, research reports need to provide adequate detail on their framework, methods, analyses and results to accurately communicate the findings of the study and to demonstrate the quality of the research conducted, as well as the program delivery so that the quality of the research can be properly assessed.

Recommendations for Schools and Programs    

evaluate programs on a consistent basis; use external evaluators who have experience in evaluation research, perhaps by partnering with university researchers; use tools such as program logic models or theory of change to identify the expected program outcomes and associated measurement tools; and establish an evaluation schedule that is sensitive to measuring change overtime.

As noted by the Chief Science Officer to the Prime Minister, Sir Peter Gluckman (2011), evidence-based practice contributes to effectiveness and therefore should be viewed as an integral part of all programs.

ACKNOWLEDGMENTS The review was conducted with support from the Ministry of Youth Development and the Health Research Council of New Zealand Partnership Programme. We would like to thank Joy Davidson for her hard work searching databases and extracting data, and Bev CassidyMackenzie of the Youth Mentoring Network for her assistance in contacting programs.

REFERENCES References marked with an asterisk indicate studies included in the systematic review. *Adams, R. J. (2004). Constructing meaning from mentoring: The experiences of mentors and mentees. Unpublished master‘s thesis, University of Canterbury, Christchurch, New Zealand. *Afeaki-Mafile'o, E. (2007). Affirming Works: A collective model of Pacifika mentoring. In: P. Culbertson, M. Nelson Agee and C. 'Ofamakasiale (Eds.), Peninauliuli: Contemporary


17

challenges in mental health for Pacific peoples (pp. 16-25). Honolulu, HI: University of Hawai'i Press. *Ave, K., Evans, I., Hamerton, H., Melville, L., Moeke-Pickering, T., and Robertson, N. (1999). Mentoring for children/youth at risk demonstration project: Final evaluation report. Wellington, New Zealand: The Crime Prevention Unit, Department of the Prime Minister and Cabinet. Anderson, C. A., Berkowitz, L., Donnerstein, E., Huesmann, L. R., Johnson, J. D., Linz, D., Malamuth, N. M., and Wartella, E. (2003). The influence of media violence on youth. Psychological Science in the Public Interest, 4, 81-110. DOI:10.1111/j.1529-1006.2003. *Ballinger, B., Mason, N. and Waring, M. (2009). YWCA Future Leaders Evaluation: A report for the YWCA Auckland. Auckland, New Zealand: AUT University, Consulting and Training. Burkhardt, J. T., Schroter, D. C., Magura, S., Means, S. N., and Coryn, C. L. (2015). An overview of evidence-based program registries (EBPRs) for behavioral health. Evaluation and Program Planning, 48, 92-99. DOI: 10.1016/j.evalprogplan.2014.09. 006. Brooker, B., Ellis, G., Parkhill, F., and Bates, J. (2010). Māori achievement in literacy and numeracy in a sample of Canterbury schools. New Zealand Journal of Educational Studies, 45, 49-65. Danick, B. P. (1991): Importance of peer review. The Serials Librarian, 19, 91-94. DOI:10. 1300/J123v19n03_11. *Deane, K. and N. Harre (2008). Executive summary of results for Project K quantitative evaluation measures from pre- to post- programme. Auckland, New Zealand: The University of Auckland, Department of Psychology. DuBois, D. L. (2014). Programme evaluation. In: D. L. Dubois and M. J. Karcher (Eds.), Handbook of Youth Mentoring (2nd ed., pp. 481-498). Thousand Oaks CA: Sage. DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorne, N., and Valentine, J. C. (2011). How effective arementoring programmes for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12, 57-91. DuBois, D. L., Holloway, B. E., Valentine, J. C., and Cooper, H. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157-197. DOI: 10.1023/A:1014628810714. Elliott, R., Fischer, C. T. and Rennie, D. L. (1999). Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, 38, 215-229.DOI: 10.1348/014466599162782. *Enkey, R. F. (2001). A Research and Programme Evaluation of the North Shore Education Trust Project K. Auckland, New Zealand: Project K. Farruggia, S. P., Bullen, P., Davidson, J., Dunphy, A., Solomon, F., and Collins, E. (2011a). The effectiveness of youth mentoring programmes in New Zealand. New Zealand Journal of Psychology, 40, 52-70. DOI: New Zealand Journal of Psychology, 40, 52-70. Farruggia, S. P., Bullen, P., Solomon, F., Collins, E., and Dunphy, A., (2011b). Examining the cultural context of youth mentoring: A systematic review. Journal of Primary Prevention, 32, 237-251. DOI: 10.1007/s10935-011-0258-4. Flick, U. (2008). Managing Quality in Qualitative Research. London: Sage.

18


Gluckman, P. (2011). Improving the transition: reducing social and psychological morbidity during adolescence / a report from the Prime Minister's Chief Science Advisor. Auckland, New Zealand: Office of the Prime Minister's Science Advisory Committee. Grossman, J. B. (2005). Evaluating mentoring programs. In: D. L. DuBois and M. J. Karcher (Eds.), Handbook of Youth Mentoring (pp. 251-265). Thousand Oaks CA: Sage. *Hammond, J. (2007). Relationships matter: An evaluation of the Student Engagement Initiative at a girls‟ secondary school. Unpublished research project, University of Canterbury, Christchurch, New Zealand. *Heke, J. I. (2005). Hokowhitu: A sport-based programme to improve academic, career, and drug and alcohol awareness in adolescent Māori. Unpublished doctoral thesis, University of Otago, Dunedin, New Zealand. Herrera, C., Grossman, J. B., Kauh, T., and McMaken, J. (2011). Mentoring in schools: An impact study of Big Brothers Big Sisters school-based mentoring. Child Development, 82, 346-361. DOI: 10.1111/j.1467-8624.2010.01559.x. Herrera, C. and Karcher, M. (2014). School-based mentoring. In: D. L. Dubois and M. J. Karcher (Eds.), Handbook of Youth Mentoring (2nd ed., pp. 203-220). Thousand Oaks CA: Sage. Herrera, C., Sipe, C. L. and McClanahan, W. S. (2000). Mentoring school-aged children: Relationship development in community-based and school-based programs. Philadelphia: Public/Private Ventures. *Hill, J. (2008). ‗I Have A Dream‟ A report for the IHAD Foundation: An evaluation of the programme and outcomes for students. Auckland, New Zealand: The Education Group Ltd. *Irving, E., Moore, D. W. and Hamilton, J. (2003). Mentoring for high ability high school students. Education and Training, 45, 100-109. Karcher, M. J. (2005). The effects of developmental mentoring and high school mentors‘ attendance on their younger mentees‘ self-esteem, social skills, and connectedness. Psychology in the Schools, 42, 65-77. DOI: 10.1002/pits.20025. Karcher, M. J. (2008). The study of mentoring in the learning environment (SMILE): A randomized evaluation of the effectiveness of school-based mentoring. Prevention Science, 14, 212-227. DOI: 10.1007/s11121-008-0083-z. *Kostuk-Warren, J. (2005). Adolescent well-being: Effects of time and intervention. Psychology. Unpublished doctoral thesis, The University of Auckland, Auckland, New Zealand. *Lennan, M. (2006). Evaluation report of the Big Buddy Mentoring Trust, Auckland, New Zealand. *Litchfield, J. E. (2006). Boys talk about mentoring and "stuff”: A small-scale evaluation of Y9-Y13 mentoring relationships in a boys' high school. Unpublished master‘s thesis, The University of Auckland, Auckland, New Zealand. Littell, J. H., Corcoran, J. and Pillai, V. (2008). Systematic Reviews and Meta-Analysis. New York, NY: Oxford University Press. *Lyon, D. R. (1992). The adjustment problems of Cambodian secondary school students: An exploratory survey and an innovative buddy system intervention. Unpublished master‘s thesis, The University of Auckland, Auckland, New Zealand. *McClean, D. (2007). Evaluating the early impact of the Rangatahi Māori Mentoring Program developed to encourage and support Māori secondary school students to pursue


19

a career in health. Unpublished master‘s thesis, Griffith University, Queensland, Australia. *McInerny, J. (2005). The Buddy Programme at Presbyterian Support Southland Child and Family Services: Five years on… Unpublished master‘s thesis, University of Otago, Dunedin, New Zealand. *Milne, B. J., Chalmers, S., Waldie, K. E., Darling, H., and Poulton, R. (2002). Effectiveness of a Community-based Truancy Intervention: A pilot study. New Zealand Journal of Educational Studies, 37, 191-203. *Ministry of Education (2009). Review of targeted policies and programmes: MapihiPounamu and He AraTika Maori participation initiatives. Wellington, New Zealand: Author. Ministry of Youth Development. (2003). 12 to 24: Young people in New Zealand. Wellington, New Zealand. Noonan, K., Bullen, P. and Farruggia, S. P. (2012). School-based mentoring: Examining the cultural and economic variations in engagement and effectiveness. New Zealand Journal of Educational Studies, 47, 47-61. *Qiao, C. and McNaught, H. (2007). Evaluation of Project K. Wellington, New Zealand: Ministry of Social Development, Centre for Social Research and Evaluation Te Pokapū Rangahau Arotaki Hapori. Schröter, D. C. and Coryn, C. (2015). Deconstructing evidence-based practice: Progress and ambiguities. Evaluation and Program Planning, 48, 90-91. DOI: 10.1016/j.evalprogplan. 2014.10.001. *Selwood, J. (2005). Perspectives on a deaf mentoring programme: Does it make a difference? Unpublished research project, Christchurch College of Education Christchurch, New Zealand. *Starpath (2006). MATES 2005 Evaluation. Starpath technical Report 15.1. Auckland, New Zealand: Author. *Starpath (2007). MATES 2006 Evaluation. Starpath Technical Report 15.2. Auckland, New Zealand: Author. *Stevenson, L. (2008). Personalised learning in a web 2.0 environment. Unpublished master‘s thesis, University of Waikato, Hamilton, New Zealand. Swain-Campbell, N. and Quinlan, D. (2009). Children‘s school, classroom, social well-being and health in New Zealand mid to low decile primary schools. New Zealand Journal of Education Studies, 44, 79-92. *Tasi, B. S. (2009). Supporting youth for work in New Zealand: A case study of the Samoan experience. Unpublished master‘s thesis, The University of Auckland, Auckland, New Zealand. Tolan P., Henry, D., Schoeny, M., and Bass, A. (2008). Mentoring interventions toaffect juvenile delinquency and associated problems. Chicago: The University of Illinois at Chicago, Institute for Juvenile Research. *Wilson, S. (2006). Improving retention and success: A case study approach for practical results. Journal of College Student Retention: Research, Theory and Practice, 7, 245261. Wood, S. and Mayo-Wilson, E. (2012). School-based mentoring for adolescents: A systematic review and meta-analysis. Research on Social Work Practice, 22, 257-269. DOI: 10.1177/1049731511430836.

20


Yates, B. T. (2005). Cost-benefit and cost-effectiveness analyses. In: D. L. DuBois and M. J. Karcher (Eds.), Handbook of Youth Mentoring (pp. 525-545). Thousand Oaks CA: Sage. *Youth at Risk of Offending Team. (2001). Evaluation of the one to one mentoring programme, Nelson. Nelson, New Zealand: Author.



Chapter 2

UPDATE ON RESPONSE-TO-INTERVENTION IN PRESCHOOL: PRELIMINARY FINDINGS FOR RESPONSE-TO-INTERVENTION IN EMERGENT LITERACY SKILLS, SOCIAL-EMOTIONAL SKILLS, AND CHALLENGING BEHAVIORS Cathy L. Grist, Ph.D., and Lori A. Caudle, Ph.D. Western Carolina University, Cullowhee, NC, US

ABSTRACT There is considerable research on RTI in the school age years and while application of RTI is increasing in the preschool years, there still seems to be issues with widespread implementation. The progression of RTI from the school-age years to RTI in the preschool years is reasonable considering the evidence of early intervention. RTI was first introduced and developed to provide identification and intervention to children who are at risk or have a learning disability (IDEA 2004). Researchers have demonstrated the effectiveness of Response-to-Intervention (RTI) in most models of the intervention while ensuring quality implementation. (VanDerHeyden, n.d.) There are typically two goals of RTI with the first goal being to provide evidence-based interventions and the second goal is to progress monitor the intervention based on the student‘s response to the intervention (East, 2006). The following chapter will discuss the methods used for implementing RTI in preschool in general as well as in a particular preschool project. Implementation of evidence-based interventions with regards to social-emotional skills, challenging behaviors, and pre-literacy skills in preschool will be discussed.

INTRODUCTION Children across the United States are attending early childhood programs in increasing numbers (National Center for Education Statistics, 2013). Some children come to school prepared and ready to learn and others do not.

22

Cathy L. Grist and Lori A. Caudle

The rate of children eligible for special education services in inclusive preschool programs, such as Head Start and state preschool programs, is on the rise. More than 1 million infants, toddlers, and preschoolers receive special education services in the United States and this number has increased 45% over the past 20 years (CEC, 2014). The need for interventions for our youngest citizens is ever more present, and research has shown that intervening early for children with disabilities can improve the outcomes in kindergarten and as children progress through their education (Karoly, Kilburn, & Cannon, 2005). Services are provided by the Individuals Disabilities Education Improvement Act of 2004 (IDEIA) in the form of early intervention services and special education instruction to young children. IDEIA provides intervening early with services for children who may require academic and/or behavioral interventions. Intervening early based on the provision in IDEIA includes providing academic or behavioral interventions to children who are not identified as need special education services. However, these students do need additional support in the general education classroom in order to progress and be successful (DEC/NAEYC/NHSA, 2013). Services classified as intervening early are referred to as response-to-intervention (RTI).

RESPONSE-TO-INTERVENTION RTI is a multi-tiered framework in which teachers provide interventions in increasing intensity and assess the progress of those interventions. RTI is a systematic data-driven process in which the ongoing assessment of a student‘s skills determines movement between the tiers, based on progress made (Ball & Trammell, 2011; Bayaht, Mindes, & Covitt, 2010). RTI in the elementary years has a fair amount of support due the evidence-based research published on implementation, while there is little research on implementation in the preschool years. Although, interest in RTI in preschool education is growing (Ball & Trammell, 2011; Bayaht, Mindes, & Covitt, 2010). Currently, RTI is not standardized across grades K-12, as it comes in a variety of forms. Furthermore, in the field of early childhood, professionals‘ understanding of how the process should work has been inconsistent. Few professionals in the field of early childhood have reported that RTI has been implemented (National Professional Development Center on Inclusion, 2012). Then again, preschool teachers are familiar with, and often implement universal screening, differentiation for all children in the classroom, and progress monitoring. It is the systematic use of these procedures and adjusting and adapting intervention based on data collected during progress monitoring that is the hallmark of RTI. Implementation of systematic procedures in RTI faces barriers in the pre-kindergarten classroom. Barriers in the classroom include understanding the process, providing professional development on the RTI process, technical adequacy of universal screening instruments, systematic progress monitoring, and evidence based interventions. The framework of RTI is well-matched with the goals of early childhood education despite these barriers.

Update on Response-to-Intervention in Preschool

23

RTI in Preschool Key components of RTI in the preschool years include a tiered framework, universal screening, assessment, progress monitoring, evidence-based standard protocols, collaborative problem solving, family involvement, and fidelity of implementation. Three tiers are included in the RTI framework. Tier 1 includes a quality early childhood program in which evidencebased curriculum is offered as well as intentional teaching, universal design and progress monitoring. Universal screening is used to establish which children may benefit from more intensive intervention in Tier 2. The universal screening also serves as a baseline in order to determine progress. Tier 2 includes more intensive intervention in which the response involves embedded learning opportunities, progress monitoring, standard protocols, and collaborative problem solving. Intervention can be provided to small and large groups of children and progress is monitored frequently to determine if adjustments are needed to help the child progress. Collaborative problem solving facilitates any adjustments needed in intervention, and parents and family members are involved in this process. Tier 3 includes additional intervention that is individualized and more intensive than Tier 2, as well as progress monitoring, and collaborative problem solving. Children who need further support, or who did not respond to the interventions in Tier 2, are provided with one-on-one interventions that are individualized and intensive. As in Tier 2, progress monitoring and collaborative problem solving are used to adjust and adapt interventions for the child (Coleman, Roth, & West, 2009). Universal screening of all children provides information about a child‘s strengths and needs and to identify if the child‘s development is on track. Universal screening facilitates early identification of young children and decision making about any child that may need additional supports and services (Coleman, Roth, & West, 2009). Progress monitoring should be systematic and standard, thus, making any changes needed to help a child progress and reach goals. Progress monitoring provides information necessary to modify interventions and identify if other supports or services are needed (Coleman, Roth, & West, 2009). Evidence-based practices and standard protocols are essential features in the Pre-K RTI process. Evidence-based practices have been implemented in RTI in grades K-12 as general practice. In the field of early childhood, there are already several practices that make RTI a natural next step, such as the use of evidence-based curriculum and instructional methods, documentation of progress, service delivery models, and collaboration of professionals in the field. Standard protocols include researched interventions that help children progress in the areas of academics and social-emotional skills (Coleman, Roth & West, 2009). Fuchs (2003) identifies two common RTI models utilized in early childhood education: problem-solving and standard protocol. The problem-solving approach includes problem identification, problem analysis, exploring solutions, and evaluating solutions (Hagans-Murillo, 2005). The standard protocol model includes small group interventions for a designated amount of time where children are then identified as being responders or non-responders (Hagans-Murillo, 2005, p. 49). There is an increasing number of standard protocols that have been recommended for the preschool years, such as The Incredible Years Teacher Classroom Management Program, which provides training to teachers on how to manage and intervene with children exhibiting challenging behaviors (Webster-Stratton, 1994). Pre-K RTI implementation allows for natural linkages between universal screening, progress monitoring and evidence-based practices by

24


providing professionals with consistent information about a child‘s progress in which modifications and adjustments can be made to intervention (Coleman, Roth, & West, 2009). Collaborative problem-solving is another important component of RTI in pre-k. Collaboration among key important individuals in a child‘s life such as parent, teacher, and other service providers working with the child promotes comprehensive planning. Each team member contributes to the process and provides unique information. Parents and caregivers are valuable partners in the RTI process. Collaborative-problem solving teams make decisions based on data collected on universal screenings and progress monitoring. This type of decision making facilitates individualization for children who need on-going intervention (Coleman, Roth, & West, 2009). Engagement of family members is crucial to the success of RTI in pre-k (Coleman, Roth, & West, 2009). Parents and caregivers should be encouraged to participate in their child‘s education, even before the child steps through the classroom door. This can be accomplished in a variety of ways, such as teachers and other professionals seeking to understand the history of the child‘s life, and accepting the values and beliefs of the child‘s family. Inviting parents and guardians to share about their child and contribute their points of view on their child‘s development, strengths, and needs is essential in the RTI process. There will be a review of general practices in RTI in preschool in the areas of assessment and intervention with regard to literacy and social-emotional skills and challenging behaviors. Application of RTI for literacy and social-emotional skills and challenging behaviors in a specific preschool program will be discussed, and procedures analyzed. Lessons learned from this RTI project will be shared. Finally, recommendations for best practices in RTI in pre-k will be offered in literacy and social-emotional assessment, instruction, and progress monitoring.

Literacy The National Early Literacy Panel‘s Report on Developing Early Literacy (2008) identifies six early literacy skills highly predictive of future literacy learning. These skills include: (1) alphabet knowledge, (2) phonological awareness, (3) rapid automatic naming of letters, (4) rapid automatic naming of objects or colors (5) writing or writing name, and (6) phonological memory. This report also describes other skills to be moderately correlated with later literacy achievement, which include concepts of print, print knowledge, reading readiness, oral language, and visual processing. These skills should be taught in preschool classrooms through a blended literacy approach. In this approach, teachers consider the importance of emergent literacy activities and environmental arrangements as well as teacherinitiated, targeted literacy experiences (Vukelich, Christie, & Enz, 2012). Many current versions of curricula design literacy learning around this combined approach. Children at-risk for literacy failure are often unsuccessful in programs that focus primarily on an emergent literacy approach (Vukelich et al., 2012). These children typically require more targeted instruction based on acquiring specific skills (Bailet, Repper, Piasta, & Murphy, 2009), such as phonological awareness, alphabet knowledge, and concepts of print (Vukelich et al.). Targeted instruction should be child-centered, yet explicitly embedded into daily activities (Hagans-Murillo, 2005). Unfortunately, many preschool programs, particularly those serving children identified as at-risk, do not embrace a blended literacy


25

approach, mainly due to a lack of teacher training, focus of the curricula, and adequacy of assessments (Hagans-Murillo, 2005). Through a sociocultural perspective, literacy is a social event and how text and words are used depends on the social situation, setting, and activity (Scull, Nolan, & Raban, 2013). Even through targeted literacy activities, teachers should remain authentic and consider contextual factors, children‘s prior experiences, cultures, home languages, and backgrounds. Literacy Assessments. Teachers should be intentional about how they plan and implement literacy activities, but systematic monitoring of literacy knowledge and growth is imperative for teachers to know how to adjust teaching strategies and intervention tasks (PeisnerFeinberg & Buysee, 2013). Assessments are also extremely important in providing baseline data before beginning RTI interventions. While considering an RTI framework, progress monitoring should be used to identify which children need continued assessment and attention, quantify data of targeted children, and develop individualized programs for children identified as needing additional supports (Fuchs & Fuchs, 2007). Combining assessments allows teachers to identify strengths, weaknesses, and learning across time within selected skills that are highly predictive of future literacy acquisition. Peisner-Feinberg and Buysee recommend preschool literacy assessments should focus on literacy skills that are strong predictors of later reading achievement, such as alphabet knowledge, phonological awareness, and vocabulary. The Phonological Awareness Literacy Screening for Preschool (PALS-PreK Assessment, 2015) is a tool that measures children‘s knowledge of: name writing, alphabet knowledge, beginning sound awareness, print and word awareness, rhyme awareness, and nursery rhyme awareness. It is based on research that identifies how knowledge in these literacy areas is necessary for later literacy success. PALS-PreK can be administered over a two-week period and only takes about 20-25 minutes per child. This tool can be used for initial assessments and progress monitoring. Test of Preschool Early Literacy (TOPEL) is an assessment frequently used with preschoolers to both (1) identify their print knowledge, oral vocabulary, and phonological awareness, and (2) monitor children‘s progress toward acquiring literacy skills within these three categories. This tool takes about 20-25 minutes to administer (Lonigan et al., 2007). Unlike the PALS-PreK, this assessment is typically conducted in one sitting. The Revised Get Ready to Read! (Whitehurst & Lonigan, 2001) measures print knowledge and phonological awareness. This assessment only takes about ten minutes to complete, but has been shown to be more useful with children from middle and upper-middle socioeconomic backgrounds, or older preschoolers (Wilson & Lonigan, 2009). The Preschool Early Literacy Indicator (PELI), created by the Dynamic Measurement Group, and authors of the Dynamic Indicators of Basic Early Literacy Skills (DIBELS), was just released to researchers in the 2014-2015 academic year. This is a preschool benchmark assessment that measures alphabet knowledge, vocabulary and oral language, phonemic awareness, and listening comprehension. Portions of this assessment can also be used for progress monitoring. Many preschool assessments that are more comprehensive, such as mCLASS: CIRCLE (Amplify Education, 2015), are used to identify literacy needs even though they also assess other content areas. When considering assessments, teachers and other professionals should be careful about making conclusions based soley on data about easy-to-test literacy skills. Using data from a variety of assessment techniques across time provides a more complete

26


representation of a child‘s literacy knowledge, and particularly those skills that are more difficult to identify within one assessment. Interventions. Implementing RTI in preschool is particularly advantageous because it provides opportunities to deliver intensive literacy and language interventions early in a child‘s life, before significant reading and writing problems emerge in elementary school (Curenton, Justice, Zucker, & McGinty, 2013). When considering high-quality literacy instruction, experiences should include (1) vocabulary-rich conversations, (2) phonological awareness, print awareness, and listening comprehension games and activities, and (3) environmental adaptations that support literacy development in context (Curenton et al., 2013; Vukelich et al., 2012). Classroom environments should be print-rich, yet meaningful to children. Labeling of objects should emerge through authentic discussions with children across the school year, and relate to specific learning experiences. Alphabet books, games, and activities centered on children‘s names increase children‘s alphabet knowledge in ways that are particularly inviting to young children (Bloodgood, 1999). Any of these types of literacy activities and strategies can be utilized across all three tiers of RTI. The process of using assessment information to identify what skills targeted children need, and the ways they learn best, informs teachers‘ decision-making within the RTI process. A literacy strategy or activity can be well-designed, yet poorly executed if it is not appropriate for a child at a specific point in time. Carta and Greenwood (2013) suggest Tier 2 instructional supports that are most effective in early childhood utilize a standard protocol and interventions that are systematic, explicit, and intense. Tier 2 literacy instruction should be directly aligned with missing or limited skills identified in baseline and progress monitoring assessment data. Small group instruction is a highly beneficial context in which teachers can teach and model targeted literacy skills, particularly through games and other hands-on activities. Further, this grouping provides ongoing opportunities for teacher-child conversations that enhance oral language development. Children receiving Tier 2 literacy interventions should be provided daily opportunities to practice targeted literacy skills in authentic ways during open-ended activities, such as center time. Intervention grouping should be flexible and fluid, with children moving between groups as they make progress or need additional supports. Tier 3 literacy interventions are highly individualized and often occur in structured, one-on-one formats with a teacher, or paraprofessional, and child. Effectiveness and Usefulness of RTI in Literacy Acquisition. VanDerHeyden, Snyder, Broussard, and Ramsdell (2007) studied the effectiveness of an RTI intervention on targeted literacy skills of children in both Head Start and public preschool classrooms. In this study, assessment data were collected through the Brigance Preschool Screen and curriculum-based assessment probes. Across five weeks, 20 rhyming interventions were implemented four times a week in a class-wide format and 20 alliteration interventions were conducted on an individual basis. Intervention strategies across both classrooms were highly structured and systematic. Progress monitoring assessments included the Dynamic Indicators of Basic Early Literacy Skills (DIEBELS) Initial Sounds Fluency sub-test and Letter Naming Fluency subtest, along with alliteration, rhyming and letter sound curriculum-based measurement probes. Results found progress monitoring and early literacy interventions appeared to relate to increased targeted literacy skills for children identified as low-performers in literacy. Buysee, Peisner-Feinberg, and Burchinal (2012) evaluated the effectiveness of the Recognition and Response (R&R) model in improving language and literacy outcomes for


27

preschoolers. R&R is a preschool RTI model that involves (1) gathering assessment data from all children, with follow-up progress monitoring of those children needing targeted language, literacy, or math interventions, and (2) providing a high-quality core curriculum with intentional teaching strategies and interventions based on formative assessment data (Buysee et al.). Twenty-six pre-k classrooms with a total of 366 children were included in the study, with a majority of children being from low-income families. Targeted children showed greater growth than the comparison group in measures of receptive language (PPVT-4), vocabulary (mCLASS:CIRCLE), and expressive language (EVT-2).

Social-Emotional Skills and Challenging Behaviors Children are entering the pre-k classroom with more issues in emotional-regulation and emotional stress, which often results in challenging behaviors. Challenging behaviors in young children are increasing every year. Gilliam (2005) found that preschoolers were expelled from their schools at rate three times as high as students in K-12. Therefore, students are missing key learning opportunities in the early years. Challenging behaviors in the classroom are disruptive for all involved including the child, teachers, and other children. It is important to intervene in the early years to facilitate positive school experiences for all children. Systematic assessment and intervention in social-emotional skills can provide teachers with a structure and framework that facilitates early identification of children who are at-risk for academic failures, as well as address challenging behaviors and other emotional issues in young children. Assessment of social emotional skills has become more systematic with young children, as professionals have more tools available to facilitate identification of issues in social-emotional functioning (Feil & Frey, 2013). However, tools for progress monitoring are less reliable for young children in this area. Measures used for early identification of children in preschool for challenging behaviors and other issues with social emotional skills will be discussed. Additionally, progress monitoring procedures will be reviewed. Assessment and Progress Monitoring. There are several standardized instruments in current use that assist in identifying children with challenging behaviors and social emotional issues. These behavioral instruments typically have Likert scales in which individuals closest to the child answer questions about social-emotional skills and behavior. These types of tools have several advantages because they are objective and reliable, provide a way to assess young children who are unable to answer questions about their own behaviors, and reflect the judgments of those who are experts about a child, such as parents and teachers (Whitcomb & Merrell, 2013). One of the most broadly used rating scales is The Child Behavior Checklist, which measures child and youth psychopathy. In particular, this instrument can be used to assess young children‘s (ages 1 ½ to 5 years) patterns of behavioral problems as well as determine symptomology related to the DSM-V Diagnosis categories and is part of the Achenbach System of Empirically-Based Assessment (ASEBA) (Achenbach, 1991). The Child Behavior Checklist can be completed by caregivers and teachers. Both versions of the checklist include 100 items in which those who know the child best can complete the form by responding to a Likert scale for each question with 0 = ―Not True,‖ ―1= Somewhat or Often True,‖ ―2= Always or Very Often True.‖ The instrument consists of 7 problem scales on the caregiver

28


version and 6 problem scales on the teacher version. The scale also contains five DSM-V Oriented scales (Achenbach, 2015). The Social Skills Improvement System-Rating Scales (SSIS-RS) is another tool utilized to assess young children‘s social skills (Gresham & Elliot, 2008). The SSIS-RS has a parent and teacher version, both of which assess two domains: social-emotional skills and behavior problems in children as young as age 3. The SSIS-RS assesses three areas (1) Social Skills (communication, cooperation, assertion, responsibility, empathy, engagement, and selfcontrol), (2) Problem Behaviors (externalizing, bullying, hyperactivity/inattention, internalizing, Autism Spectrum), and (3) Academic Competence (reading achievement, math achievement, motivation to learn). The SSIS-RS is based on a framework in which professionals can determine a child‘s strengths as well as competing behaviors (Gresham & Elliot, 2008). Naturalistic observation is another tool frequently used in the early childhood classroom. Observation in the field of early childhood has a long history dating back to the nineteenth century with Fredrick Frobel, who wanted kindergarten teachers to be observers in their own classrooms (Reifel, 2011). Observations in the early childhood classroom is a natural extension of what teachers are already doing on a regular basis, and is particularly useful when studying behavior in the classroom. This type of assessment is authentic in nature and can yield much information about a child‘s social skills and behavior problems. There are some issues with this method because it can be difficult to remain objective, as the observer‘s biases can play a role. Naturalistic observation provides a description of a child‘s behavior in the environment or context. This type of assessment can lead to more accurate and recent data (Yoder & Symons, 2010) in comparison to rating skills, which are indirect measures based on the memory of the individual who is rating the child. Progress monitoring of social-emotional and behavioral issues is challenging. Therefore, using multiple forms of assessment is valuable. One instrument, the Individual Growth and Development Indicators (IGDI), does allow for data-driven decision-making based from the results of progress monitoring (McConnell & Greenwood, 2013). Currently, IGDIs are used in research and practices, and are based on General Outcome Measures (GOM). GOMs were developed in order to help teachers and other professionals identify when intervention needed to be modified and adapted or when to continue with the intervention course (McConnell & Greenwood, 2013). GOMs can be used repeatedly and provide a measure of a child‘s growth over time. GOMs are brief in nature and are more specific to changes, due to development or intervention, than a standardized assessment. Interventions. Webster-Stratton and Reid (2013) suggest Tier 1 supports should include the use of specific social and emotional curricula that provide children a foundation for social skills, improve already developing skills, and identify classroom management strategies. Children benefit from evidence-based curriculum which focuses on emotional literacy, selfregulation, self-control, self-regulation, problem solving, friendship skills, following school rules, and anger management (Powell & Dunlop, 2009). Classroom management skills provided by a child‘s teacher can enhance social emotional development. Teachers who use praise and develop positive relationships with their students can improve social-emotional outcomes for young children. Also, teachers who enhance social-emotional competence and utilize home-school collaboration facilitate better social-emotional outcomes for children (Webster-Stratton & Reid, 2013).


29

Intervention at Tier 2 provides supports that are more intensive in nature. Supports at this level typically are provided to small groups of children and use embedded learning opportunities to promote social-emotional skills. Tier 2 supports include additional practice opportunities for specific skills, such as proactive teaching, use of praise for specific children, and opportunities to practice problem solving (Webster-Stratton & Reid, 2013). Children who need more intense interventions that cannot be addressed at Tier 2 may benefit from supports at Tier 3. Supports at this level include working with a child one-on-one more intensively. Intervention is individualized and adapted based on results of frequent progress monitoring. Similar strategies are used at Tier 3 with regards to emotion regulation, social competence, and problem solving skills; however, the intensity of the intervention and the individualized program are trademarks of this tier (Greenwood et al., 2011). There are several evidencebased programs which provide a framework for intensive interventions for social-emotional skills and challenging behaviors, a few examples include: The Incredible Years Teacher Management Program (Webster-Stratton, 1994), The Incredible Years: Dina Dinosaur Classroom Curriculum, Preschool/Kindergarten (Webster-Stratton, 2002), and Second Step (Committee for Children, 1991).

PRESCHOOL RTI PROJECT Preschool children entering a pre-kindergarten program in a southeastern state were screened using norm-based instruments for (1) pre-reading skills and readiness, and (2) social-emotional-behavioral adjustment. Many of the components of RTI were already being utilized in this high-quality preschool setting, such as research-based curriculum and instruction, problem-solving with regards to behavioral and academic issues, and screening and assessment. The RTI procedures used in this project added components such as progress monitoring, collaborative problem-solving, and data-driven decision making. Children falling into the empirical risk range on either screening device were placed into the appropriate intervention group, and specialized interventions were developed and provided for several months during the academic year. All children (both intervention groups plus all other typically developing students) were re-assessed with the original screening instruments in order to evaluate potential gains from Time 1 to Time 2 in the intervention groups. This goal was accomplished through assessment of the typically developing group of children in order to control for external factors such as normal developmental processes and general curriculum effects. Interventions were designed and monitored by faculty research members from an early childhood education program at a nearby university. The group of children identified as at risk for reading problems received the reading intervention program, which was provided by their teacher with faculty research members as consultants. Similarly, the group of children identified as at risk for social-emotional problems received the social-emotional intervention, delivered by the classroom teacher with faculty research members as consultants. As noted above, the primary intervention period was for several months in the academic year.

30


Assessment The Phonological Awareness Literacy Screening (PALS-PreK) for preschoolers was used to screen students in the areas of phonological awareness, alphabet knowledge, print and word awareness, name writing, beginning sound awareness, and rhyme awareness (PALSPreK Assessment, 2015). The PALS-PreK was administered two times during the academic school year; Time 1 which was at the beginning of the year, and then again at Time 2, after intervention was completed. The PALS-PreK assesses emerging literacy skills in young children and is developmentally appropriate for children who are 4 years old (Invernizzi, Sullivan, Meier, & Swank, 2004). Items created for the PALS were based on research that has been conducted on early literacy skills. Reliability for each tasks of the screening were .93 for beginning sound awareness, .84 for rhyme, .75 for print and word awareness, and .77 for nursery rhyme awareness. Reliability coefficients were not available for name writing and alphabet knowledge (Invernizzi, Sullivan, Meier, & Swank, 2004). The PALS-PreK was used to identify children who needed more than Tier 1 interventions. Teachers and researchers were also able to provide focused interventions and outcome goals based on results of the PALS-PreK. Challenging behaviors and social-emotional issues were assessed by the preschool form of the Achenbach System of Empirically Based Assessment (ASEBA). This is a rating form that was completed by the preschool classroom teacher at the beginning of the year, Time 1, and then again after intervention was completed, Time 2. All children were assessed during time Time 1 and Time 2. The Achenbach Teacher Report Form for children ages 1 ½ to 5 was used to assess preschool children‘s skills in the following areas: Emotionally-Reactive, Withdrawn/Depressed, Somatic Complaints, Attention, Aggressive, Internalizing, Externalizing, and Total Problems. Scores that ranged in the Borderline to Clinical range were used to identify children for intervention groups. Children were then divided into groups based on whether they were rated as having more Internalizing or Externalizing problems. These distinctions were used to develop interventions.

Intervention Tier 1. The literacy curriculum utilized in the preschool program was Letterland. Letterland is a phonics-based program that teaches reading, writing and spelling skills to children ages 3-8. Letterland focuses on letter shape and uses pictograms of letters to create stories (Letterland International, 2014). The theory behind the program is that mnemonics, or visual cues, are used to teach children letter sounds and shapes. Activities are taught with a multisensory method (Letterland International, 2014). Universal screening was completed with the PALS-PreK assessment. Progress monitoring for emergent reading skills was conducted three times during the academic year. Based on results of the universal screening and progress monitoring, students who needed further intervention were moved to Tier 2. Parents and caregivers were provided information about the literacy program and assessment process. Teachers in the preschool program use social-emotional strategies and activities based on materials developed by the Center on the Social and Emotional Foundations for Early Learning (CSEFEL). Activities implemented in the classroom include building positive


31

relationships, teaching social-emotional skills through literacy, and scripted stories for social situations. Universal assessment was conducted with the preschool form of the Child Behavior Checklist (ASEBA) in order to identify children who may need more intensive interventions in Tier 2. Parents and caregivers were provided information about the universal screening and other activities that their children would be receiving. Tier 2: Literacy supports in Tier 2 consisted of more intensive interventions and learning opportunities based on identification from the PALS-PreK assessment. Interventions were based on the data produced by the PALS-PreK assessment and interventions were used for targeting alphabet knowledge. A variety of interventions were selected from the Florida Center for Reading Research based on alphabet knowledge (FCRR, 2008), and the work of Dr. Susan Hall (I‘ve DIBEL‘d, Now What?, 2006). These research-based interventions were conducted 3-5 times per week in small groups. Progress monitoring was conducted bi-weekly by a member of the research team to ensure fidelity. Teachers were provided coaching every week by a literacy specialist on the research team in order to problem-solve and refine interventions for children in the intervention group. Families and parents were provided with information about the intervention process. Social-Emotional intervention at Tier 2 consisted of teachers providing interventions to children who were identified on the Child Behavior Checklist, preschool form. Interventions were designed based on The Incredible Years Teacher Classroom Management Program. The Teacher Classroom Management Program is designed to be a preventative program that can strengthen a teacher‘s classroom management skills (Incredible Years, 2013). The program is also designed to promote prosocial skills as well as school readiness in young children. The Teacher Management Program has led to reduced aggressive behaviors and noncompliance in the classroom (Incredible Years, 2013). It is designed to provide training to teachers in 6 full days or 42 hours, with time in between for teachers to try out techniques. Teachers and teacher assistants in this preschool project were provided training on a biweekly basis, resulting in approximately 32 hours of instruction, rather than 6 full days, due to classroom obligations. The training was broken into smaller components and was designed to allow teachers and assistants time to try activities and strategies in between sessions. Sessions were adapted and taught by a member of the research team, who is a licensed psychologist and has attended the Incredible Years Train-the-Trainers Program. Teachers were also provided with short coaching sessions 2-3 times per week, based on their needs, by a licensed psychologist from the research team. A consultation with all teachers took place once a week to determine needs and to adapt and modify interventions for each child in the intervention group. Strategies teachers utilized were from The Incredible Years Teacher Classroom Management Program and included reinforcement of prosocial behaviors, building positive relationships with children, preventing behavior problems, and motivating children (Webster-Stratton, Reid, & Hammond, 2004). Tier 3. Children who were not successful with Tier 2 interventions in the area of literacy were provided more intensive interventions which were individualized based on the results of bi-weekly progress monitoring and collaborative problem-solving. Interventions were based on activities developed by the Florida Center for Reading Research (FCRR, 2008). Children were provided interventions one-on-one in the classroom by their teacher or the teacher assistant. The literacy consultant continued to work with classroom teachers and teacher assistants in order to individualize literacy intervention for each child in the project. Activities were adapted and modified every two weeks based on the results of progress monitoring.

32


Children who were less responsive to Tier 2 interventions in the areas of social-emotional skills and challenging behaviors were provided with more intensive interventions in Tier 3. Interventions were based on The Incredible Years Teacher Classroom Program. Teachers were provided with consultation sessions 2-3 times per week in which the teacher and the coach would discuss the best way to adapt and modify plans in order to benefit each child. The following strategies were utilized from The Incredible Years Program Teacher Management Program: decreasing inappropriate behaviors and teaching problem-solving and self-regulations. Reinforcement of prosocial behaviors, building positive relationships with children, preventing behavior problems, and motivating children, which were included in Tier 2, continued in Tier 3 (Webster-Stratton, Reid, & Hammond, 2004).

Evidence of Success Data have been collected throughout the implementation period and progress has been documented in literacy skills, particularly letter naming both upper and lower-case, and in challenging behaviors and social-emotional skills. With regards to challenging behaviors, children in the intervention group were compared to the control group in the same program and made greater gains than the children in the control group on all categories of the Child Behavior Checklist, ASEBA preschool form. Significant improvement by the intervention group was found on the following scales: Emotionally-Reactive, Withdrawn/Depressed, Somatic Complaints, Attention, Aggressive, Internalizing, Externalizing, and Total Problems. Informal teacher feedback on the training sessions and ongoing coaching has been positive. Teachers have been enthusiastic about using the intervention strategies for both the literacy and social-emotional interventions. Teachers were positive about the individual training sessions as well as the consultation they received from the psychologist and the literacy specialist from the research team.

Lessons Learned One unexpected result from this project included the reactions of teachers to consistent and frequent progress monitoring of the literacy intervention. After progress monitoring took place, the progress of each child and each teacher‘s class of children who received intervention were shared with all teachers in a professional learning community. Sharing the progress results with all teachers at the same time proved to be very powerful, as it prompted discussion about why some teachers‘ students were making more progress than others. Results showed teachers seemed to be making the difference in the progress made by each child. The community discussions often took the form of sharing strategies that each teacher was using, such as frequency and groupings. Teachers that were providing intervention consistently and systematically 3-5 times per week were seeing the most progress, so their sharing with the group was especially notable. Darling-Hammond and Richardson (2009) contend the most powerful professional development opportunities are learning communities where teachers engage in ongoing dialogue about their practice and students‘ performance in efforts to develop more effective instructional practices. In this project, as the interventions,


33

progress monitoring, and collaborative meetings continued, teachers became more invested in how well their children were progressing. Progress monitoring is a very beneficial extension of the strategies early childhood teachers already use, particularly in monitoring children‘s academic skills in areas such as literacy, mathematics, and adaptive skills. The systematic use of progress monitoring can become somewhat more cumbersome for teachers due to other job requirements. Consistently reviewing progress monitoring results and making changes for each child based on those results can be time consuming for a teacher in a preschool classroom of approximately 16 children. Teachers in this project were appreciative of the assistance in this area and took part in the collaborative process of individualizing interventions based on progress monitoring. Frequent progress monitoring becomes much harder when assessing children with challenging behaviors or social-emotional issues. Several reasons contribute to the difficulty, which include: (1) there are few instruments developed for progress monitoring for socialemotional skills and challenging behaviors for young children (Feil & Frey, 2013), (2) standardized instruments are not specific enough to measure short-term growth and progress; and (3) norm-referenced instruments are not feasible with the amount of time it takes to administer the assessments (Greenwood, Carta & McConnell, 2011). One instrument can be used for progress monitoring in the area of social-emotional skills, such as individual growth and development indicators (IGDI) (McConnell & Greenwood, 2013). However, there are few examples in the research of this model of progress monitoring being used in RTI in preschool and IGDI is still considered an experimental form of progress monitoring.

RECOMMENDATIONS RTI intervention processes in preschool classrooms vary widely. Hagans-Murillo (2005) proposes using a standard-protocol for preschool RTI since children are in the beginning stages of acquiring essential pre-academic and social emotional skills. Yet, more long-term data on child outcomes related to using this RTI protocol, and other RTI approaches, in preschool classrooms are needed (Carta & Greenwood, 2013). Quality of instruction across preschool programs varies greatly, which considerably impacts RTI at all tiers of instruction (Greenwood et al., 2012). Teacher classroom management strategies vary across programs and affect improvement in social-emotional skills of young children (Webster-Stratton & Reid, 2013). Additionally, more professional development opportunities are needed for teachers, and other early childhood professionals, that identify how to implement a blended literacy approach in preschool classrooms. Coaching and consultation models that include collaborative problem-solving should be utilized. In this ongoing professional development, teachers can negotiate how to best use RTI while teaching literacy and social-emotional skills. This collaborative environment can also lead to support and procedures for addressing challenging behaviors. Lastly, teachers need more support on systematically implementing assessments, reviewing results, and using data to inform interventions at the whole class, small group, and individual levels in literacy and social-emotional skills.

34


REFERENCES Achenbach, T. (1991). The Child Behavior Checklist: Manual for the teacher‘s report form. Burlington, VT: Department of Psychiatry, University of Vermont. Achenbach, T. (2015). Preschool (ages 1 1/2 -5) assessment. Retrieved from http://www.aseba.org/preschool.html. Bailet, L. L., Repper, K. K., Piasta, S. B., & Murphy, S. P. (2009). Emergent literacy intervention for prekindergarteners at risk for reading failure. Journal of Learning Disabilities, 42 (4), 336-355. Bloodgood, J. W. (1999). What‘s in a name? Children‘s name writing and literacy acquisition. Reading Research Quarterly, 34(3), 342-367. Ball, C.R., & Trammell, B.A. (2011). Response-to-Intervention in high-risk preschools: Critical issues for implementation. Psychology in the Schools, 48, 502-512. Bayet, M., Mindes, G., & Covitt, S. (2010). What does RTI (Response to Intervention) look like in preschool? Early Childhood Education Journal, 37, 493-500. Buysee, V., Peisner-Feinberg, E., & Burchinal, M. (2012). Recognition & Response: Developing and evaluating a model of RTI for pre-K. Society for Research on Educational Effectiveness (SREE) Research report. Retrieved from http://eric.ed.gov/?id=ED530413. Carta, J. J., & Greenwood, C. R. (2013). Promising future research directions in Response to Intervention in early childhood (pp. 421-431). In D. Haager, J. Klingner, & S. Vaughn (Eds.), Evidence-based reading practices for response to intervention. Baltimore: Brookes. Coleman, M.R., Roth, F.P., & West, T. (2009). Roadmap to pre-k RTI: Applying response to intervention in preschool settings. New York: National Center for Learning Disabilities. Retreived from RTINetwork.org. Committee for Children, (1991). Second step: A violence prevention curriculum. Preschoolkindergarten. Seattle, WA: Author. Council for Exceptional Children (CEC). 2014. Improving outcomes for children with disabilities through high-quality early learning programs in Issue Brief. Retrieved from http://www.cec.sped.org/~/media/ Files/CAN%20Documents/CAN%202014/Issue%20Briefs/Early%20Learning%20Issue %20Brief%20FINAL.pdf. Curenton, S. M., Justice, L. M., Zucker, T. A., & McGinty, A. S. (2013). Language and literacy curriculum and instruction. In V. B. (Ed.), Handbook of Response to Intervention in early childhood (pp. 237-249). Baltimore, Maryland. Brookes Publishing. Darling-Hammond, L., & Richardson, N. (2009, February). Teacher learning: What matters? Educational Leadership 66(5), 46-53. Division of Early Childhood/National Association for the Education of Young Children/National Head Start Association. (2013). Frameworks for response to intervention in early childhood: Description and implications. Retrieved from http://www.naeyc.org/content/frameworks-response-intervention. East, B. (2006). Myths about Response to Intervention (RTI) Implementation. RTI Action Network. Retrieved from http://www.rtinetwork.org/ learn/what/mythsaboutrti.


35

Feil, E.G. & Frey, A.J. (2013). Assessment of social-emotional and behavioral skills for preschoolers within a response to intervention model. In Buysse, V. & Peisner-Feinberg, E.S. (Eds.), Handbook of response to intervention in early childhood (pp. 185-203). Baltimore, MD: Brookes. Florida Center for Reading Research (2008). Book one: Phonological awareness and phonics. Retrieved from: http://www.fcrr.org/for-educators/sca_k-1_rev.asp. Florida Center for Reading Research (2008). Book two: Fluency, vocabulary and comprehension. Retrieved from: http://www.fcrr.org/for-educators/sca_k-1_rev.asp. Fuchs, L. S. (2003). Assessing intervention responsiveness: Conceptual and technical issues. Learning Disabilities Research and Practice, 18, 172-186. Fuchs, L. S., & Fuchs, D. (2007). The role of assessment in the three tier approach to reading instruction (pp.29-49). In D. Haager, J. Klingner, & S. Vaughn (Eds.), Evidence-based reading practices for response to intervention. Baltimore: Brookes Gilliam, W.S. (2005). Prekindergarteners left behind: Expulsion rates in state prekindergarten systems. Retrieved from http://www.ziglercenter.yale.edu/publications/34775_National %20Prek%20Study_expulsion%20brief.pdf. Greenwood, C.R., Bradfield, T., Kaminiski, R., Linas, M., Carta, J.J., & Nylander, D. (2011). The response to invervention (RTI) approach in early childhood. Focus on Exceptional Children, 23, 1-22. Greenwood, C. R., Carta, J. J., Atwater, J., Goldstein, H., Kaminski, R., & McConnell, S. (2012). Is a Response to Intervention (RTI) approach to preschool language and early literacy instruction needed? Topics in Early Childhood Special Education, 33(1), 48-64. Greenwood, C. R., Carta, J.J., & McConnell, S. (2011). Advances in measurement for universal screening and individual progress monitoring of young children. Journal of Early Intervention, 33, 254-267. Gresham, F.M. & Elliot, S.N. (2008). Social Skills Improvement System (SSIS) – Performance Screening Guide. Upper Saddle River, NJ: Pearson. Hagans-Murillo, K. (2005). Using a response-to-intervention approach in preschool to promote literacy. The California School Psychologist, 10, 45-54. Hall, S. (2006). I‟ve DIBEL‟d, now what? Dallas: Sopris West. IDEIA. (2004). Individuals with Disabilities Education Improvement Act of 2004. P.L. No. 108-1446. Retreived from http://idea.ed.gov/download/finalregulations.pdf. Incredible Years Program (2013). Teacher Classroom Management Program. Retrieved from http://70.40.220.26/programs/teacher/classroom-mgt-curriculum/. Invernizzi, M., Sullivan, A., Meier, J., & Swank, L. (2004). Phonological awareness literacy screening. Richmond, VA: University of Virginia Printing and Copying Services. Karoly, L. A., Kilburn, M. R., & Cannon, J. S., (2005). Early childhood interventions: Proven results, future promises. Santa Monica, CA: Rand Corporation. Letterland International, (2014). What is Letterland? Retrieved from http://www. letterland.com/what-is-letterland. Lonigan, C. J., Wagner, R. K., Torgesen, J. K., & Rashotte, C. A. (2007). Test of preschool early literacy. Austin, TX: Pro-ed. mCLASS: CIRCLE (2015). Amplify Education, Inc. Retrieved from https://www. mclasshome.com/wgenhelp/circle/desktop/Assessments/Assessments_Main.htm McConnell, S. & Greenwood, C.R. (2013). General outcome measures in early childhood and individual growth and developmental Indicators. In V. Buysse, & E.S. Peisner-Feinberg

36


(Eds.), Handbook of response to intervention in early childhood (pp. 143-154). Baltimore, MD: Brookes. Mamedova, S., & Redford, J., & Zukerberg, A. (2013). Early childhood program participation, from the national household education surveys program of 2012 (NCES 2013-029). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, D.C. Retrieved from http://nces.ed.gov/pubs2013/ 2013029.pdf. National Early Literacy Panel (2008). Developing early literacy. Jessup, Maryland: National Institute for Literacy. National Professional Development Center on Inclusion. (2012). Summary from listening sessions. Chapel Hill: University of North Carolina, Frank Porter Graham Child Development Institute. Retrieved from http://npdci.fpg.unc.edu/resources/rti-summarylistening-sessions. PALS-PreK Assessment (2015). PALS™ Marketplace. Retrieved from https://www. palsmarketplace.com/assessments/pals_prek/ Peisner-Feinberg, E. S., & Buysee, V. (2013). The role of assessment within Response to Intervention in early education. In V. B. (Ed.), Handbook of Response to Intervention in early childhood (pp. 121-142). Baltimore, Maryland. Brookes Publishing. Powell, D. & Dunlop, G. (2009). Evidence-based social-emotional curricula and intervention packages for children 0-5 years and their families (Roadmap to Effective Intervention Practices). Tampa, Florida: University of Florida, Technical Assistance Center on Social Emotional Intervention for Young Children. Preschool Early Literacy Indicator (2015). Dynamic Measurement Group. Retrieved from https://dibels.org/peli.html Reifel, S. (2011). Observation and early childhood teaching: Evolving fundamentals. Young Children, 66, 62-65. Scull, J., Nolan, A., & Raban, B. (2013). Young learners: Interpreting literacy practice in the preschool years. Australian Journal of Language and Literacy, 36(1), 38-47. VanDerHeyden, A. (n.d.). Approaches to RTI. RTI Action Research Network. Retrieved from http://www.rtinetwork.org/learn/what/approaches-to-rti. VanDerHeyden, A. M., Snyder, P. A., Broussard, C., & Ramsdell, K. (2007). Measuring response to early literacy intervention with preschoolers at risk. Topics in Early Childhood Special Education, 27(4), 232-249. Webster-Stratton, C. (1994). The Incredible Years (IY) Series Teacher Classroom Management Program. Seattle, WA: Incredible Years. Webster-Stratton, C. (2002). Effective classroom management skills training and dina dinosaur‘s social skills and problem-solving curriculum training for the classroom: Leader‘s guide. Seattle, WA: Incredible Years. Webster-Stratton, C., Reid, M.J., & Hammond, M. (2004). Treating children with early-onset conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical and Adolescent Psychology, 33, 105-124. Whitcomb, S.A. & Merrill, K.W. (2013). Behavioral, social, emotional assessment of children and adolescents. (4th Ed.). New York, NY: Routledge. Whitehurst, G. J., & Lonigan, C. J. (2001). Get Ready to Read! Screening Tool. New York: National Center for Learning Disabilities.


37

Wilson, S. B, & Lonigan, C. J. (2009). An evaluation of two emergent literacy screening tools for preschool children. Annals of Dyslexia, 59, 115-131. Yoder, P. & Symons, F. (2010). Observational measurement of behavior. NY, New York: Springer.



Chapter 3

TYPES OF PARENT INVOLVEMENT AS PREDICTORS OF THE POSTSECONDARY EDUCATIONAL PLANS AND FUTURE EDUCATIONAL ASPIRATIONS OF 7TH AND 9TH GRADE STUDENTS Kristin Skells, Lee Shumow and Jennifer A. Schmidt Northern Illinois University, IL, US The material is based upon work supported by the National Science Foundation under Grant No: HRD-1136143. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not reflect the views of the National Science Foundation.

ABSTRACT This study examines the impact of different types of parent involvement on the postsecondary educational plans and future educational aspirations of 298 students in seventh or ninth grade. Parent involvement in school, parent involvement at home, and mothers‘ and fathers‘ expectations for their children were measured to assess how parent involvement predicts postsecondary educational plans (level of education the student plans to pursue immediately after high school) and future educational aspirations (goal for total years of education). Analyses tested whether these relationships differ by student gender. Results of OLS regressions indicated that, while mothers‘ academic expectations did not predict postsecondary educational plans or future educational aspirations for either grade level, fathers‘ academic expectations showed different patterns of association for seventh vs. ninth graders. For example, fathers‘ academic expectations significantly predicted both students‘ postsecondary plans and future educational aspirations for ninth graders, but was only predictive of future educational aspirations among seventh graders. Parent involvement in school also predicted future educational aspirations for ninth graders but not seventh graders. Parent involvement in homework was not predictive of postsecondary educational plans or outcomes at any grade level. Further analyses suggest that fathers‘ academic expectations are particularly predictive for female students.

40

Kristin Skells, Lee Shumow and Jennifer A. Schmidt Implications are discussed for promoting subtle, yet impactful forms of parent involvement.

INTRODUCTION Adolescence is an important period for the development of aspirations and plans for the future (Erikson, 1963; Nurmi, 1991). Parents play an important role in their young adolescents‘ academic adjustment, in general, and have been recognized as having an impact on student‘s educational aspirations and plans for the future, in particular (Eccles, Vida, & Barber, 2004). Given those findings, it is of theoretical and practical value to pursue a greater understanding of the ways in which parent involvement promotes postsecondary educational plans and future educational aspirations during adolescence. The study described in this chapter considers how various types of parent involvement predict adolescents‘ postsecondary educational plans and future educational aspirations. The extent to which the impact of parental involvement differs by grade and gender of students and parents also is investigated.

Adolescents’ Postsecondary Educational Plans and Future Educational Aspirations Adolescents‘ postsecondary educational plans and future educational aspirations are important because they predict students‘ later educational attainment as well as contributing to their burgeoning identity development. For example, a longitudinal study of adolescents' educational aspirations and their relation to educational attainment five years later found that students‘ educational aspirations and their family background combined were associated with later educational attainment (Marjoribanks, 2005). Another longitudinal study also found that having occupational aspirations in adolescence contributed positively and significantly to a model predicting young adults‘ educational attainment (Beal & Crockett, 2010). An examination of the prior literature makes it clear that there is a relationship between adolescents‘ aspirations and their future educational pursuits and achievements. This study seeks to expand our understanding of how adolescents‘ develop future aspirations and postsecondary plans by investigating the contributions of their parents controlling for background factors of immigrant status, SES, and parent educational level which have been associated with both parent involvement and the outcomes of interest in this study.

Parent Involvement Scholars, educational practitioners, and the public generally agree that parent involvement in their children‘s education has an impact on students‘ educational outcomes (Bracke & Corts, 2012). The recognition of parent involvement as an asset to students has resulted in various policies and initiatives aimed at increasing parental involvement in schools (Moles & Fege, 2011). Yet, researchers have recognized that there are different ways that parents are involved in their children‘s education and with different consequences (Epstein, et

Types of Parent Involvement As Predictors …

41

al., 2009; Green, Walker, Hoover-Dempsey, & Sandler, 2007). It is important to identify how those different types of parent involvement may differentially impact students so that initiatives can be targeted to promote particular outcomes. The current study considers how several types of parental involvement might impact adolescents‘ postsecondary educational plans and future educational aspirations including parent involvement at school, parent involvement in homework, and parents‘ educational expectations. Hossler and Gallagher‘s (1987) model of college choice and college planning highlights the importance of both the behaviors and values of parents in developing adolescents‘ future educational aspirations. A study of this model by Myers and Myers (2012) confirmed the important role that parents play in socializing their children to aspire to attend college. Socialization to attend college can be enacted in multiple ways. First, at the most basic level, parents are role models who set an example for their children (Parsons, Adler, & Kaczala, 1982). Parents‘ own level of education, for instance, predicts their children‘s eventual level of educational attainment as adults indirectly (Dubow, Boxer, & Huesmann, 2009). Accordingly, in this study, we control for parents‘ educational level in order to concentrate on the more proximal processes of their behavioral involvement and their educational expectations for a particular child. Parents can also serve as role models through their behavioral involvement with their children‘s schooling.

Parent’s Behavioral Academic Involvement with Adolescents Parents‘ behavioral involvement with their children‘s schooling has been the focus of numerous studies, many of which have associated parents‘ involvement at their child‘s school and/or their assistance with homework with students‘ grades, achievement test scores, or achievement motivation. Parents school-based involvement, in particular, has been associated positively with adolescents‘ grades and with their effort and interest during class (Shumow, Skells, Schmidt, & Kackar-Cam, 2014). Researchers also have found that school-based involvement of parents predicts adolescent students‘ decision to enroll in college (Perna & Titus, 2005). The impact of parents‘ involvement with homework on school achievement during adolescence has been somewhat unclear (Hill & Tyson, 2009; Shumow, 2009). However, monitoring, helping, and supporting adolescents‘ homework has been associated with students‘ effort, valuing of and interest in learning in class, at least during the short term (Shumow, Skells, Schmidt, & Kackar-Cam, 2014). Collectively, these findings suggest that parents‘ behavioral involvement is likely to have some impact on adolescents‘ postsecondary educational plans and future educational aspirations, a speculation that we test in the study described in this chapter. Parents’ Academic Expectations of Adolescents Jeynes (2010) has suggested that some of the more subtle forms of home-based parent involvement, such as the academic expectations of the parents, may be more influential than some of the more heavily-researched, behavioral forms of involvement such as monitoring homework completion and attending meetings at school. In various studies, high parent expectations were related to higher academic achievement among adolescents (Fan & Chen, 2001; Jeynes 2003, 2005, 2007; Yamamoto & Holloway, 2010). Importantly, Jodl and her colleagues (2001) showed that parent expectations contribute to students‘ aspirations. The impact of parents as expectancy socializers is of particular relevance for situations where adolescents are developing plans and aspirations for the future. According to Eccles and her

42

Kristin Skells, Lee Shumow and Jennifer A. Schmidt

colleagues (Eccles, Vida, & Barber, 2004; Parsons, Adler, & Kaczala, 1982), parents' expectations related to educational achievement are manifested through the subtle communication of beliefs about their adolescent's ability to succeed at a task and the value of educational attainment. Thus, depending on the expectations of the parents, students can receive different messages regarding the role of education in their future. The connection between parental expectations and students future aspirations and planning for the future is tested in the study reported in this chapter. Of course one way that parents might be influential is through discussions with their children about their future plans. Zinth (2009) reports that adolescents turn to their parents to discuss their plans more than to any other adults. Unfortunately, we were not able to include that variable in our analyses.

Gender Gender is likely to be a factor in the issues of interest in the present study in several ways. Despite the evidence that parents‘ expectations are important for adolescents, there is a relative dearth of studies that have considered potential differences in the predictive nature of the mothers‘ versus fathers‘ expectations for their children especially in combination with their behavioral involvement. For that reason, both the mother‘s academic expectations and the father‘s academic expectations will be considered. Gender of the student also is investigated as a factor that might predict future aspirations and plans among adolescent students. There are well-documented differences in the way male and female students‘ expectations about their future relate to later educational attainment. While both male and female adolescents‘ educational expectations predict later educational achievement (Mello, 2008), female students anticipate more barriers to their educational attainment than their male peers (McWhirter, 1997). The impact of different types if parent involvement may differ for male and female students. Parent monitoring and regulation are associated with better grades in school for boys while the relationship with this type of involvement has not been found for female students (Stolz et al., 2004). Parsons' understanding of parents as expectancy socializers suggests that male and female students may glean different messages about education from their parents, while conceptualizing parents as role models suggests that male and female students may exhibit different educational behaviors based on the behaviors of their same sex parent. Thus, in addition to considering the impact of the expectations of mothers and fathers on future educational aspirations and postsecondary plans while controlling for gender, the impact of mothers and fathers expectations are tested together with other forms of parent involvement separately for female and male students. Although prior studies have looked at how parent involvement has differential effects of educational outcomes based on characteristics of the family including race and socioeconomic status, researchers have not yet thoroughly considered the relationship between gender, parent involvement, and students‘ postsecondary plans and future educational aspirations.


43

Grade Level Although researchers have established adolescence as an important period for the development of aspirations for the future and goal-directed behaviors, the differential impact of types of parent involvement during different periods of adolescence is still being examined. Hill and Tyson (2009) have suggested that cognitive changes in adolescence foster the development of goals and aspirations for the future thereby decreasing the need for and value of more direct forms of home and school based involvement in favor of more subtle academic socialization by parents. Indeed, parents‘ direct involvement with homework declines across adolescence and appears to make contributions to children‘s success during elementary school but not during middle school (see Shumow, 2010 for a review). Parents of ninth graders are less involved at school than are parents of middle school students but they are more involved in discussing future plans (Shumow, Skells, Schmidt, & Kackar-Cam, 2014). For this reason, this study considers the influence of different forms of parent involvement on students‘ postsecondary plans and future educational aspirations during two points of early adolescence. Recent research has shown that parent involvement both changes between seventh and ninth grade and has differential impacts on adolescent motivation (Shumow, Skells, Schmidt, & Kackar-Cam, 2014), so we conducted analyses for the sample as a whole and by each of these two grade levels.

Research Questions The study described in this chapter aims to: (a) assess how mothers‘ and fathers‘ academic expectations for their children impact their students‘ postsecondary plans and future educational aspirations; (b) consider how different types of parent involvement predict young adolescents‘ postsecondary educational plans and future educational aspirations and whether there are differences in those relationships by grade level (7th vs. 9th); (c) explore whether those types of parent involvement predict 7th and 9th graders future plans differently by student gender. Drawing on previous research, we predicted that parent involvement, specifically parents‘ expectations, would predict postsecondary educational plans and future educational aspirations more than parent involvement at home or parent involvement at school.

METHODS Participants Data were collected from 7th and 9th grade students from a large diverse Midwestern school district over the course of a school year. The sample was 49% male, 49% immigrant, 64% low-income, and 21% white. Background characteristics of the 7th grade and 9th grade samples can be found in Table 1. The vast majority of immigrants were from Mexico.

44

Kristin Skells, Lee Shumow and Jennifer A. Schmidt

Procedure This study is a secondary analysis of data from the IMUScLE project (Schmidt, Shumow, & Durik, 2011). Researchers collected information about students‘ demographic characteristics from students (gender, race, immigration status, parents‘ level of education) in seventh and ninth grade science classrooms early in the school year and from school records (free and reduced lunch status). Data were collected about various features of the students‘ school experience over a six week period. Later in the school year, surveys were used to assess students‘ perception of their mothers‘ and fathers‘ academic expectations, parental involvement in school, parental involvement in homework, as well as students‘ postsecondary educational plans and their future educational aspirations. Table 1. Sample Demographics

Male Female Non-Hispanic White Immigrant Highest level of parent education Less than HS Graduated from HS Some college Graduated from College Advanced degree Free/reduced lunch

7th Graders n = 120 44% 56% 20% 50%

9th Graders n = 178 52% 48% 21% 47%

16% 26% 20% 20% 18% 57%

23% 30% 16% 16% 14% 69%

Measures Demographic Variables Gender, race/ethnicity, immigrant status, and highest parent level of education were reported by students through the student survey. In the analyses reported here, a non-Hispanic White variable was created from the race ethnicity variable (non-Hispanic white = 1, other = 0). This variable was created because non-Hispanic white parents were significantly different from all other race/ethnicity groups in terms of their reported parent involvement while the other race/ethnicity groups did not significantly differ from each other. Also, racial/ethnic minorities had similar SES characteristics as each other, and these differed significantly from the SES characteristics of whites. Students were considered immigrants if the student reported that he/she and/or at least one parent were born outside of the United States. The highest parent level of education was reported by the students with options such as did not graduate from high school, graduated from high school, completed some college but did not graduate, graduated from college, and earned an advanced degree. Free/reduced lunch eligibility data was provided by school administrators, and was used as a proxy for socioeconomic status.


45

Parent Involvement in Homework Three items measured parent involvement in homework. Using a four point scale, students reported how often their parents: checked to see if they had completed their homework, helped them with homework, or found help for them if they were unable to help (Mean = 1.06., SD =.84). Cronbach‘s alpha was .71. Parent Involvement in School Parent involvement in school was created from the mean of five items which asked students to report their parents‘ involvement in school. Students were asked to respond yes or no to items asking if this school year their parents know their teacher‘s name, have talked with their science teacher, have attended events, meetings, or activities at school, have attended an event or activities at school to watch them participate or perform, or have done volunteer work at school. (Mean = .40, SD = .32). Cronbach‘s alpha for those items was .71. Mothers’ and Fathers’ Academic Expectations Students were asked to report how far in school their mother expects them to go and then how far in school their father expects them to go. Students responded using an eight point scale ranging from 0 = less than high school to 7 = Ph.D, M.D. or other advanced professional degree. Students were also able to respond ―I don‘t know what they want for me.‖ This response was recoded as a 0 on the scale because if students do not know of their parent‘s educational expectations, it is unlikely that this is communicated to them (Mother: Mean=4.78, SD=2.16; Father: Mean=4.64, SD=2.36). Postsecondary Educational Plans Students responded to an item asking what they plan to do after leaving high school. Responses were rated on a four point scale with varying levels of postsecondary educational plans including (1) get a full-time job after high school, (2) attend a trade/professional school, (3) attend a two-year college, and (4) attend a four year college (Mean = 3.49, SD= .91). Future Educational Aspirations Students‘ future educational aspirations were assessed through one survey item. Students responded to this item on an eight point scale ranging from (0) Less than high school to (7) Ph.D, M.D. or other advanced professional degree (Mean = 5.02, SD=1.57). Students were also able to respond ―I don‘t know how far I‘ll go‖ and this response was not included in the analyses.

RESULTS OLS regression analyses tested models predicting students‘ future educational aspirations and models predicting their postsecondary educational plans. Models were tested for the full sample (Table 2), for female students only (Table 3), and then for male students only (Table 4).

Table 2. Predicting Students Future Educational Aspirations and Postsecondary Educational Plans

White Immigrant Free/Reduced Lunch Parent Level of Education Mother‘s Expectations Father‘s Expectations Parent Involvement in School Parent Involvement in Homework R2 Adj R 2

Future Aspirations Full Sample (n=298) (n=289) -.09 -.09 .16* .16*

7th Graders (n=120) (n=119) -.06 -.09 .07 .06

9th Graders (n=178) (n=170) -.10 -.09 .21** .23**

Postsecondary Plans Full Sample (n=298) (n=289) .03 .03 .08 .07

7th Graders (n=120) (n=117) .13 .10 .10 .08

9th Graders (n=178) (n=170) -.05 -.02 .05 .06

-.13*

-.11

-.07

-.04

-.19*

-.17*

-.17**

-.14*

-.16

-.15

-.19

-.15^

.27***

.26***

.34**

.30**

.21**

.22**

.17*

.14*

.18^

.11

.16

.15^

.04

---

-.06

---

.09

---

-.03

---

-.16^

---

.04

---

---

.18**

---

.22*

---

.16*

---

.20**

---

.19

---

.20**

.19**

.17**

.17^

.14

.20**

.18*

.06

.01

.05

.01

.07

.02

-.07

-.08

-.04

-.08

-.07

-.06

.02

.03

.03

-.01

.01

.07

.17*** .15***

.19*** .17***

.16* .10*

.20** .14**

.19*** .16***

.20*** .16***

.07** .05**

.11*** .09***

.10 .03

.11 .04

.09* .05*

.13*** .09***

Note * p.05]. This result shows that although there is a certain amount of difference between the two groups‘ mean scores, this difference does not suggest that concordancing activities induced target vocabulary production for the experimental group. However, when the issue is vocabulary production, delayed effects of the learning process have to be considered. With this concept in mind, the learners were given another writing exam (delayed post-test), and target vocabulary density scores were calculated as before. Delayed post-test mean and corrected mean scores for target vocabulary density for the experimental and control groups are presented in Table 9. Table 8. ANCOVA Test Results Comparing the Experimental and Control Groups for Target Vocabulary Density Scores (Post-test) Source Pre-test Group Error Total

Sum of Squares 586.285 168.563 2240.365 11194.522

df 1 1 34 37

Mean Square 586.285 168.563 65.893

F 8.898 2.558

p .005 .119

Table 9. Mean and Corrected Mean Scores for Target Vocabulary Density for the Experimental and Control Groups (Delayed Post-test) Group

N

x

Corrected Means

Experimental

18

22.297

22.302

Control

19

13.928

13.923

Again, Levene test for equality of variance was carried out and the result came out negative (p=.643>.05), which showed that the scores were suitable for parametric comparisons. After variance equality was determined, an ANCOVA test was carried out with the corrected delayed test scores by taking the pre-test scores as the covariant. The results are presented in Table 10. Table 10. ANCOVA Test Results Comparing the Experimental and Control Groups for Target Vocabulary Density Scores (Delayed Post-test) Source Pre-test Group Error Total

Sum of Squares 105.665 648.907 3837.058 16577.421

df 1 1 34 37

Mean Square 105.665 648.907 112.855

F .936 5.750

p .340 .022

ANCOVA test results comparing the experimental and control groups for target vocabulary density scores gathered from the delayed post-test clearly indicate that there is a statistically significant difference between the groups [F(1-34) = 5.750, p