Mar 1, 2005 - For their many contributions to this paper, the authors would like to acknowledge the entire research team: Katherine Baisden, Paul Hu, and ...
March 2005
EVALUATION OF THE CALIFORNIA SUBJECT MATTER PROJECTS YEAR 1 REPORT
Prepared for: Robert Polkinghorn Executive Director, University of California Office of the President
Submitted by: H. Alix Gallagher, Christopher Chang-Ross, Heather Hough, Juliet Tiffany-Morales, Camille Esch, Tiffany Price, Cimone Satele, Patrick Shields, and Heidi Skolnik
ACKNOWLEDGEMENTS For their many contributions to this paper, the authors would like to acknowledge the entire research team: Katherine Baisden, Paul Hu, and Harold Javitz. We would also like to thank the executive directors and site directors of the California Subject Matter Projects and the staff at the University of California Office of the President for participating in interviews, completing our survey, and compiling the documents necessary for our work. This report would not have been possible without their cooperation.
CONTENTS
Chapter 1—Introduction ...............................................................................................................1 Chapter 2—Budget and Policy .....................................................................................................7 Chapter 3—Scope and Scale of the CSMP.................................................................................27 Chapter 4—Nature of the Work..................................................................................................41 Chapter 5—Impact of CSMP on California Teachers and Their Students.................................63 Chapter 6—Summary .................................................................................................................79 References.................................................................................................................................R-1 Appendix A.............................................................................................................................. A-1 Appendix B ...............................................................................................................................B-1
SRI Proprietary
i
SRI Proprietary
CHAPTER 1—INTRODUCTION To improve the instructional skills and content knowledge of beginning and experienced teachers, California’s policy-makers have created a series of professional development programs. Key among these initiatives has been the California Subject Matter Projects (CSMP), a longstanding operation administered by the University of California Office of the President (UCOP) and housed on university campuses across the state. The network of nine CSMP, which was established in 1988 by SB 1882 (Swofford), provides content-rich subject matter professional development in the following areas: foreign languages, history-social studies, international studies, mathematics, physical education and health, reading and literature, science, writing, and art. Each of the Projects supports a number of regional sites throughout California (Exhibit 1-1). Exhibit 1-1 Projects and Number of Sites, 2003-04 Abbreviation
Number of Sites (FY 2003-04)
CFLP
9
CH-SSP
8
California International Studies Project
CISP
10
California Mathematics Project
CMP
22
CPE-HP
3
CRLP
14
California Science Project
CSP
19
California Writing Project
CWP
17
The California Arts Project
TCAP
7
Project California Foreign Language Project California History-Social Science Project
California Physical Education-Health Project California Reading and Literature Project
Source: UCOP (2004a).
The CSMP’s approach consists of bringing teachers together for extended periods to build content-specific expertise and pedagogical skills. The Projects’ key strategies are to provide professional development to teachers, develop teacher leadership, and create and maintain discipline-specific networks of teachers and university faculty. Each Project’s work is based on standards or frameworks in that subject area; that is, professional development activities reflect the skills and content knowledge that students must master. The Projects have historically provided professional development that draws on many features supported by research on effective professional development. These features include:
SRI Proprietary
1
sufficient duration (longer events are more likely to improve teacher practice), collective work (teachers are more likely to implement new knowledge and skills if they attend events with colleagues), content focus (teachers need deeper content knowledge to teach to challenging academic standards), active learning (the more engaged teachers are, the more likely they are to learn), coherence (teachers are more likely to implement new strategies if they are aligned with curricular materials, standards, assessments, and other policies affecting teachers), differentiation (professional development should match the skill level of participating teachers), and ongoing “reform” activities like study groups or mentoring that seek to change teacher practice (one-time workshops tend to be minimally effective) (Birman, Desimone, Porter, & Garet, 2000; Garet, Birman, Porter, Desimone, Herman, & Yoon, 1999; Kennedy, 1999; Loucks-Horsley, 1999). During its tenure, the CSMP has been subject to many changes as the state’s education policy environment has evolved. Periodically, the UCOP issues a Request for Proposals (RFP) to approve potential and existing CSMP sites. Each RFP includes specific requirements that sites must meet in order to be funded, and policy-makers have used these periodic reauthorizations to focus the work of the CSMP on state priorities. Although the Projects have continuously provided discipline-specific training to teachers, their target audiences and menu of activities have shifted in response to new RFPs and federal requirements. In recent years, the state has increased efforts to boost achievement in low-performing schools and has focused reform initiatives around the content standards. To support these efforts, AB 1734 (1998, Mazzoni) and the reauthorizing RFP directed the CSMP to reserve 75% of available institute slots for teachers from “low-performing schools” (schools in the bottom 4 deciles of the state’s Academic Performance Index [API]), to partner with low-performing schools, and to provide standards-based professional development. In addition, the RFP required sites to evaluate site activities, with a particular focus on student achievement. The 2001 RFP expanded partnership work to focus on districts and also reflected the state’s growing attention to English language learners (ELLs) by requiring that Projects provide activities that addressed literacy through the content areas and the needs of teachers of ELLs. Recently, funds and policies associated with the federal No Child Left Behind Act (NCLB) have added two new requirements: providing professional development and training programs for teachers who need to achieve “highly qualified” status, and serving and supporting schools that have missed their targets for Adequate Yearly Progress (AYP) under the federal law. These activities complement the work the CSMP has been undertaking for several years.
SRI Proprietary
2
OVERVIEW AND DATA COLLECTION FOR THIS EVALUATION Periodically, throughout the history of the CSMP, the University of California Office of the President (UCOP) has commissioned evaluations of the Projects, the most recent of which was completed in 2002 (Quick, Lieberman, Farr, Cole, Wang, et al.). This Year 1 report, which is required under AB 2950 (2002, Strom-Martin) and SB 611 (2003, Ducheny), is one of two SRI International (SRI) is undertaking under contract with UCOP. The legislation requires the evaluation to document the CSMP’s impact on student achievement (that component will be addressed in the forthcoming Year 2 report); review internal documents and evaluations conducted by the CSMP; detail the quantity, quality, cost-effectiveness, and inclusiveness of the CSMP; and describe the CSMP’s impact on low-performing schools affiliated with it. To this end, the evaluation is to address the following set of five research questions; this report focuses primarily on the first three. 1. How are the CSMP responding to the current policy context, including changes in funding and organizational pressures for fiscal efficiency? 2. What are the scope and scale of CSMP activities? To what degree do CSMP activities address the needs of: a. Teachers of English Language Learners (ELLs)? b. Low-performing schools? c. Underprepared teachers? 3. What is the nature of the professional development provided by the CSMP, and to what extent is it consistent with the literature on high-quality professional development? 4. How do the CSMPs affect teachers’ abilities to provide high-quality, standards-based instruction in specific content areas? How do the CSMPs affect teachers’ abilities to provide high-quality, standards-based instruction for ELLs in specific content areas? 5. What is the relationship between CSMP participation and student achievement? The evaluation will encompass several data collection activities over 2 years (Exhibit 1-2). The Year 1 report is based on findings from a review of prior research on the CSMP, a review of CSMP internal data files, interviews with Project leaders, and a survey of site leaders. Next year’s expanded report will include findings from additional evaluation activities, including surveys and case studies of participating teachers and an analysis of the CSMP’s impact on student achievement.
SRI Proprietary
3
Exhibit 1-2 Evaluation Activities Year 1
Year 2
• Review of prior research conducted on the CSMP
• Review of prior research conducted on the CSMP
• Review of CSMP internal data files
• Review of CSMP internal data files
• Interviews with Project leaders
• Interviews with Project leaders
• Survey of site leaders
• Survey of participating teachers • Case studies of participating teachers • Student achievement analysis
This year, the review of prior research was based on literature provided by the CSMP. Documents ranged from summaries of evaluation data collected by sites to full-scale published evaluations. We also conducted a review of Project materials, including budgets and Web-based program descriptions. These documents provided descriptions of Projects’ events and evidence of their quality. We supplemented the descriptions by visits to three Science Project institutes and a site director meeting, conducted in conjunction with a related evaluation of the Science Project. Because these data pertained only to one of the nine Projects, they have been used to illustrate only those themes that other data sources suggest exist across multiple Projects. The review of CSMP internal data files consisted of an analysis of Project data collected in UCOP’s Online Information System (OIS). The portions of the OIS that we analyzed included those containing data on site partnerships, CSMP participants, and CSMP events. The technical appendix for this report (Hough, et al., 2005) provides details for these analyses. We also interviewed the executive directors (or co-executive directors) of all nine Projects. Interviews included questions about Project goals, how the current context (funding, policies, CSMP directives) influenced the Project, and how the Project has responded to changes in context in the past few years. Finally, we conducted an online survey of all site directors at the nine Projects during the fall of 2004. Although the UCOP reported 109 sites in existence when we administered the survey, over the course of the survey administration, we excluded one Arts Project site, all three Physical Education-Health Project sites because they were not funded in 2003-2004, and one Mathematics Project site because it did not offer summer programs or events for teachers during 2003-04. Out of 104 site directors, 100 completed the survey, for a 96% response rate. Appendix A includes detailed reporting of responses. The analysis also has a longitudinal component, in which we compared site director responses in 2004 to responses to surveys in 2001 and 2000 conducted as part of the previous evaluation of the CSMP (Quick et al., 2002). The 2004 survey contained a mix of closed- and open-response items. In this report, we sometimes quote open-response items
SRI Proprietary
4
to allow a site director’s own words to come through directly. In each of these instances, the response is one among similar statements made by several individuals. For a more detailed discussion of survey methods, please see Appendix A. REPORT ORGANIZATION The Year 1 report presents data that describe the work of the CSMP in 2003-04 in the context of the years since its last evaluation in 2001. The report consists of six chapters. Chapter 2 discusses important budget and policy changes that have affected the CSMP. Chapter 3 describes the scope and scale of the CSMP, including the partnership structures, participants served, and types of events offered. Chapter 4 describes the nature of CSMP’s work. Chapter 5 addresses the effectiveness of the CSMP by discussing findings from sites’ and Projects’ evaluations of their work. Chapter 6 summarizes report findings.
SRI Proprietary
5
SRI Proprietary
6
CHAPTER 2—BUDGET AND POLICY Throughout its 16-year existence, the CSMP has been affected by policy shifts and budget fluctuations. Both have challenged the Projects to be responsive to the changing needs of California’s teachers and students, as well as to the ebb and flow of state and external financial resources. This section first examines the impact of budget cuts on the CSMP and then considers how state and federal policies have affected the purpose, goals, and operation of the nine Projects. CSMP BUDGET: 1999-2000 TO 2003-04 Over the past few years, the CSMP, much like the state’s economy, has experienced a boomand-bust funding cycle. In the early 2000s, as part of a larger effort to improve teacher quality in California, the state dramatically increased CSMP allocations from $15 million in 1999-2000 to $35 million in 2000-01—the largest amount in the program’s history. Although the additional funding introduced new resources and created opportunities for the Projects to expand services, it also produced challenges as Projects quickly added more regional sites and began working with many more teachers within a short period of time. The Mathematics Project, for example, added 8 new sites to its roster, and existing sites increased their capacity (e.g., 1 site served an additional 6,000 teachers in 1 year, according to the executive director). UCOP funding to the sites proceeds through a formal RFP. For each funding cycle (which typically occurs every 3 years), UCOP issues an RFP that invites institutions of higher education and county offices of education across the state to become a CSMP site or to continue their work as a member of the CSMP network. Sites are approved for up to 3 years and undergo an annual review and rebudgeting process. The 2001 RFP required sites to offer institutes (each of which had to include a focus on content knowledge, an emphasis on ELLs, and discipline expertise from university academic departments), support subject-specific regional teacher networks, and evaluate site activities, with a particular focus on student achievement. In addition, UCOP expected that CSMP sites working with the same partner district would collaborate with each other to produce “a coherent, coordinated set of plans and program goals.” Each site approved in the 2001 RFP process was to receive $130,000 in core funding for the infrastructure required to operate essential programs and services. The state also provided supplementary funding to sites for two activities—district partnerships and Teacher Pathways (discussed below).
SRI Proprietary
7
UCOP, which wanted sites to move from a school partnership model to a district partnership model, developed a funding formula that favored the latter model.1 After submitting a partnership letter of intent, each site received district partnership funds based on the size of the district, the number or percent of low-performing schools, and the degree of service outlined in the partnership plan. The money was to be used to offset the overhead costs of initiating and managing a district partnership. The funds covered only 75% of the expected costs, however. The state expected the districts to pay the “lion’s share” of associated programming costs (i.e., fees required to produce activities), and the site would be responsible for helping its partners determine how to meet costs with internal and external resources (e.g., AB 466, MPDI, CPDI, ELD, Title I, etc.). Because sites received allocations for each partnership, they were able to offer partner districts and schools resources in return for entering into formal partnership agreements. Sites could also receive up to $40,000 for committing time and resources toward aligning teacher education programs along Teacher Pathways. These programs supported the development of teachers throughout their professional career, from their undergraduate years to retirement. Teacher Pathways’ funds supported site infrastructure for planning and maintaining agreements with teacher education entities on and off campus. Recent Decline in State Funding Before the Project leaders had fully accommodated their work to the increased funding levels, however, declining state revenues and policy-makers’ decisions sharply reduced CSMP funding over the next several years. As Exhibit 2-1 illustrates, after rising dramatically at the beginning of the decade, state funding for the CSMP declined precipitously, falling from $35 million to $5 million (the level at which it is expected to remain for 2004-05). A percentage of this decreased allocation was offset by an annual federal contribution of $4.4 million from NCLB (beginning in 2003-04 and continuing to 2004-05).
1
Sites could apply for funds up to the following amounts per eligible district: $3,500 for very small districts (fewer than 1,000 students); $10,000 for small districts (1,000–9,999 students); $35,000 for medium districts (10,000– 34,999 students); and $50,000 for large districts (35,000 students or more). Sites working with low-performing schools that were not in low-performing districts were eligible for half of the amounts listed above.
SRI Proprietary
8
Exhibit 2-1 State Funding for CSMP, 1999-2000 to 2003-04 40
Millions of Dollars
35 30 25 20 15 10 5 0 1999-2000
2000-01
2001-02
2002-03
2003-04
Years
Source: UCOP (2003, 2004b).
Some Projects received more state and federal funding than others, with a disproportionate share going to those subjects that were heavily weighted in state and federal accountability measures. The Mathematics, Writing, and Reading and Literature Projects received more total state and federal funds than the other Projects (except for the Science Project, which also received a separate NCLB grant), reflecting subject matter priorities in California. Exhibit 2-2 shows state and NCLB allocations (determined by UCOP) for all nine Projects in 2003-04. We note that the budget data presented in Exhibit 2-2 represent only direct funding from the state and one stream of federal money. As further detailed below, Projects and sites obtained operating funds from a variety of external sources (e.g., foundations, districts, federal programs). The diversity of these sources complicates any attempt to create a clear overall picture of the total amount of CSMP funding.
SRI Proprietary
9
Exhibit 2-2 Funding for Each CSMP: 2003-04
Project
Number of Sites (FY 2003-04)
Total NCLB (FY 2003-04)
Total State (FY 2003-04)
Total Overall (FY 2003-04)
CMP
22
$490,398
$1,109,602
$1,600,000
CSP
19
$825,397
$624,603
$1,450,000
CWP
17
$650,397
$449,603
$1,100,000
CRLP
14
$700,397
$249,603
$950,000
TCAP
7
$510,397
$339,603
$850,000
CH-SSP
8
$450,396
$299,604
$750,000
CFLP
9
$0
$700,000
$700,000
CISP
10
$400,397
$299,603
$700,000
3
NA
NA
NA
CPE-HP Source: UCOP (2004c).
The decline in funding since 2001-02 raised concerns among site leaders. As Exhibit 2-3 shows, the majority of site directors (87%) agreed that it was difficult to obtain enough funding to serve teachers in the way that they believed was most effective. Similarly, only 34% agreed that they had the staff needed to serve teachers, schools, and districts adequately.
SRI Proprietary
10
Exhibit 2-3 Percent of Sites Reporting “Strongly Agree” or “Agree” to Statements about Funding Issues All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
It was difficult to obtain enough funding to serve all of the teachers in the way that we believe is most effective.
87%
100%
88%
71%
86%
85%
95%
94%
50%
Our site was successful in securing enough money to reach its goals for 2003-04.
49%
29%
88%
43%
43%
46%
37%
50%
100%
The site director/s spent too much of her/his/their time writing grants.
51%
43%
50%
71%
55%
8%
68%
56%
40%
Our site had the staff needed to serve teachers/schools/districts.
34%
43%
25%
14%
29%
39%
32%
38%
67%
50%
0%
10%
17%
5%
13%
17%
Our site will have enough funding this coming year (2004-05) to serve 16% 43% teachers/schools/districts in the way that we believe is most effective. Source: SRI CSMP Site Director Survey (2004).
Although all sites reported that reduced funding strained their operations, some Projects felt less affected than others. The Arts Project expressed the most positive outlook on funding matters for 2003-04 (100% of all Arts Project sites reported that their site was successful at securing enough money to reach its goals in that year). Arts Project sites were also the most likely to report that their site had the staff needed to serve teachers, schools, and districts (67%), and the Arts Project had the lowest percentage (50%) of sites that indicated difficulty in obtaining the needed funding in 2003-04. In contrast, the Foreign Language Project expressed a more negative outlook on funding, with only 29% of sites reporting that they had enough money in 2003-04 to reach the site’s goals. And even among the Arts Project sites that felt positive about 2003-04 funding, site directors’ outlook for the future was pessimistic. Across all sites, only 16% of site directors agreed that they would have enough funding in 2004-05 to serve teachers, schools, and districts in the way that they believe is most effective. The International Studies Project and the Science Project had the lowest percentage of sites agreeing to that statement (0% and 5%, respectively).
SRI Proprietary
11
Impact of the Funding Decline The decline in funding allocations from the UCOP adversely affected the capacity and work of many CSMP sites. The reductions caused some sites to reduce their staffs to a skeleton fulltime crew or a few people working part-time. That change, in turn, reduced their ability to provide services to teachers and partner districts. “Reduced state funding has limited our ability to buy out teacher leaders' time to work with partnership districts… The reductions in funding have caused us to downsize our infrastructure,” a Writing Project site director noted. Although they had fewer resources at their disposal, sites were still expected to meet their goals and responsibilities. A Reading and Literature Project site director noted: We have less funding to recruit new leaders. The director is on a reduced schedule, and we only have an administrative assistant on regular assignment. With all the downsizing, we are still required to work with the same if not more districts, schools and participants. This decline in UCOP funding may have limited sites’ ability to offer and conduct highquality professional development. For those remaining staff members, many found themselves spending more time away from core tasks and dedicating more resources to keeping the site funded. “[The d]irector spends more time building relationships with districts and arranging contracted services and does not have as much time to present content during institutes,” another Reading and Literature Project site director indicated. Executive directors also reported spending more time providing technical assistance to site directors as they sought, applied for, and managed additional funds. Before the decline in funding, many Projects paid teachers stipends to encourage their participation in their professional development activities. The 2001 RFP, and previous RFPs, allowed the provision of stipends to participating teachers. For 2001, each Project received an average of 25 stipends of $1,500 per teacher ($100 per day per teacher) per site for invitational institutes. With the budget reductions, UCOP funding no longer supported previous levels of stipends. Their elimination had a negative impact on the participation levels at some sites because many teachers, accustomed to receiving payment for their attendance, balked at attending without compensation. A Science Project site director noted: We tried to charge a fee, and we could no longer give stipends. Our summer institutes in 2004 were canceled because of lack of participants. We decreased what we were going to pay [university] professors, also. We are now trying to hold our summer institute during Saturdays at no charge to teachers,...[but] teachers don't like attending Saturday workshops during the school year. As Exhibit 2-4 shows, 45% of CSMP sites strongly agreed or agreed that they had difficulty recruiting enough teachers to attend their site’s events or activities, (although reports varied greatly by Project). SRI Proprietary
12
Exhibit 2-4 Percent of Sites “Strongly Agreeing” or “Agreeing” that It Was Difficult to Recruit Teachers 100
Percentage
80 60 71
40 30
20 0
37
63
38
33 7
All
14
CFLP
0
0
CHSSP
CISP
38
31 10
CMP
Strongly Agree
16 0
CRLP
0 17
0
CSP
CWP
TCAP
Agree
Source: SRI CSMP Site Director Survey (2004).
Although the decrease in stipends was not the only possible explanation for increased difficulty in recruiting teachers, interview data suggested that it was an important factor. Executive directors noted that with the shift to partnerships with low-performing districts, the loss of stipends was especially detrimental. Teachers in low-performing districts are more likely than those in higher performing districts to be underprepared (and, therefore, concurrently enrolled in teacher preparation programs) and thus under greater stress from accountability policies. Without the stipends, some sites lost a key incentive for recruiting teacher participants from these districts. In addition, without the stipends, sites lost the leverage they had with teachers to have them complete outside assignments, like portfolios, or participate in evaluation activities. A Science Project site director stated: In the past we met with our participants monthly, but these get-togethers have been severely curtailed. We were able to provide stipends for teachers and were able to require they submit a portfolio of lessons and collaborative work at their school site for that stipend. This has been discontinued. Some sites found the loss of stipends to be particularly problematic and pursued outside funding sources that allowed them to offer teachers stipends (as well as other services).
SRI Proprietary
13
Seeking and Securing External Funding Projects pursued multiple alternative funding and operating strategies to compensate for the budget shortfalls. CSMP leadership at UCOP urged executive directors to demonstrate increased creativity and entrepreneurial savvy to offset the funding reductions. The executive directors, in turn, asked their site directors to aggressively pursue and leverage external sources to obtain additional funds. Although many Projects pursued external funding before the budget cuts, doing so and being successful at doing so assumed added urgency. “We feel extra pressure on the site to secure additional funding, which increases the work load and forces us to make do with less,” a History-Social Science Project site director explained. Sites indeed increased their efforts to secure external funding. As illustrated in Exhibit 2-5, the majority of sites (65%) reported writing more proposals for supplementary grants in 2003-04 than they did in 2001. The remaining sites reported never having written grants (12%), writing fewer grants than in 2001 (8%), or writing the same amount of grants as in 2001 (16%). Exhibit 2-5 Percent of Sites Reporting that They Wrote More Grants for Supplementary Funds in 2003-04 than They Did in 2001 100
Percentage
80 60 100
40
75
65
20
52
78
71
CSP
CWP
80
46
29
0 All
CFLP
CHSSP
CISP
CMP
CRLP
TCAP
Source: SRI CSMP Site Director Survey (2004).
Many variations existed among the Projects in terms of the availability of large external grants and sites’ success in attaining them. Recently instituted federal initiatives, the Teaching
SRI Proprietary
14
American History Grants Program (TAH)2 and the California Mathematics and Science Partnership (CaMSP)3, enabled the History-Social Science, International Studies, Mathematics, and Science Projects to access additional funds. “We have been fortunate to receive a Teaching American History grant which has helped us meet our site's goals and activities,” an International Studies site director noted: “Without this additional funding it would have been very difficult to reach our goals.” Although only local education agencies (LEAs) are eligible for these competitive grants, both of the programs require that applicants partner with organizations with content expertise and the four Projects were proactive in approaching and offering to partner with eligible districts and support them during the application process. The History-Social Science Project was the most successful of the four. All but 1 of the 10 History-Social Science sites had at least one TAH partnership (the one site that did not have one received another federal grant to improve the teaching of geography), and several had agreements with multiple districts. This success may be a function of the size of the TAH program ($150 million in 2004-05) and the relatively small number (8) of History-Social Science sites competing for grants. At least 2 International Studies sites also participated in the TAH program. The Mathematics and Science Projects were less successful in securing CaMSP grants. Although all 22 Mathematics Project sites sought to partner with LEAs for the grants, only 4 received them. The Science Project had a similar experience, but on a much smaller scale. The CaMSP is not as large as TAH ($20.3 million in 2004-05), and more Mathematics and Science Project sites thus competed for fewer dollars. Some of those sites, however, obtained other federal funds by partnering with a university to win California Postsecondary Education Commission (CPEC)4 grants, and the Science Project 2
3
4
In 2001, Congress established TAH to promote teaching American history, improve instruction in the subject, and provide high-quality professional development for teachers. The competitive grant program provided funds (averaging about $1 million per recipient) to LEAs to develop and operate 3-year professional development programs. As part of the program, grantees were required to partner with one or more institutions of higher education (IHEs), nonprofit history or humanities organizations, or libraries or museums. These entities were expected to provide content expertise and instructional support in American history to the grantees In 2002, Title II, Part B of NCLB, created CaMSP. This competitive grant program allows LEAs serving highneed student populations to partner with an engineering, mathematics, or science department of an IHE, with the goal of improving the content knowledge of teachers in those fields. Partnerships may also include a nonprofit or for-profit organization. Recipients of the grant funds must use the money for California state standards-based professional development of mathematics teachers in grades 5 through 9 and science teachers in grades 4 through 8. CPEC administers federal Teacher and Principal Quality Training and Recruitment Program funds to support teacher professional development in mathematics and science. CPEC gives competitive grants to IHEs to collaborate with LEAs to create teacher development initiatives that increase teachers’ pedagogical and content knowledge in the subject area, their awareness of changing assessment strategies, and their understanding of how best to serve a culturally diverse student population. The program targets teachers in schools with student populations that have low college attendance rates and low participation in mathematics and science college preparation.
SRI Proprietary
15
also received NCLB money. Because Projects and sites did not receive this money directly from the federal government (they contracted with the recipient LEA or IHE) and because the size of the partnership varied, the total amount the Projects received cannot be easily determined. Some Reading and Literature Project sites successfully tapped into Reading First funding by receiving contracts from LEAs to provide professional development.5 Other smaller external sources support the work of the Writing Project (National Writing Project) and the Foreign Language Project (Title III, Beginning Teacher Support and Assessment program (BTSA), and foreign governments). Although outside sources provided funding for CSMP sites, they often stipulated specific programmatic or evaluation requirements. These restrictions, to the degree that they did not correspond with the core goals of the Projects and sites, often proved burdensome to already stressed site staffs. “Having funding from outside sources adds directives and goals from these other sources and adds to the workload of the site,” one History-Social Science site director noted. “This means having less time/manpower to concentrate on the goals and activities that you think bring about the most good/change.” Another History-Social Science site director echoed those sentiments: “Because some of our funding comes from sources other than CSMP, we need to make sure that our programs meet the requirements of each of our funding sources.” For Projects with multiple funding streams, managing the different requirements can sap a site’s capacity to provide services to teachers. The additional overhead involved with researching different grant opportunities or building relationships with potential partner districts can also tax the resources of each site. “Our goals are much more modest because we have less money to work with and only so much time can be spent by the site director writing grants,” a HistorySocial Science site director explained. Providing technical assistance to districts around funding issues also detracted from the sites’ core mission. “We have also had to become familiar with school funding sources,” the same site director continued, “so that we can give suggestions to teachers and districts about how to pay for our programs.” Fee-for-Service Regardless of their ability to attract external funds, most sites in all nine Projects came to charge districts directly for their services; that is, they employed a “fee-for-service” model. For 5
Reading First is a national initiative aimed at enabling all students to become successful early readers. Funds are distributed among state education agencies through a competitive grant process, through which states then award subgrants to local education agencies, with priority given to districts with high rates of poverty and reading failure. Funds are provided to train teachers in the five essential components of reading (phonemic awareness, phonics, vocabulary development, reading fluency, and reading comprehension strategies), and to identify children who may be at risk of reading failure. States may also reserve up to 20% of their funds for professional development, technical assistance, and planning, administrative and reporting activities. California’s Reading First Program is based on California’s standards-based adopted reading instructional materials.
SRI Proprietary
16
some Projects, the approach was new, but others (the Writing, Arts, and Foreign Language Projects) have always operated this way. Fifty-eight percent of CSMP sites reported that they charged fee-for-service more in 2003-04 than they did in 2001 (Exhibit 2-6). Exhibit 2-6 Percent of Sites Reporting That They Charged Fee-for-Service More or the Same Amount in 2003-04 than They Did in 2001 0
100
8
80
40
24 28
60
67 12
57 100 38
40
83 67
58
20
53 29
60 33
25
0 All
CFLP
CH-SSP
CISP
More than in 2001
CMP
CRLP
CSP
CWP
TCAP
Same amount as in 2001
Source: SRI CSMP Site Director Survey (2004).
The fee-for-service model complicated some site directors’ ability to strategically plan activities and manage resources. Site directors operating in this model, for instance, could not predict how much professional development a particular district or school would request the next year, nor could they anticipate having funds sufficient to organize and implement summer institutes. “The changes in funding have affected our site in the following ways—we will have to cut back or not offer our summer institutes and follow-up programs,” a Science Project site director explained. Furthermore, with district funding levels also being reduced as a result of the state’s funding crisis, districts had less money to pay for CSMP professional development, especially in nonpriority subjects. “As school districts' budgets are cut, they are less and less willing to put money towards professional development in the arts,” an Arts Project site director reported. “Our work with school districts was mostly with those that had received Arts Work Grants, which were cut from the state budget this year.” The subject areas that were not prioritized in federal or state policies (foreign language, international studies, arts and physical education) were placed in jeopardy because the sites had few large grants for them to apply for and because it was difficult to convince districts to prioritize these subjects in tight budgets.
SRI Proprietary
17
Even the Arts Project, which operated with the fee-for-service model for several years, encountered challenges as a result of the new policy and funding changes. An Arts Project site director noted: Fee-for-service programs sustained the programs themselves and enough office support for those particular programs. However, maintaining enough funds to support site director travel for meetings and consultations, general office support, partnership work that doesn't involve fee-for-service, and R &D to create the new programs required by new priorities and funding was extremely difficult. Considering the state’s budget crisis, UCOP’s push for sites to adopt the fee-for-service model is understandable, but that strategy has negatively affected some CSMP sites and produced uneven results for others. CSMP Site Reductions: 2000-01 to 2003-04 In addition to scaling back staff, reducing stipends, and seeking external funding, several Projects were forced to cut sites in the wake of declining funding. The total number of sites decreased from 129 to 104 from 2000-01 to 2003-04, with most of the reduction taking place in 2002 and 2003. The number of sites eliminated varied by Project (Exhibit 2-7). The Foreign Language Project maintained all of its sites in this period, and the Science Project added one new site. On the other hand, the History-Social Science, International Studies, and Physical Education-Health Projects (UCOP cut the Physical Education-Health Project entirely in 2003-04) were most heavily affected, losing a combined total of 18 sites.
SRI Proprietary
18
Exhibit 2-7 Number of CSMP Sites: 2000-01 and 2003-04 Project CFLP
Number of Sites FY 2000-01
Number of Sites FY 2003-04
9
9
CH-SSP
17
8
CISP
16
10
CMP6
22
22
CRLP
16
14
CSP
18
19
CWP
18
17
TCAP
10
6
3
0
129
104
CPE-HP Total
Sources: Quick et al. (2002); UCOP (2004c).
Executive directors made decisions about which sites to close in collaboration with their advisory boards. Site tenure was not the determining factor in the elimination process; rather, key criteria included the financial sustainability and independence of the sites, as well as their ability to develop district partnerships; that is, how successful sites were at limiting their reliance on state funds and leveraging external funds. In addition, sites’ abilities to adapt to the requirements of the 2001 RFP were considered. Some veteran History-Social Science sites, for example, were closed because they were not comfortable with additional responsibilities associated with the literacy work. From 2002-03 to 2003-04, the Mathematics Project eliminated one site for not being interested in developing partnerships and serving few teachers, and dropped another because it did not focus on content. UCOP policy decisions about a perceived service and curricular overlap also affected the reduction of International Studies and HistorySocial Science sites. Restructuring of this magnitude placed additional stress on some of the remaining sites. Some site directors reported difficulty in conducting follow-up because of the greater geographical distances teachers had to travel to the fewer and now further dispersed regional sites. One History-Social Science site director noted: What’s difficult is keeping teachers connected. We are regional sites so some teachers travel great distances to attend and once you get back to the classroom, with all the things pulling at 6
CMP went from 22 sites in 2000-01 to 24 in 2001-02 before dropping back down to 22 sites in 2003-04.
SRI Proprietary
19
you, it can be a challenge to keep teachers energized and wanting to continue with the follow-up.
An Arts Project site director voiced similar concerns about the coverage area of her site doubling over the course of a year. Projects also lost key resources, knowledge, and leadership when sites closed. The most extreme example was the case of the Physical Education-Health Project. The Cost of Saving Money: The Case of the Physical Education-Health Project The experience of the Physical Education-Health Project can serve as a cautionary tale about what can happen when investment in the CSMP quickly and dramatically declines. Losing sites through attrition or elimination can have profound impacts on leadership, resources, and professional community. The Project started in 1994 and at one point had five established sites; that number was eventually reduced to three. It never had a substantial amount of funding. “We’re always amazed that we could run institutes on somewhere between $60,000 and $80,000 per site,” the executive director noted. “Our total Project budget was around $250,000. We didn’t have the capacity of others nor the time to build the capacity.” In 2002-03, the executive director resigned, and the Project received no state funding, effectively dismantling the Project. The sites continued their professional development activities using carryforward funds into 2003-04. When they exhausted that money, the sites closed, and the site leadership pursued other activities. During 2004-05, the interim executive director and the San Jose State University (SJSU) site director have been working to reestablish sites and have found the process to be daunting. “[It is one] of the most difficult things to do because you lose your teacher leaders and they get involved doing other things,” the SJSU site director explained. “This past year has been about trying to reinvolve our leadership and get the sites back and active. Beyond changing or improving sites, like the other Projects, we have to reactivate ours. Once the forward momentum has stopped, it’s hard to get it going again from the ground up.” Site leaders continued their work, but not within the structure of the CSMP. The executive directors reported that the site leaders were struggling with what to do because they had assumed other responsibilities. “They’re truly committed to CSMP model and professional development in PE and health, but they’re facing a moral and ethical dilemma [about] whether or not they can find enough energy in their group to continue. Two of them are that point.” This case illustrates how Projects and sites stand to lose a great deal of leadership capacity, professional community, and institutional knowledge when the state reduces or eliminates professional development funding.
Changes in UCOP’s financial support for the CSMP have substantially affected the Projects. The decline in state funding led sites to seek external funding from federal and state programs, and also in “fee-for-service” from the districts they served. Sites that were not successful at these efforts had to cut staff, or in the worst cases, close. Although the effect of budget cuts has
SRI Proprietary
20
been obvious, other policy changes also led to significant changes in the work of the CSMP. We turn to these next. IMPACT OF STATE AND FEDERAL ACCOUNTABILITY POLICIES ON THE CSMP Like budget decisions, federal and state policies regarding accountability have affected CSMP operations. In 1998, California adopted a set of standards-based accountability measures to help boost school performance and improve student achievement. The state standards are connected to a system that holds schools accountable for improving student performance. The core of California’s accountability policy is the Academic Performance Index (API), which is used to quantify school performance and improvement over time. Although the API is calculated slightly differently for different levels of schools, in general it weights students’ performance in reading/language arts and mathematics much more heavily than their performance in other subjects like science or social studies. The CSMP’s focus on state standards, reinforced by the 2001 RFP, had a significant impact on site goals and activities. When asked to what extent various factors influenced their site’s activities, more site directors answered that state standards and accountability affected their site to a large extent (84%) than any other option, including CSMP policies (69%) and NCLB requirements (40%) (Exhibit 2-8).7 Exhibit 2-8 Percent of Sites Reporting State, Federal, and CSMP Policies Influenced their Activities to a Large Extent All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
State standards and accountability measures
84%
71%
75%
100%
81%
93%
84%
81%
83%
CSMP policies and regulations
69%
71%
75%
75%
71%
64%
68%
63%
67%
NCLB requirements
40%
29%
50%
25%
14%
79%
53%
38%
33%
Source: SRI CSMP Site Director Survey (2004). 7
Note that even this strong response may underestimate the importance of state standards on CSMP. Several site directors commented that because the state has not yet developed standards in their areas, the question did not apply to them. Also note that although only 69% of site directors described CSMP policies as having a “large effect” on their sites, the higher importance rating given to standards could be an outcome of the 2001 RFP requiring standards-aligned professional development. Another possible interpretation is that as state funding declined and sites received a larger proportion of funds from other sources, CSMP policies played a smaller role in site directors’ decision-making.
SRI Proprietary
21
California’s accountability policies apply pressure on low-performing schools to improve student achievement in reading/language arts and mathematics more so than in other subjects. This emphasis had a sizable impact on sites covering subjects that the API does not address or addresses only minimally. Not only were districts less likely to pay for professional development in these areas, but (especially in the elementary grades) site directors reported that teachers were being told to focus their instructional time on tested subjects, to the detriment of the remaining subjects (e.g., science, art). NCLB and the CSMP The CSMP also felt the impact of recent federal legislation. NCLB, the 2001 reauthorization of the Elementary and Secondary Education Act, added a federal layer of accountability policies over California’s existing system. NCLB seeks to improve student achievement by setting minimum nationwide standards for teacher qualifications and levying federal sanctions on schools with low and stagnant performance. To be designated “highly qualified” under NCLB, teachers of core academic subjects must have: • A bachelor’s degree; • A state credential or be enrolled in a credential program (e.g., an intern program) for no more than 3 years; • Demonstrated subject matter competence. California teachers can demonstrate subject matter competence by passing an appropriate examination in their content area (e.g., the California Subject Examinations for Teachers [CSET]), having a major in their subject area (for single-subject teachers only), or by documenting their qualifications through the High Objective Uniform State Standard of Evaluation (HOUSSE). HOUSSE regulations are designed so that veteran teachers who did not major in a content area and did not have to pass a content area test to get their credential have an alternative strategy for establishing their qualifications. Specific HOUSSE regulations are set by each district, but generally include documentation of participation in a wide variety of activities (e.g., content-based professional development, serving as a department chair) that are presumed either to require or provide content area expertise. NCLB also requires states to develop a way of identifying schools and districts that fail to make AYP. In California, performance under AYP is calculated through a formula that includes API scores, assessment participation rates, graduation rates, and percent proficient in schoolwide and subgroup annual measurable objectives. In fall 2003, California released its first list of the 3,942 schools and 611 districts that failed to meet AYP.8 As more schools and districts are 8
This fall, the state released its 2004 list, under which more schools and districts failed to meet AYP. Because this
SRI Proprietary
22
identified as failing to meet AYP over time, the CSMP expects to provide increased levels of technical assistance and support to help them improve. AYP provisions are in the early stages of implementation, however; thus, although some site directors reported helping districts conduct assessments of their district’s needs under NCLB, AYP provisions had not yet had a large effect on the CSMP. Although NCLB added additional requirements for the state, it also partially offset the loss of some professional development money. For 2003-04, the CSMP was given a $4.4 million allocation in NCLB Title II funds (actually received in January 2005) and will be given the same allocation for 2004-05; the Science Project also received an additional allocation from NCLB in 2002-03, although the Science Project did not receive the funds until June 2004. Such assistance does not come without restrictions or requirements on the type of professional development that the CSMP should provide. By NCLB definitions, effective professional development activities: • Improve teachers’ content knowledge and help them become “highly qualified” • Are integrated into school and district improvement plans • Provide participants with the knowledge and skills to ensure students meet standards • Are sustained, intensive, and classroom-focused (and are, therefore, not 1-day or shortterm workshops) • Advance teachers’ understanding of effective instructional strategies • Are aligned with state standards and assessments • Provide teachers with the knowledge and skills necessary to provide appropriate instruction to ELLs. (U.S. Department of Education, 2004) Projects like the Writing Project translated these guidelines into criteria for their sites to provide professional development under NCLB. As is apparent, the federal government’s expectations for professional development complemented the work the CSMP has been undertaking for several years, including providing content-rich professional development, supplying technical assistance to high-need districts, and partnering with low-performing districts. Accordingly, although NCLB requirements may have required CSMP sites to modify the focus of services they offered, the requirements did not force the sites to drastically change most of their offerings. For the most part, according to our site survey (see Exhibit 2-8 above), NCLB’s impact appears to have been minimal compared with other factors: only 40% of site directors reported that NCLB requirements influenced their site to a “large” extent. This opinion probably stemmed from the CSMP’s longstanding approach of offering standards-based, content-rich professional development. “All of our programs have been standards-aligned for many years,” a list was released too late to affect the CSMP during the 2003-04 school year, we do not discuss it in this report.
SRI Proprietary
23
History-Social Science site director explained. “Because of this, we haven't had to rethink our curriculum significantly, even given our new NCLB mandates.” Although NCLB did not lead to major changes in the CSMP, the federal law altered some site activities. Many sites provided professional development and training programs for teachers who needed to achieve “highly qualified” status. Some sites offered CSET test preparation for teachers who had not yet received a preliminary credential in their content area. The more common impact was that veteran teachers asked sites for documentation of their CSMP participation as part of their evidence of subject matter competence under HOUSSE. These activities became an increased focus of the CSMP. As Exhibit 2-9 illustrates, more than 60% of sites in each Project reported giving more attention than they did in 2001 (the first year of NCLB) to the need to make teachers “highly qualified.” Exhibit 2-9 Percent of Sites Reporting Working with Schools/Districts to Ensure Teachers Are Highly Qualified More than They Did in 2001 100
Percentage
80
60
40
77
86
92
88
81
75
83
63
63
20
0 All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP TCAP
Source: SRI CSMP Site Director Survey (2004).
As the number of underprepared teachers in the state continues to decline, however, the amount of CSET test preparation provided by the CSMP should decrease. Likewise, because veteran teachers need to participate in HOUSSE only once, this component should become less of a factor over time as well. AYP provisions have not been in place long enough to determine their effect on the CSMP, but they could have a longer term impact. As a result, it is not clear to what degree NCLB will influence the CSMP in the coming years.
SRI Proprietary
24
CONCLUSION The shifting policy and budgetary context at both the state and federal levels had an uneven impact on the CSMP. On the one hand, the Projects remained focused on standards-based professional development for teachers—their core mission for more than a decade. UCOP’s RFP in 2001, calling for professional development to be standards-based, and the federal call for support for high-quality professional development did not result in shifts in the Projects’ central focus. On the other hand, drastic cuts in state funding forced the CSMP to close sites and required basic shifts in the business model used by the remaining sites. Chapter 3 explores the scope and scale of the CSMP’s work in the wake of these changes.
SRI Proprietary
25
SRI Proprietary
26
CHAPTER 3—SCOPE AND SCALE OF THE CSMP Since its inception, the CSMP has sought to improve teachers’ content knowledge and understanding of how to teach that content to students. Over time, as discussed in Chapter 2, changes in state and federal policy modified the Projects’ official mandates. Similarly, funding from other sources imposed additional requirements on CSMP sites. These same shifts reduced or expanded the Projects’ capacity to reach participants. In this chapter, we assess the impact of these changes, asking whether these policy and budget shifts altered (1) the central goals of the Projects, (2) the participants targeted and served, and (3) the types of events and activities supported. We have based our findings heavily on our survey of the site directors and on UCOP’s OIS. Developed and maintained by UCOP since 2000-01, the OIS contains variables designed to enable users to quantify the work of the CSMP in terms of partnerships, participants, and events. As such, the OIS represents a major effort to provide a level of data on program outputs that is uncommon for programs as large and complex as the CSMP. Unfortunately, the database is missing a substantial proportion of data. In some cases, data are missing for participants contained in the database, raising serious questions about how to interpret the frequencies we report. To address this issue, we always report the amount of data missing for each variable to indicate how likely it is that the data contained in the database portray an accurate picture of the participants.9 A second problem is that an unknown number of participants are missing from the database altogether. This problem appears sufficiently severe for 2000-01 that we do not report data for that year. Although interpreting OIS data is difficult for these reasons, the system nonetheless is the best source available for quantifying CSMP work. Given the unevenness of data in the OIS, we attempt, wherever possible, to use the site director survey to triangulate statements about CSMP work. CSMP GOALS According to site directors, the core CSMP goals have not changed. Site directors consistently identified improving teachers’ content knowledge and their ability to teach that content as one of their top three priorities (see Exhibit 3-1).
9
Two things need to be taken into account in determining when the proportion of missing data is high enough to warrant concern in interpretation.: (1) whether the participants with missing data may differ from those for which complete data exist (e.g., teachers who lack appropriate credentials may be less likely than those who are fully qualified to complete credentialing information); and (2) the distribution of known responses when some sizable portion is missing.
SRI Proprietary
27
Exhibit 3-1 Percentage of Sites Reporting the Following as Among their Top Three Priorities Developing teacher understanding of how to teach the subject matter
73
69
Deepening teacher content knowledge Providing teachers with strategies for teaching content to ELLs
45
Developing teacher understanding of content standards in the subject
42
35
Helping teachers teach literacy in their subject area 28
Developing teacher leadership skills Preparing teachers to help students pass state assessments
8
Helping teachers meet the definition of highly qualified as specified in No Child Left Behind
5
0
20
40
60
80
Percentage
Source: SRI CSMP Site Director Survey (2004).
Several site directors noted that choosing among priorities was difficult, and that many of the goals listed were not mutually exclusive and could be pursued simultaneously. One noted that “an obvious example includes our highest priority—enhance content knowledge. This also directly helps teachers meet . . . NCLB requirements [for highly qualified].” Looking across Projects, improving teachers’ knowledge of content and how to teach it consistently ranked high, but there was some variation. The percent of sites reporting that developing teacher understanding of how to teach the subject matter was one of their top three priorities ranged from 43% (in the History-Social Science Project) to 95% in the Mathematics Project. The range was even greater on deepening teacher content knowledge, with only 29% and 33% (respectively) of the Foreign Language and Reading and Literature Projects’ site directors reporting content knowledge as one of their top three priorities, compared with 95% of the site directors of the Mathematics Project. Variations across Projects were much less prominent for the bottom two priorities, suggesting that few sites in any Project prioritized student test scores and helping teachers become “highly qualified” under NCLB, possibly because both of those goals will be achieved through work in the other six areas. (For a complete list of priorities by Project, see the technical appendix (Hough, et al., 2005). Given this set of core goals and the dynamic policy pressures on CSMP, we turn now to those whom the Projects serve and the scale of Project work.
SRI Proprietary
28
WHO IS SERVED BY THE CSMP? The CSMP partners with schools and districts to reach teachers, their primary targets. We discuss partnerships and participants separately. Partnerships In order to develop more systematic approaches to improving student achievement, the 2001 RFP called for sites to move from a school partnership model to a district partnership model. According to the RFP, the objective of this new model was for “the partnership to plan and implement an instructional program that brings all students up to the state’s high academic standards.” Professional development activities produced by the sites were to be developed with consideration for all of the other factors affecting the district, including induction, administrative leadership development, and staffing concerns. All planning, the RFP explained, should also involve the district and external providers with the goal of reaching “at least all of the teachers in all low-performing schools in the district (including new teachers each year) over 3 to 5 years.” Many sites formed partnerships with districts in which they had existing partnerships with schools. UCOP encouraged sites to draw on the lessons they learned through their school partnerships as they expanded to district partnerships. The OIS includes data on both school and district partnerships. Exhibit 3-2 shows that the number of partnerships maintained by the CSMP rose slightly from 2001-02 to 2002-03 and then dropped substantially from 2002-03 to 2003-04. Exhibit 3-2 Number of Partnerships by Project, 2001-02 to 2003-04 All
CFLP
CHSSP
CISP
CMP
CPEHP
2001-02
515
15
25
46
74
0
2002-03
573
16
26
42
84
2003-04
385
13
27
21
62
CRLP
CSP
CWP
TCAP
97
43
170
45
0
102
72
175
56
0
75
34
120
33
Source: OIS; SRI analysis.
Again, we find variations across Projects. The OIS reports that the Foreign Language and History-Social Science Projects had a relatively stable number of partnerships throughout this period. The trend for Mathematics, Reading and Literature, Science, and Writing Projects was similar to that for the CSMP as a whole, with an initial increase, followed by a sharp decrease in the number of partnerships. The International Studies had a decline in the number of partnerships each year.
SRI Proprietary
29
Site directors reported that they developed more partnerships in 2003-04 than in 2001, although the reports varied greatly by Project (Exhibit 3-3). In 2003-04, 47% of sites reported developing more partnerships than in 2001. Project reports ranged from 25% (International Studies) to 75% (History-Social Science). Of the sites that did not report developing more partnerships in 2003-04, 41% reported the same amount of partnership work in 2001 as in 2003-04. Exhibit 3-3 Percent of Sites Reporting Whether They Developed Partnerships More or the Same Amount in 2003-04 Than They Did in 2001 100 25
80
50
Percentage
41
63
40
20
25
50
53
56
CRLP
CSP
CWP
43
60
50 32
48
75 47
43 25
50
33
0 All
CFLP
CH-SSP
CISP
More than in 2001
CMP
TCAP
Same amount as in 2001
Source: SRI CSMP Site Director Survey (2004).
These data suggest that site leaders put as much or more effort into developing partnerships in 2003-04 than they did in 2001, even though the OIS indicates that the overall number of partnerships did not continue to rise. This interpretation is consistent with site directors’ reports that developing partnerships became more challenging when they no longer received funding to partially support the costs of their partnership work (i.e., they may have spent more time but developed fewer partnerships).10 Unfortunately, the available data do not enable us to draw conclusions about whether or not the extent of CSMP partnership work changed in recent years. Participants The decline in state funding over the 2001-02 to 2003-04 period raises the question of whether or not the CSMP is reaching fewer teachers now than when funding levels were higher. Exhibit 3-4 shows that the number of participants (those considered “participants” are discussed 10
While OIS data may be incomplete, it is also important to note that site directors’ perceptions of their work (especially their work several years ago) may not have been completely accurate.
SRI Proprietary
30
below) served by the CSMP declined steadily from about 57,000 participants in 2001-02 to about 46,000 participants in 2003-04. Exhibit 3-4 11 Participants Served by CSMP, 2001-02 to 2003-04 All12
CFLP
CHSSP
CISP
CMP
CPEHP
CRLP
CSP
CWP
2001-02
57,089
1,199
4,628
3,500
9,919
210
13,420
4,517
15,272
4,424
2002-03
52,825
1,336
3,501
2,510
11,635
26
7,634
5,037
16,569
4,577
2003-04
46,037
1,185
2,353
2,532
8,450
0
8,336
3,905
15,397
3,879
TCAP
Source: OIS; SRI analysis.
The one exception to this trend is the Writing Project, which reported serving a relatively constant number of teachers over the 3-year time span. If these data present an accurate picture of the CSMP, they offer support for the hypothesis that the decline in state funding led to a reduction in the number of participants the CSMP serves. CSMP participants span a wide range of education stakeholders—from teachers and administrators to college students, K-12 students, and parents. Data on participant types are represented in OIS, but a large proportion of these data are missing. For 2003-04, for example, participant type data are missing for more than half of all participants (55%).13 Exhibit 3-5 shows the type of participants the CSMP served from 2001-02 to 2003-04.
11
OIS includes data on many types of people, including attendees, presenters, and facilitators. Participant counts include only those identified as attendees. In addition, OIS can track anyone who is interested in participating but may not attend a professional development event. OIS numbers also include participants in fee-for-service grants, such as MPDI. SRI participant counts include only those who attended an event. 12 The total participant count is the sum of the participants in each Project. By definition, therefore, an individual is counted more than once in the total if the individual participated in more than one Project each year. 13 All data in exhibits that follow that describe participants’ characteristics are affected by the fact that 55% of participant type data is missing. The missing data likely leads to substantial underreporting of the number of teachers who participate in CSMP activities and is a source of potential biases in describing the characteristics of teachers served by the CSMP.
SRI Proprietary
31
Exhibit 3-5 Participants by Type, 2000-01 to 2003-0414
2001-02 2002-03
Missing Data 17,051 (32%) 22,665 (47%)
23,380 (55%) Source: OIS; SRI analysis. 2003-04
Administrators 806 (1%)
College Students 818 (2%)
Other 2,742 (5%)
21,839 (45%)
428 (1%)
535 (1%)
2,293 (5%)
47,757
16,910 (40%)
471 (1%)
480 (1%)
1,539 (4%)
42,780
Teachers 31,406 (59%)
All15 52,823
OIS data show that from 2001-02 to 2003-04 the number of teachers served by the CSMP and the percentages of participants who were teachers declined. However, without complete data, we could not ascertain whether the change in the proportion of participants who were teachers reflected a change in whom the CSMP was serving or simply was an artifact of incomplete reporting. The following analyses of teacher characteristics exclude all records with data missing on participant type (a total of 66,633 records from 2001-02 to 2003-04). Teachers’ Years of Experience The CSMP have always sought to reach teachers who are at various stages in their teaching careers. Although no specific targets exist for the percentage of novice or veteran teachers, the CSMP can best meet its goals if years of experience vary for participating teachers. The OIS data indicate that the CSMP served a disproportionate number of novice teachers (less than 2 years experience), perhaps due to their commitment to working with low-performing schools and teachers lacking full certification. CSMP work with more veteran teachers is relatively evenly distributed across the following categories: 3 to 5 years, 6 to 10 years, and more than 10 years.16
14
Teachers include those identified as a teacher or intern. Administrators include those identified as an administrator, principal, or vice principal. College students include those identified as a college undergraduate or a preservice teacher. “Other” includes those identified as a counselor, curriculum specialist, instructional aide, professor, or substitute teacher. SRI recoded some participants identified as “other” in appropriate categories if the information provided suggested that these participants fit into one of the predefined categories. Counts do not include those who reported a total of 0 contact hours in a given year. 15 Totals do not match those in Exhibit 3-4 because participants shown in that exhibit (and all subsequent exhibits) were counted only once, regardless of how many Projects they participated in. 16 Recall that from about one-third to more than half (see Exhibit 3-5) of OIS participants have no information provided about their role (e.g., whether they are teachers, administrators, preservice teachers, parents); thus, any analysis on “teachers” necessarily excludes a large percentages of CSMP participants. In addition, information on years of experience was missing for a small percentage of participants identified as teachers —13% of teachers in 2001-02, 8% in 2002-03, and 7% in 2003-04.
SRI Proprietary
32
Exhibit 3-6 Teachers by Years of Teaching Experience, 2001-02 to 2003-04 Missing Data 4,113 (13%)
0 to 2 Years 9,209 (29%)
3 to 5 Years 4,871 (16%)
6 to 10 Years 4,653 (15%)
More than 10 Years 8,560 (27%)
1,724 (8%)
7,063 (32%)
3,584 (16%)
3,579 (16%)
5,886 (27%)
21,836
1,201 (7%) Source: OIS; SRI analysis.
5,968 (35%)
2,537 (15%)
2,721 (16%)
4,483 (27%)
16,910
2001-02 2002-03 2003-04
All 31,406
Teacher Leaders Since their inception, a key strategy of the CSMP has been to prepare experienced teachers to become teacher leaders in their content area. The CSMP believe that teacher leaders are in an excellent position to teach their peers, both through CSMP activities and informal roles they play in their schools and districts. Comparisons of site survey data from 2000-01 and 2003-04 suggest that the number of teacher leaders has remained relatively constant (Exhibit 3-7). Exhibit 3-7 Number of Teacher Leaders Actively Involved in Site Activities in 2000-01 versus 2003-04 25 or fewer*
Between 26-50
Between 51-75*
More than 75
2000-01
63%
19%
6%
12%
2003-04
68%
17%
4%
12%
Sources: AIR Site Director Survey (2001); SRI CSMP Site Director Survey (2004); SRI analysis. *p < 0.05
The overall trend of stability masks variations within Projects. Some site directors reported that their teacher leader work was affected by the budget cuts. “We have decreased the number of opportunities for teacher leader activities due to the cuts in funding available for stipends,” an International Studies site director reported. “We have been required to use outside grants (other than CSMP) to develop teacher leadership opportunities.” Another site director stated it more bluntly: “Less funding results in fewer teacher leaders.” Funding cuts have also hurt recruitment for, and development of, teacher leaders. Some site directors indicated that seasoned teachers have instead turned to their district offices to perform similar roles (e.g., peer coaching, leading professional development activities) that pay more. “Without base funding, we have not been able to recruit additional teacher leaders,” a Reading and Language site director stated. “As our
SRI Proprietary
33
leaders take on leadership responsibilities at their schools or districts, they are not as available to the CSMP and we have not been able to develop new leaders.” Some of this variation becomes apparent by comparing the median number of teacher leaders involved in each Project (Exhibit 3-8). The median number of teacher leaders involved per site in the Writing and Arts Projects in 2003-04 was 40 and 35, respectively; in all other Projects, the median number of teacher leaders was 18 or fewer. In addition, some Projects, such as the Reading and Literature and Science Projects, showed significant declines in the number of teacher leaders involved since 2000-01 (median decline of 16 and 13 respectively). Exhibit 3-8 Median Number of Teacher Leaders Actively Involved in Site Activities in 2000-01 versus 2003-04 CHAll CFLP SSP CISP CMP* CRLP* CSP* CWP* 2000-01 17 8 9 8 13 34 23 43 2003-04 15 9 10 11 12 18 9 40 Sources: AIR Site Director Survey (2001); SRI CSMP Site Director Survey (2004); SRI analysis. *p < 0.05
TCAP* 24 35
Teachers of ELLs The 2001 RFP specified that sites should emphasize working with teachers of ELLs. Exhibit 3-9 shows that at least 39% of the teachers participating in CSMP activities each year from 200102 to 2003-04 reported teaching ELLs. Exhibit 3-9 Teachers of ELLs, 2001-02 to 2003-04
Missing Data 11,117 (35%)
Number of Teachers who Teach ELLs 12,602 (40%)
Number of Teachers who do not Teach ELLs 7,687 (25%)
7,652 (35%)
9,052 (42%)
5,132 (24%)
21,836
7,353 (44%) Source: OIS; SRI analysis.
6,664 (39%)
2,893 (17%)
16,910
2001-02 2002-03 2003-04
All Teachers 31,406
Like the data on the type of participants served, the percentage of teachers with missing data that would indicate whether or not they taught ELLs increased from 2002-03 to 2003-04. The increase in missing data over this period precludes any attempt to make a definitive statement
SRI Proprietary
34
about whether or not the CSMP served a greater or lesser proportion of teachers of ELLs in 2003-04 than in 2001-02. Teachers Needing to Become “Highly Qualified” Given NCLB funding, one of the CSMP’s roles is to serve teachers who need to become “highly qualified.” Because the OIS is not set up to measure which teachers meet the federal and state definitions of “highly qualified,” we could not estimate the number and percentage of CSMP participants who did or did not meet that standard.17 Specifically, the OIS does not track subject matter competency (one of the standards) or whether a teacher is working “out-of field.” The OIS does track data on the type of credential teachers hold and also the subject area(s) in which they hold credential(s). Teachers are not “highly qualified” if they are teaching on an emergency permit or waiver. Exhibit 3-10 presents a breakdown of credential term for teachers who reported holding a multiple- or single-subject credential for 2003-04. Unfortunately, 40% of teacher participants were missing information on their credential type.18 Exhibit 3-10 Multiple-Subject and Single-Subject Credentials by Credential Term, 2003-04 Multiple-Subject Teachers 11% (808)
Single-Subject Teachers 23% (741)
Emergency or waiver
2% (141)
5% (172)
Intern
2% (146)
2% (79)
Preliminary
13% (884)
10% (328)
Professional clear
72% (5082)
59% (1909)
Missing data
Source: OIS; SRI analysis.
The majority of teachers who hold a multiple- or single-subject credential report having a preliminary or professional clear credential. The OIS reports that only 5% of teachers with single-subject credentials and 2% of teachers with multiple-subject credentials hold an emergency permit or waiver and thus fail to meet NCLB standards for “highly qualified” based
17 18
Because the federal and state definitions are relatively new, this should not be considered a flaw in the database. Because teachers can hold multiple credentials, Exhibit 3-11 probably counts some teachers more than once. For example, a teacher could hold a multiple-subject professional clear while being an intern in a single-subject program.
SRI Proprietary
35
on their credential. Exhibit 3-10 may underestimate the number and proportion of CSMP participants who are not “highly qualified,” but this possibility cannot be confirmed. Survey Data on the Characteristics of Teachers Participating in CSMP Our survey asked site directors to estimate a series of characteristics (e.g., whether the teachers were from a low-performing school) of the teachers who participated in their events. These estimates are problematic because many of these characteristics are difficult to observe. Nonetheless, these estimates could help address some of the gaps caused by data missing from the OIS. Exhibit 3-11 shows site directors’ estimates of the characteristics of the teachers they served during 2003-04. Exhibit 3-11 Mean Percentage of Site Reports of Teacher Participant Characteristics All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
Teachers at lowperforming schools
55%
34%
60%
43%
54%
66%
60%
55%
42%
Teachers of ELLs
64%
53%
70%
50%
64%
83%
73%
52%
35%
Teachers lacking at least a preliminary credential in the subject they teach
15%
11%
16%
15%
28%
14%
11%
9%
8%
19%
40%
34%
59%
39%
19%
15%
Teachers who need to demonstrate subject matter competency 31% 20% through HOUSSE (to become "highly qualified" under NCLB) Source: SRI CSMP Site Director Survey (2004).
Exhibit 3-11 shows that while site directors estimated that the majority of teachers they served taught in low-performing schools, the range across Projects was substantial. Site directors in the Mathematics, Writing, History-Social Science, Science, and Reading and Literature Projects all estimated that the majority of participating teachers taught in lowperforming schools. The sharpest contrast was the Foreign Language Project, where site directors estimated that only about a third of their teachers taught in low-performing schools. All Projects’ estimates for the percentage of teachers they served who taught ELLs were higher than the OIS reports (Exhibits 3-9 and 3-11). Estimates of the percent of teachers who taught ELLs varied from a low of 35% for the Arts Project to a high of 83% in the Reading and Literature Project.
SRI Proprietary
36
Site directors also estimated that an average of 15% of the teachers they served lack at least a preliminary credential in the subject they taught, with a high of 28% for the Mathematics Project. These estimates were also much higher than the percentages reported by the OIS, but it is impossible to reconcile the differences. Finally, site directors estimated that 31% of the teachers they served were working to demonstrate subject matter competence under HOUSSE. As mentioned above, the OIS does not contain comparable data. Because many of these teachers might ask the site directors to verify their CSMP participation as part of the HOUSSE Project, this estimate is likely to be more accurate than some of the others, but it cannot be independently confirmed. EVENTS The CSMP offer many types of professional development activities. The OIS allows site directors to enter their activities into the database in any one of 19 categories, including a category called “other.” For this analysis, we combined somewhat similar types of offerings into six categories: institute, follow-up, series, coaching, inservice, and other.19 Exhibit 3-12 shows the median duration of each type of event from 2001-02 through 2003-04. The data show that institutes tended to be substantially longer than other activities. However, the length of the typical institute declined dramatically in 2003-04 compared with previous years. Research on professional development has shown that the duration of professional development activities is a determinant of their ability to effect changes in teacher practice, suggesting that institutes were less likely to be effective in 2003-04 than in previous years (Birman et al., 2000; Garet et al., 1999). All other forms of professional development had a relatively constant median length over this time period.
19
Some categories combine similar activity types. Institute includes the OIS categories of invitational institute, open institute, and mini-institute; series includes the categories of partnership series and workshop series; inservice includes the categories of partnership inservice, inservice, workshop, and conference; “other” includes the categories of academy, action research, conference, committee or planning meeting, partnership meeting, other, retreat, or study group. Note that because the OIS does not require sites to use consistent definitions in categorizing activities in the OIS, what one site director may define as a conference, another may define as a retreat.
SRI Proprietary
37
Exhibit 3-12 Median Length of Professional Development Activities in Hours, 2001-02 to 2003-04 Institute
Follow-up
Series
Coaching
Inservice
Other
2001-02
40.0
7.5
3.5
2.0
5.0
4.0
2002-03
40.0
7.5
5.0
6.0
5.0
4.0
2003-04
20.0
6.5
4.0
3.0
6.0
4.0
Source: OIS; SRI analysis.
In addition to a drop in the median duration of CSMP institutes, the total time teachers spend participating in all CSMP activities decreased. Exhibit 3-13 shows the decline in the number of “contact hours” (i.e., the number of hours each teacher participated in the CSMP) from 2001-02 to 2003-04. Exhibit 3-13 Number of Contact Hours per Teacher, 2001-02 to 2003-04 1 to 39 Hours 13,508 (43%)
40 to 79 Hours 11,039 (35%)
80 or More Hours 6,859 (22%)
2002-03
10,578 (48%)
5,697 (26%)
5,561 (25%)
21,836
2003-04
10,967 (65%)
2,744 (16%)
3,199 (19%)
16,910
2001-02
All Teachers 31,406
Source: OIS; SRI analysis.
The 2001 RFP set a benchmark of 40 hours of annual participation for 200 participants, but sites increasingly fell short of that target. Exhibit 3-13 shows that the percentage of teachers who participated in the CSMP for 39 or fewer hours increased each year from 2001-02 to 200304. The trend for teachers participating 40-79 hours was the reverse, falling from more than 11,000 teachers (35%) in 2001-02 to fewer than 3,000 teachers (16%) in 2003-04. No trend is clear for the percentage of teachers participating 80 or more hours in the CSMP over that period; however, because of the overall decline in CSMP participation, the number of teachers participating for 80 hours or more declined by more than 50%. The number of teachers participating in institutes and related follow-up activities also declined dramatically from 2001-02 to 2003-04, although the number of most other types of activities offered each year was relatively stable (see Exhibit 3-14).
SRI Proprietary
38
Exhibit 3-14 Number of Teachers Who Participated in Various Event Types, 2001-02 to 2003-04 Institute
Follow-up
Series
Coaching
Inservice
Other
2001-02
18,071
12,054
12,520
509
13,147
8,569
2002-03
7,938
2,797
11,711
355
10,924
4,691
2003-04
5,657
2,145
11,186
527
13,956
6,252
Source: OIS; SRI analysis.
Research on effective professional development suggests that institutes are likely to be among the more effective strategies, especially when coupled with follow-up during the school year (Birman et al., 2000; Garet et al., 1999). Coaching, another form of professional development that research suggests is likely to lead to improved teacher practice, did not see a decline in frequency like institutes and follow-ups; however, the CSMP uses this strategy infrequently compared to other strategies. CONCLUSION A substantial portion of data in the OIS is missing, limiting the system’s usefulness in producing conclusive findings. For example, because of missing data, it is impossible to know the extent to which the CSMP met the needs of teachers of ELLs or teachers working to become “highly qualified.” If the OIS data are accurate, however, the period from 2001-02 to 2003-04 was one in which the scope and scale of the CSMP decreased. The OIS reports declining numbers of partnerships, participants served, and events offered. Both the drop in the number of teachers participating in institutes and the decline in the number of teachers participating in CSMP events for more than 40 hours a year are troubling, given research that suggests that the length of professional development activities in which teachers participate (all other things being equal) is predictive of improvements in teacher practice. Chapter 4 examines whether the work of the CSMP has changed qualitatively over this time period.
SRI Proprietary
39
SRI Proprietary
40
CHAPTER 4—NATURE OF THE WORK The CSMP consist of a vast and varied network of professional development providers. All CSMP share certain features, including: participating in the CSMP network, promoting teacher leadership, offering particular formats for professional development (e.g., institutes and coaching), providing content-based professional development, and responding to the state’s need to improve teachers’ skills in working with ELLs and teaching literacy across the content areas. This chapter looks at each of these common features in turn, noting variations across Projects in these areas, describing the quality of professional development the CSMP offered, and discussing whether the opportunities offered by the CSMP changed since 2001-02. As the chapter shows, however, the great variations within the CSMP make it impossible to make broad generalizations about effectiveness and change over time. CSMP AS A NETWORK The CSMP are, by definition, a network of professional development providers. As one executive director noted, the role of the network is to support coherent professional development at the statewide level so that individual sites do not just offer “random acts of professional development.” In our site survey and interviews with Project leaders, we found a number of benefits of the network structure. First, it was an effective policy tool for the state: through the RFP process established for the network, the state influenced decisions and actions at the local site level. Earlier, we showed that site directors reported that CSMP policies heavily influence site goals (developing teachers’ content knowledge and skills in teaching that content). The second benefit of the network is that it helped build shared capacity to develop and provide high-quality professional development within each subject and across subjects in the same region. Some of the capacity the CSMP built was created in the Projects’ statewide offices. In the Reading and Literature Project, for example, the statewide office works to develop materials that can then be implemented throughout the state. The most well-known example of the Reading and Literature Project’s work in this area was the RESULTS model for professional development around assessment and instruction in literacy.20 More recently, the Reading and Literature Project developed Secondary Academic Language Tools (SALT), which provides secondary teachers in various content areas with tools to diagnose student skills in reading and writing, and corresponding strategies for helping students improve literacy skills. The Reading and Literature Project created instruments like SALT, and trained leadership teams from sites 20
RESULTS was part of the 1999 Governor’s Elementary Reading Initiative, which was funded through the California Professional Development Institutes.
SRI Proprietary
41
across the state in the use of SALT and in how to train teachers to use SALT. The network structure thereby facilitated both the creation and diffusion of tools important to key state goals. The network also facilitated sharing of ideas across sites within each subject area. Projects routinely convened meetings for all site directors. These meetings served to disseminate information about key issues in the CSMP, discuss ideas for fundraising, and share strategies for effective professional development and instruction. One Science Project site, for example, had success in developing teachers’ skills at using a “notebook” strategy for having students record scientific observations. While students learn scientific content and processes through recording data and interpretations in their notebooks, they simultaneously build their literacy skills. This site found the notebook strategy to be highly effective in improving the science, literacy, and mathematics skills of students, many of whom were ELLs. The success this site experienced led the statewide office to facilitate a miniconference around the notebook method, which was then adopted by several other sites. Such cross-pollination was facilitated by the formal and informal relationships across the sites, more so than it would have been without the network. This collaboration can be enhanced when Projects located in the same region work together. In fact, the 2001 RFP called for sites across Projects that are located in the same region to collaborate when possible to design a cohesive professional development plan that meets the needs of partner districts. There is ample evidence that this type of collaboration occurred. For example, the multiple Projects sponsored by the University of California, Los Angeles worked together in their partnership with the local district, providing training in literacy through various subject matter areas. Although the main focus of the collaboration was literacy, the Projects also addressed other district needs, including classroom management, training of new teachers, lesson planning, and instruction on different teaching methods. Given the many possible benefits of collaboration in the CSMP community, it is not surprising that the majority of sites reported collaborating with other sites in the CSMP (63%) and organizations outside the CSMP (80%) to reach their goals (Exhibit 4-1).
SRI Proprietary
42
Exhibit 4-1 Percent of Sites Reporting “Strongly Agree” or “Agree” to Statements about Collaboration
Our site collaborated with other sites in the CSMP in order to reach its goals.
All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
63%
43%
63%
83%
57%
77%
41%
69%
100%
86%
91%
77%
53%
81%
67%
Our site collaborated with organizations 80% 100% 100% outside the CSMP to reach its goals. Source: SRI CSMP Site Director Survey (2004).
Sites in all Projects reported collaborating with organizations outside of the CSMP more than with other CSMP sites, with the exception of the Arts Project, in which 100% of sites collaborated with other sites in the CSMP. The extent to which various Projects collaborated varied, however. The Science Project site leaders reported the least collaboration, with only 41% of sites indicating collaboration with other CSMP sites and 53% indicating collaboration with outside organizations. Other Projects seemed to collaborate much more; all of the sites in the Foreign Language and History-Social Science Projects worked with outside organizations to meet their goals. Although the majority of sites reported collaborating with other CSMP sites in 2003-04, they reported engaging in fewer collaborative activities in 2003-04 than they did in 2001 (Exhibit 42). Some of the most dramatic declines since 2000-01 were in holding joint events and visiting each others’ sites to observe activities.
SRI Proprietary
43
Exhibit 4-2 Percent of Sites Reporting Various Types of Collaboration with Other CSMP Sites, Either Within or Outside of the Subject Area, in 2000-01 and in 2003-04 45
Developed cross-disciplinary events*
36 50
Collaborated on grant-writing*
38 60
Visited each other’s sites to observe activities*
32 62
Co-sponsored or held joint events*
47 69 66
Provided expertise to these sites
76
Held meetings with the site staff to plan or share* ideas
66 78
Drew on the expertise of these sites*
66
0 2003-04
2000-01
20
40
60
80
100
Percentage
Sources: AIR Site Director Survey (2001); SRI CSMP Site Director Survey (2004); SRI analysis. *p < 0.05
Taken as a whole, these data suggest that the network still plays an important role in facilitating the CSMP’s work, but the extent to which site directors use the network may be decreasing. PARTNERSHIP WORK The CSMP also engaged in activities with its partner schools and districts (Exhibit 4-3). Like many other characteristics of the nine projects, a wide range of difference and similarity exists within the group. There is a great degree of homogeneity among activities that form the core features of partnerships outlined in the RFP. Most CSMP sites, according to our site director survey, reported that they frequently or very frequently invited teachers from their partnership schools/districts to attend activities/events (96%), assess the specific professional development needs of partner schools/districts (85%), and develop specialized programs tailored to the unique needs of a partner school/district (84%). Other site characteristics revealed a greater degree of heterogeneity. Although 60% of sites reported that they frequently or very frequently worked with partner schools/districts to ensure that teachers were highly qualified under NCLB, certain
SRI Proprietary
44
Projects’ sites engaged in this activity more often than others. For example, 75% of HistorySocial Studies and Writing Project sites engaged in such work frequently or very frequently, whereas only 38% of International Studies Project sites did so. The explanation for these differences is not clear, however. Fifty-seven percent of Foreign Language Project sites, which received no NCLB funds, reported that they undertook this activity frequently or very frequently. Arts Project sites, meanwhile, received more than $500,000 from NCLB, but 83% reported that they never engaged in that work or did so infrequently. Data on coaching in partner schools/districts also revealed a range of variation. Overall, 53% of site directors reported engaging in this type of activity frequently or very frequently, but only 29% of Foreign Language sites did so, whereas 75% of Writing Project sites did. Exhibit 4-3 Percentage of Sites Reporting that They Engaged in the Following Partnership Activities “Very Frequently” or “Frequently” All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
Invite teachers from their partnership schools/districts to attend activities/events
96%
100%
100%
100%
90%
100%
89%
100%
100%
Assess the specific professional development needs of partner schools/districts
85%
86%
88%
88%
76%
85%
89%
84%
100%
Develop specialized programs tailored to the unique needs of a partner school/district
84%
71%
88%
75%
76%
77%
80%
100%
100%
Work with partner schools/districts to ensure that teachers are highly qualified under NCLB
60%
57%
75%
38%
57%
69%
63%
75%
17%
50%
38%
62%
46%
47%
75%
67%
Coaching in partner 53% 29% schools/districts Source: SRI CSMP Site Director Survey (2004).
These data suggest that, regardless of the diversity across the Projects, the sites worked with partner districts and schools and their teachers on a range of activities.
SRI Proprietary
45
TEACHER LEADERS CSMP sites also devoted resources to developing teacher leaders. Teacher leaders are generally experienced teachers whom sites commonly use to plan and provide professional development programming. The definition of their role and their use varied among the Projects and sites, sometimes as a function of their partners’ needs. This model is part of the “teachers training teachers” approach to professional development and, in theory, helps diffuse more content knowledge and pedagogical skill among a greater number of teachers and also helps build cadres of instructional leaders at school sites. Once a critical component of the CSMP strategy, the focus on teacher leaders has been somewhat diminished as a result of shifting priorities in the state. The 2001 RFP, for example, no longer required this activity as necessary for receiving core funding. Many sites, however, continued to develop teacher leaders and rely on them for many professional development tasks. For some sites, for example, the teacher leader component included having teachers on release time available to coach others. “Districts are also asking for us to provide more coaches than ever,” a Mathematics Project site director noted. “We have spent more time working with our leaders on how to coach in the mathematics classroom.” Other sites had teacher leaders plan and implement workshops and summer institutes. According to another Mathematics Project site director: Teacher leaders have been more involved in the planning and delivery of instruction at summer institutes. This is due in large part to asking teacher leaders to observe, to chronicle notes about teaching strategies and content being used, and to debrief the instructors. In this way we were able to develop teachers to begin to take the lead for instructional delivery. Other sites used teacher leaders for research and development purposes. An Arts Project site director reported: Another aspect is that the new priorities [district needs and larger coverage area] require new programs to be developed. This means that teacher leaders are being asked to do more research and development. Teacher leaders have also been asked to do more consultation work in terms of aligning standards to curriculum and assessment. All in all, there is a growing need for more “behind-the-scenes” work as opposed to the more traditional presentation/demonstration work. Although many sites report that funding pressures have decreased the number of teacher leaders they are able to enlist (see Chapter 3), across all sites, teacher leaders still led more events/activities than any other group. Teacher leaders provided 40% of events, as compared with 34% provided by site directors, 14% by professors in the subject area, and 6% by external experts (Exhibit 4-4). Again, we note that the Projects varied in this area: only 22% of the
SRI Proprietary
46
International Studies Project events and activities were provided by teacher leaders, compared with 65% of Writing Project events and activities. Exhibit 4-4 Mean Percentage of Activities Led by Various Types of Providers All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
Teacher leaders
40%
52%
27%
22%
27%
59%
23%
65%
45%
Site directors
34%
23%
36%
35%
43%
21%
39%
2%
50%
Professors or faculty in the subject area
14%
4%
23%
31%
19%
2%
24%
3%
4%
External experts (not from the site, partnership district, or university)
6%
13%
3%
6%
3%
10%
8%
2%
1%
Professors or faculty in education
4%
8%
1%
5%
5%
3%
6%
3%
1%
2%
1%
2%
5%
1%
1%
1%
School and district 2% 0% administrators Source: SRI CSMP Site Director Survey (2004).
Other priorities, like working in low-performing schools and with curriculum-based professional development programs, have altered how sites use teacher leaders. “The teacher leadership component of our Project takes a lesser percent of the time we are involved in delivering [professional development],” a Mathematics Project site director noted. “This resulted from the mandates to serve a wider audience through [Mathematics Professional Development Institutes], AB 466 programs, and other demands.” A History-Social Science site director cited a similar experience in managing other CSMP demands: “Since 2001, we have worked hard to expand our leadership cadre to address the needs of our low-performing school partnerships as well as provide leadership for many different grade level-specific and discipline-specific programs.” It appears, then, that site directors were able to adapt existing CSMP structures to meet the evolving policy requirements. As Exhibit 4-5 indicates, teacher leaders were still involved in all of the activities mentioned above in 2003-04; in general, however, sites used teacher leaders to a lesser extent than they did in 1999-2000. The biggest differences were in the percentage of sites that used teacher leaders to organize events or activities (64% in 1999-2000, versus 38% in 2003-04), the percentage of sites that had teacher leaders lead institutes (65% in 1999-2000 versus 53% in 2003-04) and workshops (75% in 1999-2000 versus 65% in 2003-04).
SRI Proprietary
47
Exhibit 4-5 Percent of Sites Reporting that Teacher Leaders Assumed the Following Roles to a "Large Extent" in 1999-2000 and in 2003-04 64
Plan or organize events or activities*
38 65
Lead part of an institute*
53 75
Lead workshops or follow-up activities*
65 34 25
Coach teachers*
30 27
Promote or publicize events Observe and provide feedback to teachers in other ways
23 22 28 28
Lead study/discussion groups 0 2003-04
20
1999-2000
40
60
80
100
Percentage
Source: SRI CSMP Site Director Survey (2004). *p < 0.05
In both 1999-2000 and 2003-04, teacher leaders were used to a lesser extent in coaching teachers, promoting events, observing other teachers, and leading study groups, and few, if any, changes took place in these areas. Exhibit 4-5 shows the overall trends in ways that sites used teacher leaders, but masks variations among Projects (see the technical appendix (Hough, et al., 2005) for details about these differences). Next we discuss the broader context for these activities by describing the various event formats CSMP used. THE FORMAT OF PROFESSIONAL DEVELOPMENT Chapter 3 discussed the many types of professional development the CSMP offered. Of all the event types, two were particularly noteworthy, given research on effective professional development: (1) institutes, which were usually held during the summer and typically included some kind of formal follow up, and (2) coaching, in which teacher leaders or mentors worked directly with participating teachers at the school site (Birman et al., 2000; Garet et al., 1999). We discuss each in turn.
SRI Proprietary
48
Institutes Intensive professional development institutes have been a hallmark of the CSMP for several years. They allow teachers to practice skills and/or develop content knowledge over a sustained period, often with follow-up during the school year. Data from the OIS (presented in Chapter 3) suggest that institutes continued to be one of the major forms of professional development the CSMP provided. Institutes varied significantly in their focus (e.g., content knowledge, strategies for teaching ELLs in the content area) and length (1-4 weeks, with or without follow-up); in general, however, they provided teachers with at least a week of intensive study on a given topic. Research on effective professional development suggests that CSMP institutes have several characteristics that are likely to affect teachers’ practice and their students’ achievement (Birman et al., 2000; Garet et al., 1999). First, institutes were generally of moderate duration, typically lasting about 40 hours. When follow-up was included, some institutes provided teachers with 120 hours or more of professional development. Institutes generally focused on content, and included strategies for teaching that content to students (see the example in the text box).
SRI Proprietary
49
A Fourth- and Fifth-Grade Science Institute One Science Project site held a 2-week summer institute for fourth- and fifth-grade teachers that focused on teaching to the 4th and 5th grade standards. Institute participants came largely from partner districts, but teachers also came from elsewhere in the region. The site directors admitted teachers in pairs from the same district to ensure a degree of collective participation, even though they were serving teachers from 11 districts. Each day, morning presentations focused on topics such as teaching science to ELLs, strategies for integrating literacy skills into science instruction, and pedagogical approaches to teaching science. A variety of workshops were held each afternoon, but teachers also had time to identify a scientific question of interest to them and work with other teachers interested in the same question to design experiments and use outside sources of information to answer the question—in other words, to “do” science. In the afternoon workshops, participants were often broken into groups by grade level. For example, one afternoon the fourth-grade teachers convened for an activity on magnets—a topic that site directors had identified as one that teachers had difficulty teaching well. This workshop sought to give participants an understanding of how to use inquiry teaching to help students meet the content standards. The workshop opened with an inquiry activity in which teachers were given several types of magnets and other materials to explore. After initial exploration, the presenter asked teachers what they knew about magnets. He wrote their comments on the board and asked them to test particular assertions that he drew from their comments. The assertions he chose were related to important concepts about magnets or dealt with specific misconceptions people hold about magnets. The workshop’s culminating activity was watching a video of a teacher from the San Francisco Exploratorium teach a magnetism lesson in a fourth-grade classroom. The presenter frequently paused the tape to comment on how the teacher was following the inquiry model and on what specific things she was doing to shape the students’ work so that they were exposed to all of the key content standards during the lesson. This was an excellent example of a presenter helping teachers see how to do something they frequently struggle with—namely, structuring inquiry lessons to meet specific content standards—through modeling of the technique and guiding them through reflections on exemplary practice.
This approach is likely to increase teachers’ ability to apply what they learned to their teaching. In the given example, the institute’s participants attended separate afternoon sessions based on the grade level they taught—an example of the differentiation supported by research on effective professional development. The site directors also reported that in previous years they followed up institutes with coaching to help teachers apply what they learned, another feature of high-quality professional development. Because of reduced funding, however, they could not do so in 2003-04. CSMP follow-up activities offer institute participants the opportunity to reconnect with the CSMP after they have started to implement what they learned at the institute. For teachers trying new practices that differ dramatically from those to which they are accustomed, follow-up can
SRI Proprietary
50
play a key role in helping teachers persist with different, but more effective, practices until they become comfortable with them. The CSMP employed a great variety of follow-up activities, including workshops, minicourses, and coaching. The research on effective professional development suggests that this type of ongoing support, especially if it is based on the teachers’ context (like coaching), can be a particularly effective form of professional development (Birman et al., 2000; Garet et al., 1999). Coaching Having the opportunity to observe high-quality instruction by peers and receiving feedback on one’s own teaching are often cited as particularly helpful activities for teachers. Some sites were able to expand coaching in conjunction with partnerships. A Mathematics Project site, for example, offered many coaching sessions, including a peer observation and coaching opportunity for teachers of grades 6 through 12. For 2 hours in the morning, teachers observed two periods of mathematics instruction taught by an experienced teacher leader. Following that, they spent 2 hours discussing their observations, the instructional strategies used, and the methods the teacher used to engage her students in the learning process. For the final 2 hours, participants discussed and planned grade-specific lessons that included the instructional techniques they saw modeled earlier in the day. Some CSMP sites used another type of coaching as a form of institute follow-up. In this approach, teacher coaches who presented at the institute or were aware of the strategies introduced in the institute visited teachers in their classroom. Sometimes, the teacher asked the coaches to conduct a model lesson with the teacher’s students, illustrating how to use the strategies in the teacher’s context. Other times, the coach might visit and provide feedback on implementation. As these examples show, the nature of coaching varied across the CSMP. Although the first example provided infrequent opportunities to observe and discuss exemplary teaching and then work with peers to apply lessons learned, the second was part of a repeated process designed to guide teachers through implementation of new strategies. Not all sites, however, were able to use coaching in their work, and some that used to employ more coaching reported cutting back in that area due to budget cuts. As shown in Exhibit 4-6, although 51% of sites overall reported an increase in coaching since 2001, some sites reported a sizable decline in this activity. In the History-Social Science, Mathematics, Writing, and Arts Projects, the vast majority of site directors (63%, 67%, 73%, and 83%, respectively) reported doing more coaching in 2003-04 than in 2001. In contrast, many site directors in the Foreign Language and Reading and Literature Projects reported that they had limited their school-site coaching activities since 2001 (43% and 33%, respectively). Funding cuts, as opposed to not
SRI Proprietary
51
believing in the efficacy of coaching, appeared to underlie the decline seen at some sites. In response to a question about how funding affected site goals and activities, a Science Project site director said, “We are doing less coaching with teachers at the site level.” Similarly, a Writing Project site director reported that changes in funding “reduced [their] ability to develop a larger cadre of coaches for [their] work.” Given the potential strength of coaching in supporting teachers’ implementation of new knowledge and skills, funding cuts could be hurting the effectiveness of the CSMP in this area. Exhibit 4-6 Percent of Sites Reporting that They Did More Coaching in 2003-04 than in 2001 100
Percentage
80
60
83
40
73
67
63 51 43
20 21
38 25
33 33 25
33 22
14
20
5
0
0 All
CFLP
CH-SSP
CISP
CMP
More than in 2001
CRLP
CSP
CWP
TCAP
Less than in 2001
Source: SRI CSMP Site Director Survey (2004).
THE CONTENT OF PROFESSIONAL DEVELOPMENT Developing and deepening teacher content knowledge in specific subject areas have formed the core of the CSMP since its inception in 1988. Each of the nine Projects provides disciplinebased professional development geared toward improving teacher quality and student achievement in their respective fields in accordance with state standards and curriculum frameworks. Research supports the connection between teachers’ content knowledge and their
SRI Proprietary
52
ability to teach their students effectively (Garet et al. 1999). In short, teachers need to know the breadth of the content, how to teach it, and how it was produced in order to be most effective. Sites offered a variety of activities to help increase teachers’ knowledge and skill. For some Projects, a key component of these activities involved “doing” the discipline; that is, engaging in work that historians, mathematicians, and scientists do as part of their professions. This approach, it is believed, allows teachers to return to their classrooms and teach their students how to do the same. Over the course of a summer institute, for example, a History-Social Science Project site offered teachers the opportunity to learn the skills that budding historians practice in graduate school. Those skills included the following: surveying the historiography about their topic; developing, defining, and refining their historical questions; identifying, collecting, and analyzing possible sources of evidence from a variety of media; constructing interpretations that account for conflicting perspectives, power relations, cultural contexts, and contingency; and engaging in peer review. Through attending lectures by historians and reading history books, teachers became familiar with the historiography of their respective research topics. With the assistance of Ph.D. graduate students, they developed, defined, and refined historical questions through the collection and analysis of archival data. Finally, the participants shared their findings with their peers and submitted a three-page research prospectus with an annotated bibliography. The institute, however, did not “provide teachers with explicit help in constructing interpretations or subjecting those interpretations to peer review.” Instead, teachers provided basic reports of their findings and discussed how they might use that information in a classroom (Medina, Pollard, Schneider, and Leonhardt, 2000). Sites also worked to increase teachers’ skills in presenting content to their students. This was often accomplished through activities designed to increase teachers’ content knowledge and pedagogical skills simultaneously. The example of the fourth-/fifth-grade science institute presented above exemplifies how a single activity met these dual goals. Exhibit 4-7 illustrates the emphasis that sites placed on teaching subject matter content and pedagogy: more than 70% of sites reported helping teachers acquire content knowledge and learn appropriate pedagogy “very frequently.” Content and pedagogy were emphasized more than strategies that helped teachers’ students do work in the subject (57%), or the development of teacher leadership skills (30%). We also note the variations among Projects. For example, 75% of sites reported that subject matter content was provided “very frequently,” but only 39% of Reading and Language Project sites did.
SRI Proprietary
53
Exhibit 4-7 Percent of Sites Reporting Giving Teachers the Opportunity to Learn About or Acquire Various Skills and/or Knowledge “Very Frequently” All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
Knowledge of subject matter content
75%
71%
63%
75%
100%
39%
84%
63%
100%
Strategies for using the appropriate pedagogy for their subject area
73%
86%
63%
63%
62%
62%
84%
75%
100%
Strategies that help teachers’ students do work in the subject area (i.e., research, writing)
57%
57%
75%
50%
57%
62%
26%
69%
100%
25%
13%
33%
0%
37%
44%
67%
Leadership skills to improve 30% 29% the content instruction in their schools/districts Source: SRI CSMP Site Director Survey (2004).
Many partnership districts were interested in receiving professional development that helped teachers use adopted curriculum and materials. In addition, some funding streams (e.g., AB 466) required that teachers receive training on adopted materials. As a result, many sites developed professional development opportunities that helped teachers learn how to use the curriculum available in the partner district or how to teach their content within the framework of the curriculum. The influence of district-adopted curriculum seems to have affected various Projects quite differently, as Exhibit 4-8 illustrates. Although 93% of Reading and Language Project sites reported that district adopted curriculum influenced their activities to a “large extent,” only 14% of the Foreign Language sites and 17% of the Arts Project sites did so.
SRI Proprietary
54
Exhibit 4-8 Percent of Sites Reporting District-Adopted Curriculum Influenced their Activities to a Large Extent 100
Percentage
80
60 93
40 56
20
39 25
29
CHSSP
CISP
31
24
17
14
0 All
CFLP
CMP
CRLP
CSP
CWP
TCAP
Source: SRI CSMP Site Director Survey (2004).
Although only 39% of sites reported that district adopted curriculum influenced their activities to a large extent, 61% of all sites reported providing more time to training teachers to use adopted curriculum in 2003-04 than they did in 2001 (Exhibit 4-9). Although Projects varied in this regard, more than half of the site directors in the Foreign Language, History-Social Science, Reading and Literature, Science, and Writing Projects reported devoting more time to training teachers in the curriculum than in 2001. Seven percent of all sites reported that they devoted less time to training teachers to use adopted curriculum than they did in 2001. This number varied by Project: 25% of the International Studies Project sites reported training teachers to use adopted curriculum less in 2003-04 than in 2001 as compared with 5% for the Mathematics Project, 11% for the Science Project, 6% for the Writing Project, and none for the other Projects (see the technical appendix (Hough, et al., 2005) for complete data by Project).
SRI Proprietary
55
Exhibit 4-9 Percent of Sites Reporting that They Trained Teachers to Use Adopted Curricula More in 2003-04 than They Did in 2001 100
Percentage
80
60 100
40
92
71
68
61
63
43
20
40
13
0 All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
Source: SRI CSMP Site Director Survey (2004).
ELLs A key component of CSMP strategy for improving teacher quality and student achievement involves a focus on ELL instruction. The 2001 RFP required the CSMP to focus on the needs of teachers of ELLs, and NCLB also promotes improving instruction of ELLs as a priority. With accountability measures requiring all students to meet California’s rigorous academic standards, teachers need to have access to professional development activities that provide them with the skills to address this population. The CSMP worked to meet these demands by offering events that focused on teaching ELLs in the content areas and, more broadly, teaching literacy in all content areas. As shown in Exhibit 4-10, the vast majority of sites (91%) reported that they “frequently” or “very frequently” offered professional development designed to help teachers teach their subject area to ELLs.
SRI Proprietary
56
Exhibit 4-10 Percent of Sites Reporting that Teachers at their Site Had the Opportunity to Learn Strategies for Teaching ELLs “Frequently” or “Very Frequently” 100
8
80 39
Percentage
47
50 25
60
52 50 92
57
40
25
69 52
50
20
50
53 38
33
14
0 All
CFLP
CHSSP
CISP
CMP
Very Frequently
CRLP
CSP
CWP
TCAP
Frequently
Source: SRI CSMP Site Director Survey (2004).
Most sites in all Projects appeared to have a substantial focus on ELLs, with 100% of the site directors in the History-Social Science, Reading and Literature, and Science Projects reporting a regular focus on ELLs. The Reading and Literature Project had a greater focus than other Projects, with 92% of site directors reporting that teachers “very frequently” had the opportunity to learn strategies for teaching ELLs. As discussed previously, the Reading and Literature Project worked to develop materials centrally and then used the network to disseminate them to all sites and throughout the state. Because the state office worked to develop and disseminate materials specifically geared to the needs of ELLs, it is not surprising that almost all sites focused on ELLs very frequently. In 2000-01, the CSMP placed less emphasis on ELLs. More than 70% of site directors in every Project reported that they devoted more time to preparing teachers how to teach ELLs in 2003-04 than they did in 2001 (Exhibit 4-11). These data strongly suggest that sites were working to respond to the needs of California’s teachers and the requirements of current state and federal policies.
SRI Proprietary
57
Exhibit 4-11 Percent of Sites Reporting that They Focused on Preparing Teachers to Meet the Needs of ELLs More in 2003-04 than They Did in 2001 100
Percentage
80
60
40
79
86 71
100
92
71
79
80
CSP
CWP
50
20
0 All
CFLP
CHSSP
CISP
CMP
CRLP
TCAP
Source: SRI CSMP Site Director Survey (2004).
Like most other features of the nine Projects, the approaches used to build teachers’ skills with ELLs varied. The Science Project invested significantly in building and disseminating knowledge about teaching science to ELLs. The Project hired a consultant to compile relevant research on teaching ELLs science, and posted a summary of the findings on-line so the information could be widely accessed (Dobb, 2004). The Project also supported the development of a unique assessment that enabled scorers to differentiate students’ science knowledge from their language proficiency. The Foreign Language Project used a radically different, but highly innovative approach. For most of its history, the Foreign Language Project had been working with foreign language teachers at higher performing schools, but found that teaching foreign language was not the most critical need at low-performing schools. Working with ELLs, on the other hand, is critical. The Foreign Language Project executive director explained: We now come into a school and look at their entire language program, including the ELL program, language arts program, and the foreign language program, and work with teachers to develop language acquisition strategies for both ELL students and students in traditional foreign language programs. We now look at the differing needs of ELL students. Some are heritage speakers—born in the United States—and have been exposed to foreign language
SRI Proprietary
58
but not formally schooled in foreign language. They may have a limited foundation of vocabulary but not an oral command of the language.” Sites also worked with teachers of native language speakers who had “an oral command of the language (born abroad) but [were] not literate in their native language” by assessing these different needs and developing strategies for helping all students develop language skills. Overall, the Project was trying to develop good language learners and to “help students learn language patterns so that they [could] succeed in all other content areas.” To launch its nascent ELL component, the Physical Education-Health Project enlisted the assistance of the Reading and Literature Project to help it understand how to implement ELL strategies in the professional development activities it provided. “We are not just paying lip service to ELLs,” the codirector of the Physical Education-Health Project stated, “we are now trying to get our head around what that means to a PE or health teacher.” Reading and Literature Project staff members shared information and strategies with teacher leaders and Project leadership about assessing students’ language development. Much of this knowledge was new to them because school site administrators did not regularly share these data with PE teachers nor had these teachers been exposed to many ELL instructional strategies during their preparation or professional development. The co-director noted: [The Reading and Literature Project codirector] talks about scaffolding etc., but there’s not a PE teacher I know who understands that. They’re desperate to understand the specific strategies and the theoretical framework and the research that informs the classroom teachers about those strategies so they can begin to incorporate them into the physical world. Teachers are thirsty for this. This work helped the Physical Education-Health teacher leaders think more critically about their practice and how to align future professional development activities for participants with the ELL goals in mind. As these examples suggest, the Projects were at different stages of building site directors’ knowledge of strategies for teaching ELLs and they pursued the state’s goals in unique ways. Nonetheless, ensuring that teachers could support ELLs’ success in all content areas was a unifying goal across the CSMP. Literacy across the Content Areas Another consistent goal across the CSMP was preparing teachers to teach literacy across the content areas. As discussed previously, part of the pressure to move in this direction at the elementary level came from the API, which disproportionately favors reading/language arts and mathematics scores in calculating school performance. At the secondary level, where students need to be able to read and write fluently in order to succeed in most courses, the impetus for teaching literacy across the content area came from a broader base. More than 65% of site directors in each Project reported emphasizing literacy strategies (Exhibit 4-12). SRI Proprietary
59
Exhibit 4-12 Percent of Sites Reporting that Teachers Had the Opportunity to Learn Literacy Strategies “Frequently” or “Very Frequently” 100 23 37
Percentage
17
88
83
CWP
TCAP
38
80 60
58
14 38
40 20
13
50
57
60
63
77
38
37 5
0 All
CFLP
CHSSP
CISP
CMP
Very frequently
CRLP
CSP
Frequently
Source: SRI CSMP Site Director Survey (2004).
Eighty-seven percent of site directors reported that literacy strategies were taught “frequently” or “very frequently.” The Mathematics Project that had the lowest emphasis on literacy; only 5% of site directors reported “very frequently” offering opportunities to teach literacy strategies in mathematics, but an additional 60% reported they offered such opportunities “frequently.” Data presented earlier suggest that, in addition to educational reasons for wanting to help all students improve their literacy skills, the need to raise funds from sources outside UCOP might have increased Projects’ focus on literacy. Teaching literacy through content can help legitimize the expenditure of scarce district funds on subject matter that might be ignored in lowperforming, accountability-sensitive districts. “If we can continue to enlarge teachers’ capacity to cover academic content while paying attention to literacy, then we’ve accomplished two things,” the International Studies Project executive director explained. Some Science Project sites sought to meet this challenge by advertising how some pedagogical strategies for teaching science could concurrently build students’ skills in reading, language arts, and/or mathematics. For example, one site referred to the strategy of using student notebooks to achieve the learning of both science and literacy skills. At this site, student notebooks were seen as a tool to organize and explain various processes of scientific experiments. As one Project director stated: It’s not just for students to write down the scientific method. [It’s a] notebook to collect primary source data from their activities, and to use the notebook as practice to write sentences—justification, evidence from their experiments, method of scientific inquiry. It’s SRI Proprietary
60
not just a place to store a “feel-good” thought or “This is what happened today in science” ... We’re really trying to get the critical thinking piece in, and get it in full sentences. We want teachers to grade it by science content, and then re-grade it for literacy. CONCLUSION The context in which the CSMP operates changed significantly in the past few years. The CSMP responded to the state’s changing needs by offering activities that focused specifically on preparing teachers to teach literacy across the content areas and meet the needs of ELLs. Although aspects of the CSMP evolved, core structures and purposes remained intact. The CSMP network continued to play an important role in building the capacity of the CSMP. Additionally, the CSMP continued to offer the standards-based content and pedagogical preparation for which it is renowned in various formats, many of which are strongly supported by research on effective professional development practices.
SRI Proprietary
61
SRI Proprietary
62
CHAPTER 5—IMPACT OF THE CSMP ON CALIFORNIA TEACHERS AND THEIR STUDENTS The goal of the CSMP is to increase teachers’ content knowledge and skill in order to improve their practice—with the ultimate goal of increasing student achievement. In year 1 of the SRI evaluation, we did not collect original data that directly measured the CSMP’s impacts on teachers and students. We plan such original data collection for year 2 of the evaluation. In this chapter, we address the quality and impact of the work of the CSMP in three ways: (1) we review our findings of the degree to which the CSMP professional development practices were consistent with the research literature on effective professional development; (2) we review previous evaluations of CSMP work and the evidence they provide about the benefits of CSMP participation;21 and (3) we comment on the degree to which the CSMP’s evaluation practices meet the Projects’ needs. THE CHARACTERISTICS OF CSMP PROFESSIONAL DEVELOPMENT As noted in Chapter 1, researchers have reached consensus about the characteristics of effective professional development. Those characteristics include sufficient duration, opportunities for groups of teachers to learn collectively, a clear focus on content, opportunities for active learning, and a coherent approach across activities (Birman et al., 2000; Garet et al., 1999; Kennedy, 1999; Loucks-Horsley, 1999; Putnam & Borko, 2000). We have not attained direct measures of the CSMP’s performance for each of these characteristics, but we present relevant data that begin to address these issues. We begin with a simple question of whether the Projects relied on current research to structure their work. Most site directors (96%) in all Projects reported that current research in teacher professional development influenced their activities to a “large” or “moderate extent” (Exhibit 5-1). Relatively little variation occurred across Projects, with site directors’ responses ranging from 88% for the International Studies Project to 100% for the Foreign Language, History-Social Science, Mathematics, Reading and Literature, and Arts Projects.
21
During the literature review, we uncovered research and development studies, which, although useful for describing some of the activities the Projects were engaged in, did not directly relate to our evaluation questions. Broadly defined, the purpose of this type of research was to inform researchers and practitioners outside of the CSMP about instruments and measures developed to evaluate teachers’ instructional practices, assess student learning, or examine the methods used to evaluate Project activities. These studies focused on aspects other than the nature of the CSMP or their impact on teachers or students. The Foreign Language Project, for example, produced a study describing the Foreign Language Oral Skills Evaluation Matrix (FLOSEM), a rating scale for assessing communicative proficiency among students (Padilla and Sung, n.d.).
SRI Proprietary
63
Exhibit 5-1 Percent of Sites Reporting Current Research Influenced their Activities to a “Large” or “Moderate” Extent 100
7
14
Percentage
80
29
22 50
17 26
13
25
60 40
93
86 74 63
71
68
75
83
50
20 0 All
CFLP
CHSSP
CISP
CMP
Large Extent
CRLP
CSP
CWP
TCAP
Moderate Extent
Source: SRI CSMP Site Director Survey (2004).
We also asked site directors a series of questions regarding the type of approaches they used to help teachers acquire the relevant knowledge and skills for teaching the respective disciplines. The approaches we asked about are considered consistent with the research on effective practice because of their content focus, coherence with teachers’ context, use of active learning strategies, and/or collective participation (Birman et al., 2000; Garet et al., 1999). More than 90% of all site directors reported that they frequently or very frequently modeled effective instructional practices for teachers, had teachers “do work in the discipline,” and had them undertake more complex activities than they would do with their students (see Exhibit 5-2). More than 80% of site directors reported using the remaining strategies shown in Exhibit 5-2 frequently or very frequently.
SRI Proprietary
64
Exhibit 5-2 Percent of Sites Reporting Using Various Professional Development Approaches “Frequently” or “Very Frequently” Engaging teachers in experiences that have them do work in the discipline (e.g., writing a piece in the Writing Project, investigating a problem in the Mathematics Project)
67
Modeling effective instructional practices for teachers
28
79
Engaging teachers in activities that are more in-depth or more complex than they would do with their students
15
52
Providing information on subject matter content, through lecture or demonstration
42
53
Having teachers work in collaborative groups with colleagues from their school
35
42
Allowing teachers to practice (or learn about) exact lessons that they can use with their students
44
52
0
20
31
40
60
80
100
Percentage Very Frequently
Frequently
Source: SRI CSMP Site Director Survey (2004).
In addition to selecting types of activities supported by research, site directors often attempted to structure key features of event format in ways that were research-based. Site directors reported that an average of 59% of participants attended events with colleagues from their schools (Exhibit 5-3). These data suggest that many CSMP events had the attribute of “collectivity” that research suggests increases the likelihood that professional development will positively affect teacher practice.
SRI Proprietary
65
Exhibit 5-3 Mean Percentage of Teachers Attending Site Events/Activities with Other Teachers from their Schools 100 80
Percentage
60 85
83
40 59
56
63
60
55
54
48
CSP
CWP
20 0 All
CFLP
CHSSP
CISP
CMP
CRLP
TCAP
Source: SRI CSMP Site Director Survey (2004).
Previous chapters described the prevalence, format, and content of many CSMP activities. Data from the OIS detail the numerous types of activities offered and indicate the characteristics of some of those activities. We know from those data, for example, that many institutes last a week or more and include some type of follow-up—consistent with research on the importance of a strong “dosage” of professional development (Birman et al., 2000; Garet et al., 1999). However, OIS also indicates that many activities last only a short time and that the number of follow-up activities decreased in 2003-04. Accordingly, it is difficult to generalize about the effectiveness of CSMP events. CSMP SELF-EVALUATION The 2001 RFP tasked sites with evaluating their work for evidence that student achievement increased as a result of teacher participation in CSMP activities. Minimally, UCOP wanted these studies to address what participants learned, how what they learned affected their instructional practices, what standards-related topics students learned, trends in student achievement in lowperforming schools (disaggregated by subgroups), and the link between what participants and students learned. In this section, we review the Projects’ efforts to support internal evaluation and research. The first finding is that self-evaluation proved difficult for many sites to manage. Financial constraints, site goals, and insufficient evaluation expertise resulted in a range of studies, many of which fell short of meeting UCOP’s goal of assessing the impact of the CSMP on student achievement. SRI Proprietary
66
Like other activities, CSMP’s evaluations have been affected by funding reductions. Site directors reported conducting fewer evaluation activities or conducting none at all due to a lack of funds. “We have been collecting data from our partnership programs,” a Writing Project site director stated, “but we do not have the financial resources or time available to do much with what we're collecting. It's one of the results of decreased funding.” An Arts Project site director reported a similar experience: We have been in survival mode during the last year, which has made it difficult to be reflective about our work. Our site [evaluation] work has continued, but having the person power and expertise to collect, manage, and analyze the material is an issue. We hope there will be more support for this when the site is to package a report. However, although sites reported that the decrease in funds had put a strain on their evaluations, site directors’ opinions about evaluation had not changed much since 1999-2000 (Exhibit 5-4). For example, although only 28% in 2004 agreed or strongly agreed that they had the resources needed to conduct their evaluation, the number in 1999-2000 was quite similar (32%). Site directors seemed to view their evaluations positively overall; a majority of sites in 1999-2000 and 2003-04 reported that they had a clear plan for evaluation (68% in 1999-2000 and 72% in 2003-04) and that the process of conducting the evaluation was useful (82% in 19992000 and 85% in 2003-04). Although these responses would appear to be quite positive, further examination revealed more mixed results. In 2003-04, for example, 60% of site directors reported that “deciding how to collect the data has been a significant challenge of our evaluation.” This is a decrease from 77% in 1999-2000, but it suggests a problem: data collection is merely the first stage in an evaluation process, and if the majority of sites struggled with basic evaluation planning, it is likely that sites did not uniformly have the capacity to carry out effective evaluations.
SRI Proprietary
67
Exhibit 5-4 Percent of Sites Reporting “Strongly Agree” or “Agree” to Statements about Site-Level Evaluation in 1999-2000 and 2003-04 We have the support of researchers/ consultants to guide our evaluation
57 59
The process of conducting our site-level evaluation has been useful for us
82 85 55 62
We have a good handle on how to analyze the data we collected * 32 28
We have the resources needed to collect the data for our evaluation
77
Deciding how to collect the data has been a significant challenge for our evaluation *
60 68 72
We have a clear plan for our site-level evaluation
0 2003-04
1999-2000
20
40
60
80
100
Percentage
Sources: AIR Site Director Survey (2000); SRI CSMP Site Director Survey (2004); SRI analysis. *p < 0.05
Some sites have been able to access additional capacity to undertake evaluations. The addition of federal funds through the TAH program, for example, allowed the History-Social Science Project to undertake more sophisticated and comprehensive evaluations. Starting in 2003-04, TAH grants required that recipients measure the impact of their projects on student achievement.22 One site, for example, was conducting three evaluations for each of its three grant-related projects. “Our first major evaluation effort… featured the collection of close to 4,000 writing samples from more than 2,000 students from both test and control groups over the 2003-04 school year,” the site director noted. “A random sample of these essays is being scored by our outside evaluator.” The other two evaluations will employ an experimental design. Another site will assess the impact of both its literacy efforts and historical content activities on teacher practices and student achievement. The first will examine these two elements before and after teachers participate in TAH, whereas the second study will compare the students of teachers who participated in the professional development with the students of those who did not attend activities. Although we have not seen the study designs for these evaluations, their general structure and approach have some of the characteristics of meaningful and useful research. 22
Before 2003-04, TAH grants required evaluations but did not specify that evaluations needed to include measures of impacts on student achievement.
SRI Proprietary
68
The state offices of some Projects also used their resources to develop instruments that sites could use in their evaluations. As noted in Chapter 4, the state office of the Science Project, for example, worked to design assessment items for accurately assessing the scientific knowledge and skills of students with varying English language proficiency. Similarly, the state office of the Writing Project developed a rubric system for scoring writing samples, which we discuss below. Another issue for some sites was collecting data that addressed questions of interest to both the site and UCOP. The 2001 RFP made it clear that UCOP was interested in tangible impacts of CSMP participation on teacher knowledge and skills, teaching practices, and student achievement. Exhibit 5-5 indicates the types of measures that sites reported they used to conduct their evaluations. More than half of all sites (52%) reported that they found their evaluations of teacher perceptions of the quality of events “largely valuable,” with little variation across Projects. However, although UCOP encourages impact evaluations, fewer site directors reported that their evaluations using teacher content knowledge (39%), teacher practice (49%), and student achievement (27%) were “largely valuable.” It is important to note the variation across Projects. For example, although 27% of all CSMP sites found student achievement to be a useful measure, Project reports ranged from 0% (International Studies) to 50% (Reading and Literature). Exhibit 5-5 Percent of Sites Reporting that They Found Various Evaluation Measures “Largely Valuable” All
CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
Teacher perceptions of the quality of events
52%
43%
57%
67%
43%
42%
63%
53%
60%
Teacher content knowledge
39%
29%
43%
17%
52%
8%
47%
29%
80%
Teacher practice
49%
86%
71%
17%
29%
33%
26%
82%
100%
Student achievement
27%
29%
29%
0%
19%
50%
11%
47%
40%
Source: SRI CSMP Site Director Survey (2004).
These data suggest that frequently, for whatever reason, site leaders do not collect or use evaluation data on student achievement—and to a lesser extent, data on teacher content knowledge and practice. We turn to site evaluation next.
SRI Proprietary
69
RESULTS OF INTERNAL STUDIES OF THE CSMP As part of the overall evaluation, we requested that each of the nine CSMP executive directors submit studies they had conducted or supported since 2000.23 We received evaluations from all Projects except the Reading and Literature and the Physical Education-Health Projects.24 Appendix B provides a list of the studies we reviewed. The studies varied both in scope and purpose. Many studies focused on a single institute or set of workshops and, as a result, had a relatively small number of participants; others investigated the effects of a program of work on many teachers and students. We divided the evaluations into four main groups on the basis of their primary focus: teachers’ perceptions of professional development activities, effects on teacher knowledge, effects on teacher practice, and effects on student achievement. The studies also varied substantially in the level of detail of reporting. Some evaluations were reported in articles published in refereed journals; some had full descriptions of the intervention, research design, and analysis; and others mostly consisted of raw data or brief descriptions that did not enable SRI to evaluate the findings. The Mathematics Project, for example, included a document that mentioned studies that showed positive impacts of the Project’s work on teachers’ content knowledge and practices, and student achievement; however, the specific studies were not submitted and thus could not be part of this review. As a result of uneven reporting, the discussion below does not include all evaluations conducted by the Projects over the past few years. Teachers’ Perceptions of Professional Development Activities Much of the research reviewed focused on teachers’ perceptions of the quality and utility of their professional development, often limited to a particular set of workshops or a single summer institute. Such studies cannot be used to support broad statements about the effects of the CSMP, although they may be highly relevant to site-level decision-making. One such evaluation, for example, evaluated a summer institute and presented copious data that included participants’ perceptions of the applicability of each workshop to their classrooms, the reported effect of each workshop on participants’ content knowledge, and perceived quality of each presenter. The study also extensively categorized the characteristics of the approximately 30 teachers who participated (Mark, 2002). The amount of information presented in this report suggests a genuine effort to provide comprehensive data on the institute to the site director. 23 24
Some of the research submitted antedated 2000, but was still included in our review. Although we did not receive studies from the Reading and Literature Project, it is in the final stage of an external evaluation of its RESULTS work to examine the relationship between teachers’ RESULTS participation and student learning.
SRI Proprietary
70
The site could use the data collected to decide which presenters to retain (or replace) for future institutes or to identify perceived weaknesses in the institute. “We have used our evaluations to help with the planning of follow-ups and other activities,” a Mathematics Project site director reported. As such, this type of evaluation could be valuable for a site director; however, it does not provide strong evidence of the quality or impact of the CSMP. Several studies sought teachers’ perceptions on the impact of participation on student achievement. Stokes, Dickey, Heenan, Hirabayashi, Murray et al. (2003), for example, relied on teachers’ reports that they were increasing their use of effective classroom practices as a result of their participation in the Writing Project and that their participation in Project activities was beneficial to students. Unfortunately, the authors did not report analyses of student work, student reports, or other proxies such as test scores that could have substantiated teachers’ claims about the benefits of their Writing Project participation. The authors concluded that the Writing Project offered professional development of significance and value to the participating teachers, but were unable to make stronger claims about the impact of teachers’ participation on teachers’ practices or student achievement. Effects on Teacher Content Knowledge Many sites used pre-/post-teacher content knowledge tests as a measure of growth in participants’ knowledge over the course of an institute or professional development series. Some of these tests were analogous to the examinations a teacher might give at the end of an instructional unit: they were designed, in large part, to test whether participants recalled particular information presented during an institute. These tests can be useful to site directors seeking to determine whether their institute was effective at delivering specific content knowledge to participants. The Inland Area Science Project (Inland Area Science Project Summer Institute, 2003), for example, created a standards-based assessment of teacher content knowledge using items developed by university professors. Administering the test at the beginning and the end of the 1-week institute, they found that the average increase in scores ranged from 8% to 35% on 5 science strands tested. Although identifying the effects of teachers’ participation on their content knowledge is clearly important for evaluating the quality of an institute, the tests generally appear to have lacked the psychometric properties necessary to generalize much beyond the specific items included in the assessments. Moreover, changes in teacher content knowledge do not necessarily lead to improved teaching or increased student achievement. As a result, whereas tests of teacher content knowledge can provide useful feedback to site directors, they do not in and of themselves provide evidence that the CSMP contributed to improved student outcomes.
SRI Proprietary
71
Effects on Teacher Practice Other sites tried to assess the degree to which teachers practiced what they learned through the CSMP in their own classrooms. One study, for example, investigated the impact of CSMP participation on teacher leadership. Heller, Kaskowitz, and Schecter (2001) sought to investigate outcomes of the Sierra North Arts Project Summer Invitational and Follow-up Institutes, with primary focus on analyzing the development of leadership capacity in institute participants. Participants reported significantly greater confidence in their leadership abilities and a shift in their perception of leadership to a more collaborative, inclusive definition. In addition, according to the report, more teachers assumed leadership roles (15%) in the months between the Invitational and Follow-up Institutes. Another investigation ( Heller, Kaskowitz, Jaffe, De La O, & Alexander, 2003) focused on how participation in the Arts Project had changed teachers’ practices. A brief pre- and post-test Likert-scale measure (five items) revealed that teachers' participation increased their familiarity with arts standards. The authors also found that after participating in the Arts Project institute, teachers expanded their instructional goals beyond creative expression, placing more emphasis on analysis and conceptualization. This study also incorporated a student measure, of which one item was reported in the document we reviewed. From this item, it was determined that students in classrooms taught by Arts Project teachers wrote about art more frequently than students in classrooms with non-Arts Project teachers. Teachers reported that their students demonstrated an improved ability to critique artwork and performances, and the changes appeared to enhance some students’ achievement in the arts. The majority of other studies that evaluated the effect of the CSMP on teacher practice also included measures of student achievement (that are stronger than those reported by Heller et al., 2003), and are thus reviewed in the following section. Effects on Student Achievement Teacher professional development is predicated on the assumption that teachers will acquire information and skills; implement this new knowledge properly on returning to their classrooms; and contribute, in turn, to the academic achievement of their students. Requirements for evidence that demonstrates such impacts are increasing as federal and state accountability requirements continue to mount. We review one summary document and nine studies that address the CSMP’s impact on student achievement. These studies used a varied mix of data collection methods and different types of quantitative and qualitative analyses. Generally, the studies tended to find that the CSMP had a positive, if imperfect (i.e., some of the teacher or student outcomes fell short of the providers’ expectations), impact on California’s teachers and students.
SRI Proprietary
72
The Writing Project (California Writing Project, n.d.) compiled summaries (typically about three pages long for each study) of many evaluations. The studies generally found that teachers’ participation in the Writing Project had myriad positive effects on teachers and their students. Several incorporated pre- and post-writing assignments to measure the academic growth of students in participating teachers’ classes. Taken as a whole, the studies suggest that significant evidence exists that participation in the Writing Project improved student performance on a range of measures, including locally developed quarterly tests, standardized tests (e.g., SAT-9, CAHSEE), and schools’ API scores. Because the document provided summaries only (which was entirely appropriate for its original purpose), it did not afford the level of detail necessary for SRI to analyze the research in greater detail. The Writing Project (California Writing Project, n.d.) also participated in the development of the Improving Students’ Academic Writing rubric for secondary students. The rubric evaluates the competencies tested in the Subject A examination. The rubric was used in the 2000-01 and 2001-02 waves of a 4-year, multisite evaluation of the Writing Project that included 12,789 students and 85 teachers. Using the writing assessment, the Writing Project was able to show pre-/post-test gains from fall to spring for more than 80% of all students and 90% of ELLs. Like several evaluations included in the summary document discussed above, this study did not use a comparison group; thus, although it is clear that students of participating teachers improved, this study did not attempt to show whether they would have improved similarly had their teachers not participated in the Writing Project. An externally conducted evaluation for the University of California, Irvine Writing Project (Land & Olson, 2001) used a quasiexperimental design to study the impact of the Pathways Project in 13 schools, among 45 teachers, 61 classrooms, and 1,724 students in grades 9 though 12.25 Each Pathway teacher was paired with a control teacher at the same school, who taught students of equivalent ability level. The evaluation had 1,516 “control” students along with the 1,724 “treatment” students. Treatment and comparison students were given a timed pretest writing assignment in October and a similar posttest in May. From each classroom, 14 writing assignments were randomly selected for scoring by 2 readers. Students in the Pathways classrooms achieved significantly higher pre-/post-test gains than did the control students on the scale used to assess writing assignment. As a result of the difference in gains between treatment and control students, the authors had strong evidence for their conclusion that teacher participation in Pathways benefited their students.
25
The Pathways Project is a partnership between University of California, Irvine and Santa Ana Unified School District.
SRI Proprietary
73
Like the Writing Project, several Science Project evaluations assessed the impact of participation in the Project on student achievement. Amaral, Garrison, and Klentschy (2002) connected elementary school students’ achievement to teacher participation in a 4-year, kit-based science instruction program developed by the Valle Imperial Project in Science. The authors compared student achievement in science and writing proficiency across the different participant groups and found that the achievement of ELLs increased relative to the number of years they participated in the Project. They also found a positive correlation between the number of years students had teachers who had participated in the program and their achievement in science, reading, writing, and mathematics. These findings support the site’s continued professional development around kit-based science. Stoddart (1999) conducted an evaluation of a 4-week summer school academy for teachers, preservice teachers, and ELLs in grades K through 6.26 The goal of the Language Acquisition through Science Education in Rural Schools (LASERS) academy was to show teachers how to use inquiry methods of teaching science to simultaneously build students’ language and science knowledge and skills. Researchers collected pretest and posttest data on teachers’ beliefs about the integration of science and language instruction; evaluated videotapes of teacher practice for evidence of the integration of science and language development strategies; and administered pretests and posttests of student achievement in science and on multiple measures of language development. The study found that participation in the academy led to increases in teachers’ beliefs in the benefit of integrating language and science in instruction, as well as improvements in student knowledge of science and language skills on all measures. Measures of teacher practice also roughly corresponded to teachers’ beliefs (although the authors acknowledged that the small sample size of 10 teachers made this conclusion tentative). Their use of data from multiple sources supports the authors’ positive findings about the benefits of the academy. Oliver (2000) examined a partnership between Life Lab at the University of California, Santa Cruz, and seven rural school districts (in the central California coast region over a 4-year period; the partnership was also part of the LASERS project. The program, which involved 51 elementary schools, 1,650 teachers, and 34,000 students (about 50% ELLs), emphasized the interaction between English language acquisition and science. The professional development intervention evolved over the 4 years to become fairly intensive and entailed a range of modes of delivery. Data collection occurred over 4 years, beginning with baseline data collection in 199596. It included a survey of 300 randomly selected teachers, annual surveys of all 51 principals, classroom observations, interviews with 10 randomly selected K through 5 teacher-participants, 26
Although the program studied did not occur under the auspices of the CSMP, we include the evaluations by Stoddart in 1999 and Oliver in 2000 in the review because LASERS evolved into the Monterey Bay Science Project in 2000 and continues to use many of the strategies evaluated in the study.
SRI Proprietary
74
and observations of 43 professional development events over the course of 4 years (5 to 12 per year). Classroom videotapes confirmed teacher reports of practices they implemented as a result of participation. Teachers with 100 hours of professional development were compared with those with 20 hours in regard to their readiness to teach various aspects of the science curriculum. Those with more professional development had consistently higher means on selfreported survey measures of preparedness to teach science content and to use pedagogical strategies taught by LASERS. Those with more professional development also spent more classroom time on science, until year 4, when time spent on this subject decreased. The author speculated that the decline was due to accountability pressures that focused more instructional time on reading and mathematics. Stoddart and Clinton (2002) conducted the final evaluation of the Monterey Bay Science Project’s work—an evaluation of LASERS that compared SAT-9 performance in language, reading, and mathematics for students whose teachers had participated in LASERS to those whose teachers had not participated in LASERS. More than 3,000 students (1,300 of whom had teachers who had participated in LASERS) participated in an initial study, which was then replicated in a second district with 1,300 students (650 of whom had teachers who had participated in LASERS). The authors compared student gains for 2 years and consistently found that students whose teachers had participated in LASERS performed significantly better in all three subjects, regardless of the students’ English proficiency. The effect was generally strongest for students who had LASERS-participating teachers both years. In the context of this review, the most salient feature of this study is that it was replicated in a second district: the results are more convincing because they do not appear to be contingent on the particular context of the first district studied. Nordstrom and Delgado (2002) conducted an evaluation of the effectiveness of a different project designed and implemented by the Monterey Bay Science Project. The Designing Integrated Garden Systems Project (DIGS) used a lead teacher model at six elementary schools to disseminate a strategy of using the school garden as the context for activities that integrated science and language. The evaluation reported positive findings on the impact of the program on teacher knowledge of standards and inquiry strategies for teaching science. It also presented data that suggest that having a teacher or teachers who participated in DIGS positively affected student achievement (although many of the comparisons between DIGS and non-DIGS students were not statistically significant). For the International Studies Project, Scarloss, Gehlbach, Herring, and Freeman (2003) employed a quasiexperimental design to measure the impact of professional development on teacher knowledge, classroom practice, and student achievement. The study sample encompassed 24 schools, 59 ninth and tenth grade teachers, 120 classes, and 2,318 students from schools across California. Each participating International Studies Project teacher was matched SRI Proprietary
75
with a comparison teacher from the same school who taught the same course, but did not participate in the International Studies Project. Students in classes taught by International Studies Project teachers were compared with students in classrooms of the comparison group teachers. A wide variety of measures were used to assess outcomes in teachers and students. Teacher measures included a teacher survey assessing teaching philosophy and classroom practice, measures of teachers’ content knowledge and pedagogical skills, and observations of classroom practice (quantitative and qualitative). Student data included surveys (pre- and post-) regarding motivation and self-efficacy related to learning world history, pre- and post-tests of content knowledge, and “concept mapping.” International Studies Project teachers outperformed the comparison group on measures of confidence and pedagogical knowledge. Students in classrooms with International Studies Project teachers also placed a higher value on placing historical events in context—a key historical skill. Finally, the study found that students in classrooms taught by International Studies Project teachers realized an approximately 20% greater gain in history content knowledge than did students in comparison classrooms.27 Finally, Medina et al. (2000) examined two key research questions about the History-Social Science Project: (1) How did participation in the History-Social Science Project affect teachers’ understanding of history?, and (2) How did teachers’ understanding of history affect their students’ learning? The sample included 35 teachers, 6 of whom were selected to participate in a smaller case study group (2 each at elementary, middle, and high schools), and 390 students in grades 5 through 12. Data were collected using a mix of methods, including teacher interviews, classroom observations, reflective teaching journals, student surveys, and student pre- and posttests. The study found that teachers learned the roles of primary and secondary sources and the importance of multiple perspectives in history and passed these understandings on to their students. The professional development activity did not, however, focus on teaching teachers how to ascribe degrees of value and credibility to sources examined, and student understandings reflected this as well. The researchers also found that teachers and students fell short of the program’s goal of teaching and learning historical interpretation. We note that, because the complex skills needed for historical interpretation are generally acquired over several years of study, the participants did meet reasonable expectations (although they were not what the authors had hoped for). This study provides an excellent example of what can be learned from tracing the benefits of CSMP activities to teacher content knowledge, teacher practice, and then student achievement.
27
Differences for students in low- and medium-reading groups were statistically significant, whereas the gain for students in higher reading groups was not.
SRI Proprietary
76
The authors concluded: The presumed relationship between teachers’ professional development work and the subsequent experiences and understanding of their students holds true. In our analysis we found that we could link students’ knowledge to specific elements that had been emphasized in our professional development program….A fuller understanding of the links between professional development and student instruction allowed the Project to address gaps in teacher instruction and thus, we hope, in student understanding” (Medina et al., 2000, p.10). Their summation highlights why such studies provide the CSMP with the most useful types of evidence about effectiveness. CONCLUSION CSMP leadership at both the state and site level was aware of research on effective professional development and attempted to provide professional development consistent with that research. Of particular note, the CSMP’s core focus on content, subject-specific strategies for teaching content, and the collaborative work of teachers set it apart from many other professional development efforts. Site leaders were also aware of the importance of evaluating their efforts, both to improve their work in the future and to report to policy-makers and funders. The evaluations we reviewed typically offered site directors feedback on their current offerings. A relatively small set of studies, focused on a limited number of Projects and sites, also showed the beneficial effects of the CSMP on teacher practice and student achievement. Unfortunately, the majority of evaluations do not enable an outside audience to know with great certainty how and to what extent the CSMP affected teachers and students in California. Studies that would provide such evidence are costly and require substantial capacity to conduct. As site budgets decline, it will become increasingly less likely that sites will be able to conduct the types of evaluations that satisfy the need for strong evidence of the CSMP’s effect on California’s teachers and students. The instruments for measuring student learning developed by the statewide offices of the Science and Writing Projects offer a possible strategy for improving the quality of evaluations in the context of declining funding.
SRI Proprietary
77
SRI Proprietary
78
CHAPTER 6—SUMMARY This Year 1 report is the first of two in a legislatively mandated evaluation of the CSMP. The document relies primarily on secondary data obtained from the Projects and the University of California, as well as on an online survey of site directors. Subsequent data collection will include a teacher survey, case studies of local sites, and a student achievement analysis—all of which will be reported on in December 2005. Key findings of the Year 1 study are set forth below. First, the capacity of the Projects to deliver high-quality professional development was diminished by policy and budgetary shifts. Over the 2 years from 2001-02 to 2002-03, the CSMP saw state funding cut by 86%—from $35 million to $5 million. In response, the CSMP cut sites in many Projects, and the remaining sites exerted substantial effort to raise funds from other sources. Sites varied significantly in the degree to which they were successful at raising outside funding, based partially on whether or not their subject area was prioritized by the California accountability system or other state or federal policies. And even when sites attracted other funding, that funding brought with it additional goals and requirements, which site leadership had to balance against the core work of the Projects. As funding declined, the CSMP scaled back their work. Fewer teachers attended events in 2003-04 than in 2001-02, with the reductions coming largely in the number of institutes and follow-up activities offered. The contact hours per teacher also declined in this period, which is noteworthy since research has established that teachers’ practice is more likely to be affected by participation in events of longer duration. Given missing data in the Projects’ OIS, it is not possible to tell whether these declines have negatively affected the CSMP’s ability to meet its targets for serving teachers of ELLs or teachers who need to become highly qualified under NCLB. Second, in spite of budget cuts and policy changes, the Projects and their local sites retained a strong commitment to the core goals and values of the CSMP: the delivery of standards-based, content-rich professional development. Site directors consistently identified improving teachers’ content knowledge and subject-specific pedagogical skills as top priorities. Third, the Projects and their local sites retained the core organizational structure for delivering professional development. Having a network of providers across sites, within individual Projects, and across Projects remained a central strategy for building statewide capacity, sharing resources, maximizing limited resources, and avoiding “random acts of professional development.” The sites themselves continued to rely on a combination of summer institutes, follow-up to those institutes, and other types of events to meet the needs of partnership districts.
SRI Proprietary
79
Fourth, the content and form of professional development offered by the CSMP were consistent with research on effective professional development. Our analysis of secondary data suggests that, in general, CSMP-sponsored professional development was of reasonable duration, offered opportunities for groups of teachers to learn collectively, included a clear focus on content, provided opportunities for active learning, and was characterized by a coherent approach across activities. Fifth, addressing many basic questions about the operation, nature, and scope of the CSMP was compromised by the inadequacy of the OIS. A substantial portion of data was missing from the OIS, limiting its usefulness in producing conclusive findings. For example, it was impossible to determine the extent to which the CSMP met the needs of teachers of ELLs or teachers working to become “highly qualified.” Sixth, in spite of the Projects’ and sites’ commitment to evaluation, the evidence of CSMP’s impact on teachers and student achievement remained limited to specific sites in a few Projects. The evaluations conducted by CSMP sites tried to fulfill both formative purposes (helping the sites plan) and summative purposes (reporting results to external audiences). Frequently, the data collection, analysis, and reporting demands of those two goals differed. For example, summative judgments of effectiveness require more evidence than a site director might need to decide whether an event met its goals or a presenter should be asked to return. The types of evaluations that offer more conclusive evidence to outside observers often demand more sophisticated research designs, more painstakingly developed measurement tools, and more complex analyses than those necessary for local decision-making. As a result, these more complex studies required more capacity than most sites had. Nonetheless, several of the evaluations we received offered convincing evidence that particular sites and Projects had a strong, positive effect on participating teachers and their students. The challenge will be to develop a more comprehensive evaluation strategy that increases the prevalence of such studies without requiring sites to devote more of their scarce resources to evaluation.
SRI Proprietary
80
REFERENCES Agresti, A., & Finlay, B. (1997). Statistical methods for the social sciences (3rd ed.). Upper Saddle River, NY: Prentice Hall, Inc. American Institutes for Research (AIR). (2001). 2001 California Subject Matter Projects Site Director Survey data. Personal communication. Amaral, O. M., Garrison, L., & Klentschy, M. (2002). Helping English learners increase achievement through inquiry-based science instruction. Bilingual Research Journal, 26(2), 213-239. Birman, B. F., Desimone, L., Porter, A. C., & Garet, M. S. (2000). Designing professional development that works. Educational Leadership, 57(8) May. 28-33. California Writing Project. (n.d.). Documenting the effectiveness of the California Writing Project: Author. Dobb, F. (n.d.). Essential elements of effective science instruction for English learners. Retrieved 2004 from: http://csmp.ucop.edu/csp/ Garet, M. S., Birman, B. F., Porter, A. C., Desimone, L., Herman, R., & Yoon, K. S. (1999). Designing effective professional development: Lessons from the Eisenhower Program. Washington, DC: U.S. Department of Education, Office of the Undersecretary, Planning and Evaluation Service. Heller, J. I., Kaskowitz, S. R., Jaffe, R., De La O, A., & Alexander, K. A. (2003). The California Arts Project's standards-based teaching in the arts: No longer just "put your horn in your mouth and play." Unpublished manuscript. Heller, J. I., Kaskowitz, S. R., & Schecter, E. (2001). Sierra North Arts Project: 2000 invitational institute leadership study report. Unpublished manuscript. Hough, H., Tiffany-Morales, J., Price, T., Satele, C., Gallagher, H. A., & Shields, P. (2005). Evaluation of the California Subject Matter Projects: Technical Appendix. Unpublished Manuscript. Inland Area Science Project Summer Institute. (2003). Building biological standards. Unpublished manuscript. Kennedy, M. M. (1999). Form and substance in mathematics and science professional development. NISE Brief, 3(2). Land, R., & Olson, C. B. (2001). Evidence of effectiveness: California Writing Project/UC Irvine Writing Project. Unpublished manuscript. Loucks-Horsley, S. (1999). Research on professional development for teachers of mathematics and science: The state of the scene. School Science and Mathematics, 99(5), 258-271. Mark, A. (2002). 2002 California Science Project, Astronomy Institute evaluation. Unpublished manuscript. Medina, K., Pollard, J., Schneider, D., & Leonhardt, C. (2000). How do students understand the discipline of history as an outcome of teachers' professional development? Unpublished manuscript.
SRI Proprietary
R-1
National Writing Project. (2004). Improving student writing through effective classroom practices. Berkeley, CA: Author. Nordstrom, K., & Delgado, J. (2002). DIGS project final report and evaluation-Designing Integrated Garden Systems project. Unpublished manuscript. Oliver, M. W. (2000). Core evaluation report for LASERS Local Systemic Change Project: Final report (1998-99). Unpublished manuscript. Padilla, A. M., & Sung, H. (n.d.). The Stanford Foreign Language Oral Skills Evaluation Matrix (FLOSEM): A rating scale for assessing communicative proficiency. Unpublished manuscript, Stanford, CA. Putnam, R. T., & Borko, H. (2000). What do new views of knowledge and thinking have to say about research on teacher learning? Educational Researcher, 29(1), 4-15. Quick, H. E., Lieberman, J., Farr, B., Cole, S., Wang, J., Eaton, M., et al. (2002). Evaluation of the California Subject Matter Projects (CSMP). Unpublished manuscript. Scarloss, B. A., Gehlbach, H., Herring, R. B., & Freeman, J. M. (2003). Evaluating a professional development program from teacher workshop through student test scores. Paper presented at the American Educational Research Association annual meeting, Chicago, IL. Stoddart, T., Canaday, D., Clinton, M., Erai, M., Gasper, E., Gershon, A., et al. (1999). Language acquisition through science inquiry. Paper presented at the American Educational Research Association annual meeting, Montreal, Canada. Stoddart, T., & Clinton, M. (2002). Linking student learning to instructional practice: Findings from the LASERS Project. Unpublished manuscript. Stokes, L., Dickey, K., Heenan, B., Hirabayashi, J., Murray, A., & St. John, M. (2003). California Writing Project partnerships with schools: A study of benefits to teachers and students. Paper presented at the National Writing Project Annual Meeting, San Francisco, CA. United States Department of Education. (2004). NCLB: Title IX, Section 9101 (34). Retrieved 2004 from http://www.ed.gov/policy/elsec/leg/esea02/pg107.html#sec9101 University of California Office of the President (UCOP). (2003). California Subject Matter Projects budget allocations for 2000-01 to 2002-03. Personal communication. UCOP. (2004a). California Subject Matter Projects and sites, 2003-04. Retrieved 2004 from: http://csmp.ucop.edu/ UCOP. (2004b). California Subject Matter Projects budget allocations for 2003-04. Personal communication. UCOP. (2004c). Funding and Sites for the California Subject Matter Projects, 2000-01 to 200304. Personal communication. UCOP. (2004xx). University of California Davis Mathematics Project. Retrieved 2004 from: http://tepd.ucop.edu/csmp/institutedetail.php?programID=36070&viewing_mo=6&yr=2004& projectID=4
SRI Proprietary
R-2
APPENDIX A —SITE DIRECTOR SURVEY METHODS AND ANALYSIS This appendix details the design and procedures for the data collection methods and analyses of the 2004 CSMP Site Director Survey. The separate report, “Evaluation of the California Subject Matter Projects: Technical Appendix” (Hough, et al., 2005) provides technical information for exhibits throughout the report. SITE DIRECTOR SURVEY The 2004 CSMP Site Director Survey was designed to provide a representative report of the work of CSMP sites. A site director from each site was asked to report on a variety of topics, grouped into the following sections: • • • • • • •
Work of the site Teacher participants Teacher leadership Professional development provider characteristics Collaboration and support Evaluation Summary.
For most questions, respondents were asked to consider the period from July 1, 2003 to June 30, 2004. Sampling Procedures
UCOP provided a list of the 109 current site directors and codirectors from the OIS; one Arts Project site that had closed because of lack of funding was removed from the sample at the outset. Because we wanted only one response from each site, we sent only one e-mail notification of the online survey. In cases when a site had only one director, we sent the survey to that person, even if the site had multiple codirectors. When a site had multiple directors, we sent the e-mail to the director listed first on the Web site. Because we wanted to accurately represent the responses of a site rather than of a single site director, we asked site directors to collaborate with others in completing the survey. Survey Administration
The survey was administered using a Web-based survey tool. The first notification of the online survey was sent on October 19, 2004. Returned surveys were automatically logged. An automated reminder e-mail was sent on October 26, 2004 to the 75 nonresponding sites, and the deadline for completion of the survey was set for November 1, 2004. On November 1, another automated e-mail was sent to 29 sites that had not returned surveys, and the deadline was extended to November 5, 2004.
SRI Proprietary
A-1
On November 8, we began telephone follow-ups with the 20 sites that had not returned surveys. We used the contact information provided through UCOP, as well as the CSMP Web site, to contact the site directors and request their responses. Over the course of our follow-up, we removed three Physical Education-Health Project sites because they had not been funded and one Mathematics Project site because it did not offer summer programs or events for teachers during 2003-04. In addition, two Foreign Language and two International Studies Project sites did not respond to the survey. Exhibit A-1 details the final response rate. Exhibit A-1 Final Response Rate for 2003-04 CSMP Site Director Survey CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
All
Population
9
8
10
21
14
19
17
6
104
Responded
7
8
8
21
14
19
17
6
100
78%
100%
100%
100%
100%
100%
100%
96%
Response Rate
80%
Survey Analysis
Because we did not have a 100% response rate for all Projects, we weighted site directors’ responses by the number of sites in their Project. Exhibit A-2 details the weights for each Project. Exhibit A-2 Response Rate and Weights for 2003-04 Survey CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
Population
9
8
10
21
14
19
17
6
Responded
7
8
8
21
14
19
17
6
1.29
1.00
1.00
1.00
1.00
1.00
1.00
Weights
1.25
We conducted all survey analyses with the statistical software package SUDAAN, which uses a finite sampling adjustment in analysis. The finite sampling adjustment (Agresti & Finlay, 1997) is equal to zero when the sample size is equal to the population size, indicating that when all population members are respondents, there is no uncertainty with respect to the value in the population of the parameter being estimated. Consequently, if any two Projects have a finite sampling adjustment of zero, any observed differences between them are statistically significant SRI Proprietary
A-2
at the p = 0.000 level of significance. Because the chi-square statistic is testing for homogeneity of distribution across Projects, if any two Projects have a finite sampling adjustment of zero, the chi-square statistic is also statistically significant at the p = 0.000 level of significance. We conducted the following analyses: •
We examined the response distributions for each item and computed simple summary statistics.
•
For analysis of continuous variables, we used standard errors of the mean to assess the mean differences across Projects.
•
The longitudinal analyses consisted of comparing identical items across the 2003-04 site director survey, and the 1999-2000 and 2000-01 surveys conducted by the American Institutes for Research (AIR) (Quick et al., 2002). The process of our cross-sectional analysis is detailed below.
Cross-Sectional Analysis
We conducted the cohort analysis of the CSMP by using survey data from 2000 and 2001 collected for the previous evaluation conducted by AIR. Sites were weighted to the population of the Project at the time the survey was administered. Exhibits A-3 and A-4 show the Project populations in 2000-01 and 1999-2000, respectively, as well as the number of respondents and their assigned weights. AIR provided us with the 2000-01 survey dataset, which contained 113 responses. We note, however, that AIR reported a response of 114 site directors. In 2000-01, there were 3 Physical Education-Health sites, and all responded to the site director survey. All Physical EducationHealth sites were eliminated from the cross-sectional analysis because none were funded in 2003-04. These modifications changed our denominator to 126 and our number of valid responses to 110. Exhibit A-3 details the weights for each Project. Exhibit A-3 Weights for 2000-01 Survey CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
All
Population
9
17
16
22
16
18
18
10
126
Responded
8
13
16
19
13
16
17
8
110
1.13
1.31
1.00
1.16
1.23
1.13
1.06
1.25
-
Weights
AIR provided us with the 1999-2000 survey dataset, which contained 96 responses. According to AIR, there were 102 sites at the time of the 2000 survey, but 1 site that was not
SRI Proprietary
A-3
associated with a Project was eliminated from the population count because it could not be determined if this site actually existed. An additional Foreign Language site, which was a duplicate, was eliminated from the denominator. In 2000, 4 of the 5 Physical Education-Health sites responded to the survey. All Physical Education-Health sites were eliminated from the cross-sectional analysis because none were funded in 2003-04. These modifications changed our denominator to 95 and our number of valid responses to 92. Exhibit A-4 details the weights for each Project. Exhibit A-4 Weights for 1999-2000 Survey CFLP
CHSSP
CISP
CMP
CRLP
CSP
CWP
TCAP
All
Population
9
11
10
15
11
13
16
10
95
Responded
9
11
9
14
11
13
15
10
92
1.00
1.00
1.07
1.00
1.00
1.07
1.00
-
Weights
SRI Proprietary
1.11
A-4
APPENDIX B —REVIEW OF STUDIES
Exhibit B-1 Number of Studies that SRI Reviewed Studies Reviewed 9
Project TCAP CSP
20
CMP
3
CH-SSP
1
CWP
27
CRLP
0
CISP
3
CFLP
4
CH-PE
0 67
Total
STUDIES REVIEWED Amaral, O. M., Garrison, L., & Klentschy, M. (2002). Helping English learners increase achievement through inquiry-based science instruction. Bilingual Research Journal, 26(2), 213-239. Bay Area Environmental Science and Teaching Institute. (2003). Interim report of the BEST (Bay Area Environmental Science and Teaching) Institute. Hayward, CA: Author. Bay Area Environmental Science and Teaching Institute. (2004). Sheltered science instruction lessons: Make it, take it, secondary school teachers evaluation. Hayward, CA: Author. Bay Area Environmental Science and Teaching Institute. (2004). Summer science academy for teachers, grades 4, 5, 6 evaluation. Unpublished manuscript, Hayward, CA. Bequette, J., & Thibeault, M. D. (2002). Work in progress: Opus Two evaluation report. Unpublished manuscript. Bookmyer, J. (2002a). 2002 Sierra North Arts Project invitational research. Unpublished manuscript. Bookmyer, J. (2002b). California Arts Council demonstration grantees final report, 2002-2003. Unpublished manuscript. California International Studies Project. (2004). Shift happens: The impact of shifting personal goal orientations on educational outcomes. Unpublished manuscript. California Science Project. (2003). Sacramento Area Science Project: Evaluation report, summer 2003. Unpublished manuscript.
B-1
California Writing Project. (n.d.). California Writing Project at work. Snapshots of effectiveness and impact. Berkeley, CA: Author. California Writing Project. (n.d.). About the CWP Writing Improvement Initiative. Berkeley, CA: Author. California Writing Project. (n.d.). Models of successful professional development. Berkeley, CA: Author. California Writing Project. (n.d.). Programs of the CWP Writing Improvement Initiative. Berkeley, CA: Author. California Writing Project. (n.d.). Documenting the effectiveness of the California Writing Project. Berkeley, CA: Author. California Writing Project. (n.d.). Rationale for the CWP Writing Improvement Initiative. Berkeley, CA: Author. California Writing Project. (n.d.). Research and knowledge base for the CWP programs that support California's NCLB plan. Berkeley, CA: Author. Delgado, J. (2003). The Science, Culture, History and English Language Development (SCHELD) Project. Unpublished manuscript. Dickerson, A. (2004). AVANCE/MBSP Monterey Bay Science Project, residential summer academy 2004 final report. Unpublished manuscript. Dickey, K., Hirabayashi, J., St. John, M., Stokes, L., Murray, A., & Senauke, L. (2003). The National Writing Project: Client satisfaction and program impact, results from a satisfaction survey and follow-up survey of participants at 2002 invitational institutes. Inverness, CA: Inverness Research Associates. Available at http://www.invernessresearch.org/reports/nwp_cwp112003/NWP%20DOE%20%20follow%20up%20impact%20r pt122003.pdf. Donaldson, J., & Bookmyer, J. (2003). California Arts Council demonstration grantees final report, 2002-2003. Unpublished manuscript. Gehlbach, H. (2003). Motivating learners of different ability levels. Paper presented at the American Educational Research Association annual meeting, Chicago, IL. Hakansson, S. W. (2001). California Mathematics Project, 2001 site visit report. Unpublished manuscript. Hakansson, S. W. (2002). California Mathematics Project summer 2002 flagship institute site visit reports. Unpublished manuscript. Hakansson, S. W., Vierra, V., Keny, S., & Lester, T. (2000). California Mathematics Project, 2000 summer site visit reports. Unpublished manuscript. Heller, J. I., Kaskowitz, S. R., Jaffe, R., De La O, A., & Alexander, K. A. (2003). The California Arts Project's standards-based teaching in the arts: No longer just “put your horn in your mouth and play.” Unpublished manuscript. Heller, J. I., Kaskowitz, S. R., & Schecter, E. (2001). Sierra North Arts Project: 2000 invitational institute leadership study report. Unpublished manuscript.
SRI Proprietary
B-2
Heller, J. I., Kaskowitz, S. R., & Singer, N. (2003). The California Arts Project 2002-2003 invitational institutes evaluation report. Unpublished manuscript. Imperial Valley Science Project. (2004). Imperial Valley Science Project report. Unpublished manuscript. Inland Area Science Project Summer Institute. (2003). Building biological standards. Unpublished manuscript. Inverness Research Associates. (2002). NWP 2000-2001 site profiles. Inverness, CA: Author. Inverness Research Associates. (2003). Summary of evaluation and research findings. Inverness, CA: Author. Inverness Research Associates. (2004a). NWP 2002-2003 site profiles. Inverness, CA: Author. Inverness Research Associates. (2004b). Overview of the National Writing Project (NWP). Inverness, CA: Author. Inverness Research Associates. (2004c). Overview of the National Writing Project (NWP) by state-California. Inverness, CA: Author. Land, R., & Olson, C. B. (2001). Evidence of effectiveness: California Writing Project/UC Irvine Writing Project. Unpublished manuscript. Lieberman, A., & Wood, D. R. (2003). Inside the National Writing Project: Connecting network learning (1st ed.). New York: Teachers College Press. Lozano, A. S., Padilla, A. M., Sung, H., & Silva, D. (2004). A statewide professional development program for California foreign language teachers. Foreign Language Annals, 37(2), 268-277. Mark, A. (2002). 2002 California Science Project, Astronomy Institute evaluation. Unpublished manuscript. Mark, A. (2003). 2003 California Science Project “Our Changing Earth” institute evaluation. Unpublished manuscript. Medina, K., Pollard, J., Schneider, D., & Leonhardt, C. (2000). How do students understand the discipline of history as an outcome of teachers’ professional development? Unpublished manuscript. National Writing Project. (2002). National Writing Project in brief, improving student writing through effective classroom practices. Berkeley, CA: Author. National Writing Project. (2004a). Improving student writing through effective classroom practices. Berkeley, CA: Author. National Writing Project. (2004b). National Writing Project in brief: Improving student writing through effective classroom practices. Berkeley, CA: Author. National Writing Project. (n.d.). National Writing Project, program description. Berkeley, CA: Author. National Writing Project. (n.d.). National Writing Project 2002-2003 data at a glance. Berkeley, CA: Author.
SRI Proprietary
B-3
Nordstrom, K., & Delgado, J. (2002). DIGS project final report and evaluation-Designing Integrated Garden Systems project. Unpublished manuscript. Oliver, M. W. (2000). Core evaluation report for LASERS Local Systemic Change Project: Final report (1998-99). Unpublished manuscript. Olson, C. B. (n.d.). Pathway: Reading and writing development in ELLs. Unpublished manuscript. Padilla, A. M., Aninao, J. C., & Sung, H. (1996). Development and implementation of student portfolios in foreign language programs. Foreign Language Annals, 29(3), 429-438. Padilla, A. M., & Sung, H. (n.d.). The Stanford Foreign Language Oral Skills Evaluation Matrix (FLOSEM): A rating scale for assessing communicative proficiency. Unpublished manuscript, Stanford, CA. Parks, M. (2000). Assessment results: East Bay California Arts Project 1999-2000 (No. 2121). Berkeley, CA: Data Driven Decisions, Inc. Sacramento Area Science Project. (2003). Sacramento Area Science Project evaluation report, summer 2003. Unpublished manuscript. Scarloss, B. A., Gehlbach, H., Herring, R. B., & Freeman, J. M. (2003). Evaluating a professional development program from teacher workshop through student test scores. Paper presented at the American Educational Research Association annual meeting, Chicago, IL. Science & Health Education Partnership. (2004). Evaluation summary: Science & Health Education Partnership (SEP). San Francisco: University of California. SEASAND Summer Institute. (2003). SEASAND 2003: Participant tests. Unpublished manuscript. Sprinthall, N. A., Reiman, A. J., & Thies-Sprinthall, L. (1996). Teacher professional development. In J. Sikula, T. J. Buttery & E. Guyton (Eds.), Handbook of research on teacher education (2nd ed., pp. 666-703). New York: Macmillan. Stoddart, T., Canaday, D., Clinton, M., Erai, M., Gasper, E., Gershon, A., et al. (1999). Language acquisition through science inquiry. Paper presented at the American Educational Research Association annual meeting, Montreal, Canada. Stoddart, T., & Clinton, M. (2002). Linking student learning to instructional practice: Findings from the LASERS Project. Unpublished manuscript. Stoddart, T., Pinales, A., Canaday, D., & Latzke, M. (n.d.). Teachers’ conceptions of integrating science instruction and language development for English language learners. Unpublished manuscript. Stokes, L., Dickey, K., Heenan, B., Hirabayashi, J., Murray, A., & St. John, M. (2003). California Writing Project partnerships with schools: A study of benefits to teachers and students. Paper presented at the NWP Annual Meeting, San Francisco, CA. Sung, H., & Padilla, A. (1998). Student motivation, parental attitudes, and involvement in the learning of Asian languages in elementary and secondary schools. The Modern Language Journal, 82(ii), 205-216.
SRI Proprietary
B-4
The National Commission on Writing. (2004). Writing: A ticket to work...or a ticket out. A survey of business leaders. New York: College Entrance Examination Board. University of California Berkeley. (2003). National Writing Project 2003 annual report. Berkeley, CA: Author. Valadez, J. R. (n.d.). Evaluation of IIMPaC. Unpublished manuscript. Valadez, J. R. (n.d.). Evaluation of the IIMPaC Project. Unpublished manuscript. Valley Sierra California Arts Project. (2003). VSCAP NAEP adapted visual arts evaluation update. Unpublished manuscript. White, J. W., & Espinoza, D. M. (2003). Preliminary report, Summer Science Institute 2003, Redwood Science Project at Humboldt State University. Paper presented at the meeting of directors and program evaluators of the Redwood Science Project, and executive office of the California Science Project, Humboldt, CA. Woodbridge, M. W. (2003). IIMPaC: Professional development through inquiry, inservice workshops, models, practice and coaching. Evaluation report 2003. Santa Barbara, CA: University of California Santa Barbara.
SRI Proprietary
B-5
