Early Childhood Educ J (2015) 43:77–88 DOI 10.1007/s10643-014-0642-9
Instruction Via Web-Based Modules in Early Childhood Personnel Preparation: A Mixed-Methods Study of Effectiveness and Learner Perspectives Heidi L. Hollingsworth • Chih-Ing Lim
Published online: 22 April 2014 Springer Science+Business Media New York 2014
Abstract Effective personnel preparation is critical to the development of a high quality early childhood workforce that provides optimal care and education for young children. This mixed-methods study examined the effectiveness of, and learner perspectives on, instruction via web-based modules within face-to-face early childhood personnel preparation courses. The specific modules selected were designed around an evidence-based decision-making framework and focused on evidence-based practices: embedded interventions, assistive technology, and tiered instruction. Nineteen undergraduate students studying early childhood participated in pre- and post-instruction surveys and post-instruction focus groups. Participants’ written module activities provided additional sources of data. Statistical analyses of quantitative survey data and thematic analyses of qualitative survey data and focus groups were conducted. Results found the web-based modules to be effective, as indicated by learners’ knowledge and competence ratings, written explanations of the specific evidencebased practices, and completed assignments. Examination of learner perspectives indicated that learners perceived webbased modules to be effective over and above other instructional methods and preferred web-based modules to traditional instruction, at least to some degree. Learners found videos and realistic practice dilemmas to be the most useful module resources. They reported technology challenges and identified preferred supports, including more inH. L. Hollingsworth (&) Department of Education, Elon University, 2105 Campus Box, Elon, NC 27244, USA e-mail:
[email protected] C.-I. Lim FPG Child Development Institute, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
class discussion. Implications for personnel preparation and further research are discussed. Keywords Teacher education Instructional technology Professional development Early childhood education Inclusion
Introduction A well-prepared workforce is a critical component of high quality early childhood education (Buysse and Hollingsworth 2009), as it is early childhood personnel who implement the classroom practices and interactions that promote positive cognitive and social outcomes for children in their first years and on into elementary school (Peisner-Feinberg et al. 2001). In this article we report the results of a study in an undergraduate personnel preparation program investigating the effectiveness of instruction via web-based modules for increasing learners’ knowledge and competence regarding specific early childhood practices, as well as learner perspectives on instruction via web-based modules. We define a web-based module as a set of instructional resources focused on a single topic and accessible via the Internet. The article is organized beginning with a review of the literature on personnel preparation and web-based learning, followed by descriptions of the context and purpose of the study, and the modules selected. Participants, module implementation, data sources, and analyses are described in the ‘‘Methods’’ section. We then present results relating to effectiveness of instruction via web-based modules, followed by results relating to learner perspectives on web-based instruction. Finally, we discuss results in terms of implications for early childhood personnel preparation and suggestions for future research.
123
78
Review of Literature The literature suggests that effective personnel preparation is: (1) focused on specific instructional strategies rather than general content; (2) connected with program standards, curricula, and assessments; and (3) infused with active learning opportunities, guided practice, and corrective feedback (Darling-Hammond et al. 2009; Snyder et al. 2011; Zaslow et al. 2010). There is limited research on effective instructional approaches specific to early childhood teacher preparation, but research in K-12 teacher education suggests that microteaching, case method of instruction, and using interactive videos are promising strategies (Cochran-Smith and Zeichner 2005). Sheridan and Kelly (2012) recommended online scenario-based approaches to address the need to prepare early childhood professionals to solve complex challenges: ‘‘The most effective scenario-based approaches embed real-world relevance through the use of actual cases and vignettes that learners would likely encounter in job-related practice’’ (p. 74). This study examined instruction via specific web-based modules incorporating the characteristics of effective personnel preparation recommended above. Before describing the modules in detail, we present research on web-based instruction in general. There is a considerable body of literature examining teaching and learning online, but most of this literature is focused on distance education (Heirdsfield et al. 2007). However, Heirdsfield and colleagues noted increased use of online learning, even in on-campus courses. Lee and Choi (2008), for example, utilized web-based case instruction within a face-to-face undergraduate teacher education course. Their analysis of pre- and post-instruction essays indicated that after web-based case instruction, students had an increased awareness of multiple perspectives, tended to approach problems in more diverse ways (e.g., did not blame the parents or child, as in a deficit model), and used resources to justify their decision-making. Despite these positive results, their student participants did not make gains in considering the social and political contexts of education and classroom practices. Heirdsfield et al. (2007) discussed the importance of studying learners’ perceptions in order to design and facilitate effective learning experiences and alleviate barriers to learning. Their study on learners’ perspectives found that on-campus early childhood teacher education students valued the convenience, flexibility, usability, and accessibility of online learning, but reported issues with using technology and limited student-instructor interactions. Real-world relevance, opportunities to collaborate, and consideration of a variety of perspectives using a variety of resources have been outlined as characteristics of authentic activities that should be considered when choosing online learning experiences (Herrington et al. 2003). The
123
Early Childhood Educ J (2015) 43:77–88
modules we selected meet those criteria and are described in detail below. Context and Purpose of the Study CONNECT: The Center to Mobilize Early Childhood Knowledge, a project funded by the United States Department of Education (USDOE), developed the modules used in this study. They are free multi-media online modules that reflect research on effective professional development (PD) in that they are focused on specific practices, actively engage learners, and align with early childhood standards (Buysse et al. 2012). In addition, the modules were selected due to their evidence based nature, their relevance to course goals and objectives, and their familiarity to the authors due to past or present employment on the CONNECT project. The modules are evidence-based in two ways: (1) they focus on specific research-based practices for teaching and intervening in early childhood (Odom et al. 2005), and (2) they teach learners an evidence-based decision-making process for working with young children and families (Buysse et al. 2006). In each module, learners work through a 5-Step Learning CycleTM to solve dilemmas frequently encountered in early childhood education (Buysse et al. 2012). The process emphasizes solutions that integrate multiple perspectives and sources of evidence: research, policy, position statements, and families’ and professionals’ experiences. The module dilemmas tell realistic stories, promote empathy with individuals in the dilemma, are sufficiently complex to raise questions without containing unnecessary details, encourage learners to think and choose a position, and require a decision, thus meeting Barkley’s (2010) criteria of good case studies. Facilitation tips and assessment strategies provided for instructors suggest that instructors should act as ‘‘knowledge mediators’’ to support acquisition and application of knowledge and skills, even though all resources are web-based and open-access. Pilot and field tests with faculty at 2- and 4-year institutions of higher education determined that faculty found the modules to be of high quality, relevant, and useful (CONNECT 2013). However, there is a lack of learner data on the modules; one purpose of the current study was to examine learner feedback. A total of three modules were selected for implementation because they were closely aligned with course goals and objectives. Module 1: Embedded Interventions (Winton et al. 2010) and Module 5: Assistive Technology (Winton et al. 2011) were implemented within an undergraduate course on early childhood exceptionality. This course placed emphasis on research-based intervention practices in inclusive settings to address the needs of children with disabilities and their families, key content in early childhood teacher education (Chang et al. 2005; Horm et al. 2013; Maxwell et al. 2006). Course goals and
Early Childhood Educ J (2015) 43:77–88
objectives reflected this focus, and aligned well with module objectives [e.g., ‘‘Describe assistive technology interventions to promote children’s access to and participation in inclusive settings’’ (Winton et al. 2011)]. Module 7: Tiered Instruction (Buysse et al. 2012)—social emotional development component—was selected for implementation during the next semester within an undergraduate course on supporting social and emotional development in early childhood. Course goals and objectives aligned well with Module 7 objectives [e.g., ‘‘Describe effective tiered instruction related to social emotional development… for use with young children in early care and education programs’’ (Buysse et al. 2012)]. The purposes of this study were to examine the effectiveness of, and learner perspectives on, instruction via web-based modules. The following research questions guided data collection and analysis: (1) Are these webbased modules effective in increasing learners’ knowledge and competence ratings and written descriptions of module topics, and do learners successfully meet the learning objectives of the modules? (2) Do learners perceive the web-based modules to be effective and preferable to traditional classroom instruction, and which module resources do learners find most useful?
79
had been enrolled in both courses and thus completed all three modules. Module Implementation
Methods
Each module was implemented across approximately 1 week; two in-class sessions (1 h 40 min each) were devoted to each module. In addition, learners completed selected module tasks on their own between classes. The instructor briefly introduced each module in class and provided learners with a 1-page assignment sheet for each module, listing tasks for learners to complete outside of and in class (approximately half individually, and half in a small group). In class, learners engaged in small group and whole class discussions about the content and activities completed prior to class, and then continued with new content and activities. During implementation, module content was the only content covered (i.e., the instructor did not incorporate materials, readings, or assignments from other resources). Although learners proceeded through all steps of the 5-Step Learning CycleTM (Buysse et al. 2012), they were not expected to complete every written activity provided within each module. They completed 64–85 % of the standard written activities available online, depending on the module. Likewise, learners were not expected to view or listen to every handout, video, and audio clip available online. They were assigned 63–88 % of the latter resources per module.
Participants
Data Collection Sources
The participants in this study (hereafter referred to as learners) were 19 undergraduate students enrolled in one or both of the two semester-long courses in which modules were embedded, at a university in southeastern USA. All learners were part of a 4-year degree program, with Early Childhood as their major or minor field of study, except for one learner studying Elementary Education. All learners were female, 11 % were of Hispanic/Latino/Spanish origin, and 68 % were White, with others self-identifying as Black/ African American, Asian, Arab, or Multiracial. Learners ranged in age from 19 to 21 years (M = 19.0, SD = .8). The first author served as instructor for both courses, therefore, a different member of the research team collected consent forms—retaining the forms until after course grades were submitted at the end of each semester—and conducted focus groups in order to alleviate potential student concerns about participation in the study affecting their course grade. With these procedures in place, all learners enrolled in the two courses consented to participate in the study. Fourteen learners completed Modules 1 and 5 as part of the early childhood exceptionalities course, and 12 learners completed Module 7 as part of the course on supporting social and emotional development. Of the latter 12, five learners
We employed paper and pencil surveys and reviewed learners’ written module activities to answer research question one regarding effectiveness, and surveys and focus groups to examine learners’ perspectives (research question two). Whereas surveys were an efficient means for gathering and processing quantitative data (learners’ ratings of knowledge, competence, effectiveness, preference, and usefulness of module resources), focus groups allowed learners to provide detailed perceptions of instruction via web-based modules in their own words, in response to a consistent set of questions and to each other’s comments (Cohen et al. 2011; Creswell 2002). Instrumentation and Analysis The first author developed pre- and post-instruction surveys to examine both the effectiveness of, and learner perspectives on web-based instruction. Surveys consisted of three main sections: (1) ratings of knowledge and competence related to specific module content, (2) written test of knowledge on module topics, and (3) social validity and efficacy ratings of the modules as well as comments on whether learners recommend the use of web-based
123
80
modules. Pre-instruction surveys consisted of sections one and two; post-instruction surveys incorporated all three sections, followed by demographic items (see Appendix 1 for post-instruction survey for Modules 1 and 5). Written module activities (approximately 10 per module) were selected from those provided online within each module. The instructor collected and graded these written module activities upon completion of module implementation. Following the post-instruction survey, a focus group was conducted during a regular class session near the end of each course (n = 14 for the focus group following Modules 1 and 5; n = 12 for the focus group following Module 7). A member of the research team who was not the instructor conducted the focus groups. Focus groups lasted 10–15 min. Questions addressed learners’ perspectives on the use of web-based modules versus more traditional classroom instruction (e.g., readings, lectures, in-class discussions and activities); specific resources in the modules that were most helpful for learning the content; and additional supports that would have helped learners complete the modules (see Appendix 2 for focus group questions). Focus group discussions were recorded using a digital audio recorder and transcribed verbatim. Quantitative analyses of knowledge and competence ratings from survey section 1 and social validity data from survey section 3 were conducted using SPSS and are described in the ‘‘Results’’ section. Analysis of written test of knowledge data from survey section 2 was conducted as follows: Pre- and post- responses were transcribed, combined into one alphabetized list of responses by topic, and then compared with the appropriate module definition and assigned a score of 0, 1, or 2 (0 = blank/‘‘no idea’’/incorrect explanation; 1 = partially correct explanation/correct, but not detailed, no examples; 2 = correct, detailed explanation, may include examples). The first author and a graduate research assistant discussed criteria for a score of 0, 1, or 2 per topic, separately scored the responses, and then compared scores. Agreement among the two researchers’ scores ranged from 79 to 88 %, depending on the module. Disagreements in scoring were resolved through discussion. Finally, responses were sorted back into pre- and post-instruction categories and pre-post scores were compared. The authors analyzed qualitative social validity data (learner recommendations regarding use of web-based modules in courses) after analyzing focus group data and using the same codes (described below). The lead author graded written module activities by assigning a point for each correctly completed field on module activity forms [i.e., if an activity form had five fields for learner responses, a learner or small group of learners who completed all but one field correctly received 4/5 points (80 %) for that activity]. This process was repeated for all written activities, and yielded a final percentage for each learner for each module.
123
Early Childhood Educ J (2015) 43:77–88
We conducted a thematic analysis of focus group data through an iterative interpretative process of reducing data, drawing conclusions, and verifying interpretations (Miles and Huberman 1994). First, the authors, along with the graduate assistant, read the focus group transcripts separately and then met to discuss possible initial codes for identifying common information, as recommended by Creswell (2002). Closely aligned with the research questions, these initial codes were chosen to identify (a) learners’ perceived effectiveness of and preference for web-based modules over traditional classroom instruction, (b) module resources learners found most useful, and (c) learners’ preferred supports. We engaged in a consensus process to generate a final coding of focus group data under themes aligned with the research questions, as well as two other themes that emerged: a technology challenges theme, and a theme related to learners’ preference regarding the proportion of web-based instruction and traditional instruction within a course. Any disagreements in coding were resolved through discussion. The same codes and themes developed by the above procedure were used to code qualitative social validity data from survey section 3 (learners’ recommendations regarding use of web-based modules in courses). The focus in our study was not on counting the frequency of responses but on making meaning of learner responses to best understand their perspectives on instruction via webbased modules. Hence, results of the analysis are reported in terms of themes shared by a majority, several, or a few learners (Skinner et al. 1999).
Results Effectiveness of Instruction Via Web-Based Modules Module effectiveness was examined through surveys and learners’ written module activities. Survey results are presented first. Learners’ Knowledge and Competence Ratings were Significantly Higher Following Instruction Section 1 of both the pre- and post-instruction surveys asked learners to rate their level of knowledge and competence about individual module topics, and evidencebased practice and decision-making on a 5-point Likerttype scale (1 = none to very little—5 = a great deal). Paired-samples t tests (two-tailed) were conducted to compare the mean differences in learners’ knowledge and competence ratings before and after instruction via webbased modules. Cohen’s d was also used to calculate effect size estimates based on the pre-post instruction mean difference scores. Table 1 presents results showing that
Early Childhood Educ J (2015) 43:77–88
81
Table 1 Knowledge and competence ratings pre- and post-instruction via web-based modules Survey item
n
Premodule rating M (SD)
Postmodule rating M (SD)
M difference post–pre (SD)
t
df
p
d (r)
Level of knowledge about Embedded interventions in general
14
2.64 (.84)
4.14 (.54)
1.50 (.94)
5.97***
13 \.001
2.12 (.73)
Embedded interventions to support a young child’s social goals
14
2.50 (1.23)
4.07 (.48)
1.57 (1.28)
4.58*
13
.001
1.68 (.64)
Embedded interventions to support a young child’s communication goals
14
2.64 (1.28)
4.36 (.50)
1.71 (1.27)
5.06***
13 \.001
1.77 (.66)
Assistive technology interventions in general
14
2.21 (.89)
4.36 (.63)
2.14 (.95)
8.45***
13 \.001
2.79 (.81)
Assistive technology interventions to promote a young child’s access to and participation in inclusive settings
14
2.07 (.92)
4.21 (.70)
2.14 (1.10)
7.29***
13 \.001
2.62 (.79)
Tiered instruction in general
12
1.75 (1.06)
4.00 (.60)
2.25 (1.22)
6.41***
11 \.001
2.61 (.79)
Tiered instruction to enhance social emotional development in young children
12
1.83 (1.03)
3.75 (.75)
1.92 (1.38)
4.82**
11
.001
2.13 (.73)
Foundational social emotional practices
12
3.25 (.87)
4.00 (.74)
.75 (.75)
3.45**
11
.005
.93 (.42)
Targeted social emotional interventions/supports
12
3.08 (1.00)
3.83 (.58)
.75 (.87)
3.00*
11
.012
.92 (.42)
Evidence-based practices
19
2.68 (1.00)
3.89 (.81)
1.22 (1.40)
3.82**
18
.001
1.32 (.55)
Evidence-based decision-making
19
2.53 (1.26)
4.05 (.52)
1.53 (1.35)
4.93***
18 \.001
1.58 (.62)
Level of competence in Use of embedded interventions in general
14
2.21 (.80)
4.00 (.68)
1.79 (.70)
9.56***
13 \.001
2.41 (.77)
Use of embedded interventions to support a young child’s social goals
14
2.36 (1.00)
3.86 (.95)
1.50 (1.02)
5.51***
\.001
1.54 (.61)
Use of embedded interventions to support a young child’s communication goals
14
2.43 (.94)
3.86 (.86)
1.43 (.94)
5.70***
13 \.001
1.59 (.62)
Use of assistive technology interventions in general
14
2.36 (.75)
3.93 (.92)
1.57 (.85)
6.90***
13 \.001
1.87 (.68)
Use of assistive technology interventions to promote a young child’s access to and participation in inclusive settings
14
2.07 (.73)
3.86 (1.10)
1.79 (.70)
9.56***
13 \.001
1.92 (.69)
Use of tiered instruction in general
12
1.50 (.80)
3.67 (1.07)
2.17 (1.27)
5.92***
11 \.001
2.30 (.75)
Use of tiered instruction to enhance social emotional development in young children
12
1.58 (1.00)
3.58 (1.00)
2.00 (1.13)
6.14***
11 \.001
2.00 (.71)
Use of foundational social emotional practices
12
2.75 (.97)
3.50 (1.00)
2.46*
11
.032
.76 (.36)
Use of targeted social emotional interventions/supports
12
2.50 (1.00)
3.58 (.79)
1.08 (.79)
.75 (1.06)
4.73**
11
.001
1.20 (.51)
Use of evidence-based practices Use of evidence-based decision-making
19 19
2.68 (1.06) 2.42 (1.07)
3.58 (.61) 3.74 (.56)
.90 (1.05) 1.32 (1.11)
3.72** 5.18***
18 .002 18 \.001
1.04 (.46) 1.55 (.61)
* p \ .05; ** p \ .01; *** p \ .001
learners’ self-reported knowledge and competence were significantly higher for all survey items. Medium to high effect size estimates also suggest learner gains in knowledge and competence. It is important to note that learners’ knowledge and competence ratings in four items regarding evidence-based practices and decision-making in general were only analyzed in their first semester of module completion. In other words, for learners enrolled in both courses, only their first semester responses were included in the analysis for those four general items, as it was assumed that their encounter with modules during the previous semester would influence their responses during the second semester on the general items. All other survey
items addressed individual module topics (i.e., embedded interventions, assistive technology, tiered instruction) and thus responses of all learners in each course were analyzed. Learners’ Explanations of Module Practices were More Accurate and Detailed Following Instruction Section 2 on the pre- and post-instruction surveys—written test of knowledge—asked learners to provide brief explanations of module topics: embedded interventions, assistive technology interventions, and tiered instruction. Results displayed in Table 2 found that a majority of learners’ written test of knowledge scores increased following
123
82
Early Childhood Educ J (2015) 43:77–88
Table 2 Improvement in written explanations following instruction via web-based modules Survey item
n
Number of learners whose score improved
Example of pre-instruction response
Example of post-instruction response from same learner
Briefly explain what is meant by embedded interventions
14
10 (71 %)
Something about interventions with early childhood students within school and the home
Embedded interventions are interventions in daily activities with students with special needs that help meet developmental goals
Briefly describe assistive technology interventions
14
10 (71 %)
Briefly explain what is meant by tiered instruction
12
9 (75 %)
Using different types of media to help a child, like computer games or disks… Separating instruction into different levels
Assistive technology interventions are devices or systems or services that help a child with disabilities to be more independent and participate Instruction that includes foundational knowledge for all students and specialized interventions for students identified as having need
instruction. Table 2 also provides responses from three different learners, illustrating their improved explanations.
Learners Preferred Web-Based Modules to Traditional Classroom Instruction, to Some Degree
Learners’ Written Module Activities Showed Comprehension of Module Topics
The survey also included an item asking learners to respond to the statement ‘‘I would have preferred to learn about [module topic] through more traditional course instruction instead of the online module’’. Using the same rating scale as above, learners did not agree with that statement for embedded interventions [M = 2.79 (SD = 1.19)], assistive technology interventions [M = 2.71 (SD = 1.38)], or tiered instruction [M = 2.83 (SD = 1.40)], although their responses were close to the ‘‘neither agree nor disagree’’ rating. Similar results were found in response to a question asking whether learners would like to complete online modules in future courses. Learners’ mean ratings fell between the ‘‘agree’’ and ‘‘neither agree nor disagree’’ response options following Modules 1 and 5 [M = 3.79 (SD = 1.25), n = 14], and Module 7 [M = 3.42 (SD = 1.08), n = 12]. Focus group results and qualitative social validity data aligned with survey results and provided information beyond that gleaned from surveys through the rationales learners articulated for their preferences. Similar to the surveys, a few learners noted a preference for web-based modules over traditional instruction (e.g., ‘‘It was just more interesting… and we actually had to respond to what we read, so it really was more engaging than in-class activities for me’’), while a few others preferred traditional instruction. Another theme that emerged from focus group and qualitative social validity data was that there should be a balance between web-based instruction and traditional classroom instruction. Several learners expressed this (e.g., ‘‘I enjoyed [Modules 1 and 5], but I would not want every chapter—every topic we focused on—to be a module. I thought it was good to have just maybe one or two in there’’).
The final pieces of evidence for the effectiveness of instruction via web-based modules were learners’ written module activities. Final grades for written module activities ranged from 83 to 98 % for Module 1 (Embedded Interventions), from 90 to 100 % for Module 5 (Assistive Technology), and from 80 to 100 % for Module 7 (Tiered Instruction), indicating good comprehension of module content. Learner Perceptions of Instruction Via Web-Based Modules Learner perceptions of modules were examined through surveys and focus groups. Results of both surveys and focus groups are presented next. Learners Perceived Web-Based Modules to be Effective Over and Above Other Instructional Methods Learners’ perspectives on the effectiveness of these modules were gauged via ratings in section three on the postinstruction survey (1 = strongly disagree—5 = strongly agree). Mean ratings indicated general agreement among learners that web-based instruction helped them learn about module topics (particularly for Modules 5 and 7), and helped them learn to solve real-life problems and integrate information for evidence-based decision-making, over and above other instructional methods. These results are presented in Table 3.
123
Early Childhood Educ J (2015) 43:77–88 Table 3 Learner perceptions of module effectiveness
83
Survey items
n
M (SD)
Working through Module 1 helped me learn about embedded interventions (over and above what I learned through course activities, instruction, and readings that were not part of the module)
14
3.93 (.62)
Working through Module 5 helped me learn about assistive technology interventions (over and above…) Working through Module 7 helped me learn about tiered instruction (over and above…)
14
4.29 (.61)
12
4.00 (1.04)
Working through the modules [1 and 5] helped me learn how to solve real-life problems related to working with young children and families (over and above…) Working through the modules [1 and 5] helped me learn to integrate information from various sources in order to make evidence-based decisions (over and above…)
14
3.86 (1.03)
14
3.79 (.98)
Working through Module 7 helped me learn how to solve real-life problems related to working with young children and families (over and above…)
12
3.67 (.49)
Working through Module 7 helped me learn to integrate information from various sources in order to make evidence-based decisions (over and above…)
12
3.58 (.79)
Items relating to targeted practices
Items relating to problem-solving and decision-making
Learners Found Specific Module Resources to be Highly Useful, and Identified Preferred Supports The survey asked learners to rate the usefulness of module resources for their learning (1 = not at all useful—5 = very useful). Mean ratings for all resources rounded to the ‘‘somewhat useful’’ and ‘‘very useful’’ response ratings. Following is a ranking of resources from most to least useful according to results across the two semesters: (1) video clips (M = 4.65), (2) audio clips (M = 4.31), (3) written module activities completed in class (M = 4.00), (4) module-specific whole group discussions in class (M = 3.96), (5) written module activities completed outside of class (M = 3.92), (6) module-specific small group discussions in class (M = 3.92), (7) written module text (M = 3.69), and (8) handouts (M = 3.65). Focus group results provided support for the survey findings described above. There was consensus among learners that the videos were the most helpful module resources, along with the dilemma in each module: I felt the videos were… engaging because of how they followed the same storyline, so you got to hear each person’s perspective on the same story, rather than seeing one kid do something and then seeing a different kid do something else, then hearing a professional talk just generally about the subject. You got to see how it all comes together to work. Analysis of the qualitative social validity data (learner recommendations) further revealed that what many learners appreciated about module resources was their realistic nature. This was especially true for Modules 1 and 5, after which a majority of learners made comments similar to this: ‘‘It’s helpful to see the videos, hear the real people
talking about what we were learning.’’ This learner also noted appreciation for the 5-Step Learning CycleTM around which CONNECT modules are designed: ‘‘I recommend [it] because it allows for real-life applications as well as it almost serves as a case study because we assess the dilemma and use [resources within] the module to find solutions.’’ Handouts were ranked last according to survey ratings, but when asked about helpful module resources, there was general consensus in the focus group discussion that the handouts with pictorial examples of assistive technology in Module 5 were especially helpful. Finally, though a few learners did not find the written module activities to be helpful and suggested that they were repetitive and seemed like busy work, some indicated that the culminating activity integrating information in order to make a decision was a useful activity: ‘‘I think the last worksheet we did was helpful because it kind of put everything together… actually synthesized things and we had to come up with our own conclusions.’’ In each focus group, a few learners provided feedback on preferred supports that were or could be provided by the instructor and supports that were or could be provided by module developers. With regard to instructor supports, learners found the combination of at-home and in-class tasks and the assignment sheet listing assigned tasks to be helpful. However, learners said more in-class support was needed, specifically more opportunities for discussion. Regarding module developer supports, learners suggested visual layout changes for ease of navigation and clarification of jargon such as ‘‘Section 504.’’ A final theme of focus group data—noted by several learners—was technology challenges. This was also noted in qualitative social validity data. Learners expressed frustration that following
123
84
a link within a written activity form (e.g., to a handout or a video) led to the activity form window closing without saving the answers they had already written. They wanted to have the ability to keep several windows open simultaneously.
Discussion This study found that web-based modules were effective: learners’ knowledge and competence ratings for module topics improved significantly, and learners successfully met the learning objectives of the modules, as indicated by their written explanations of specific practices and final grades for module activities. Moreover, learners in this study perceived instruction via web-based modules to be effective over and above other instructional methods. However, learners were split in their preference for webbased modules versus traditional instruction, and several recommended a balance of both modes of instruction. A meta-analysis of studies examining effects of online instruction on objective measures of student learning also found that supplementing face-to-face instruction with online instruction was effective in enhancing learning (USDOE 2010), and that a blend of face-to-face and online elements in higher education instruction was more effective than either traditional face-to-face or purely online instruction. Some learners in our study found web-based modules to be more engaging than traditional instruction, perhaps because module implementation incorporated active learning, interactions and activities in and out of class, collaborative learning, exposure of students to diversity, and use of technology; all indicators of student engagement described by Barkley (2010). Similar to a study of 816 in-service early childhood professionals’ use of the internet (Weigel et al. 2012), and several studies in K-12 pre-service teacher education (Grossman 2005), this study found that instructional videos were one of the desired features of online learning. Specifically, learners in our study were positive about videos showing realistic early childhood dilemmas and real life examples of how to carry out specific practices. Grossman observed that even though instructional videos seem to hold great promise, the field needs more information about what features of video demonstrations are most important in helping learners learn about classroom practices. Learners may enjoy them, but some research indicates videos do not actually increase the amount that students learn through online instruction (USDOE 2010). Further research is needed in this area. Finally, technological challenges were a barrier and source of frustration for some learners, despite written instructions from the instructor and the fact that these
123
Early Childhood Educ J (2015) 43:77–88
learners were sufficiently young to be considered ‘‘digital natives.’’ Taken together with learners’ preference for a balance of in-class and web-based instruction, these findings support higher education research suggesting that digital natives are actually rather heterogeneous in both their ability to use technology and their preference for learning using technology (Bennett et al. 2008; Kennedy et al. 2008). There are some limitations of this study that lead to suggestions for future research. We included a relatively small sample size of undergraduate students from one higher education early childhood program, and examined three specific web-based modules by the same developers. Other web-based modules exist and were not examined in this study, but could be examined in future research with larger samples across more programs. We have provided details about the modules, the courses in which the modules were embedded, the learners, and our implementation procedures in order to allow readers to determine the similarity and thus the relevance of the evidence to their own circumstances (Brantlinger et al. 2005). Our approach relied to a large degree on learner self-report, and learners may have provided responses they felt were expected. Although agreements and consistencies across surveys, written answers to content-related questions, and focus group responses provided some measure of triangulation for individuals’ responses, to further document the effectiveness of instruction via web-based modules, future research could observe learners’ implementation of target practices and evidence-based decision-making with young children and families in early childhood settings.
Conclusion The current study contributes to the knowledge base about use of web-based instruction, particularly as a component of a face-to-face course. Taken together with findings from other studies, our finding that instruction via web-based modules was both effective and acceptable to learners underscores the potential of using this type of instruction within early childhood personnel preparation. This information could be useful for higher education faculty and other PD providers considering the use of web-based instruction. Similar to the USDOE (2010) meta-analysis finding that collaborative or instructor-directed online instruction was more effective than purely independent, self-directed online instruction, our results indicated that instructor support is important. This underscores the role of higher education instructors and other PD providers as knowledge mediators, even for information already present in web-based modules, and especially when learners are first engaging in online learning (Herrington et al. 2003).
Early Childhood Educ J (2015) 43:77–88
Learners in this study appreciated being able to see real people demonstrating specific evidence-based practices. This aspect of instruction is particularly important, given that early childhood personnel preparation programs often struggle to find high quality field experience sites where their learners can see evidence-based practices in action (Miller et al. 2003). Although not a substitute for high quality practicum experiences, web-based modules with realistic dilemmas involving children and families may
85
supplement other field experiences to help learners understand and learn to implement evidence-based practices. Acknowledgments The authors wish to acknowledge Marna Winter for assistance with consent forms and focus groups, and Xueyan Yang for assistance with scoring written test of knowledge survey data and coding of focus group data.
Appendix 1 Post-instruction Survey—Modules 1 and 5
ID code
POST-INSTRUCTION SURVEY – MODULES 1 & 5
None to very little
A little, but not much
Some
Quite a bit
A great deal
Thank you for taking the time to complete this survey! The survey asks about your level of knowledge and your level of competence with regard to some early childhood topics and practices, and asks about your responses to instruction using online modules. The results will only be reported in group form and you do not need to write your name on this survey. Thanks again for your help with this project!
Evidence-based practice
1
2
3
4
5
Evidence-based decision-making
1
2
3
4
5
Embedded interventions in general
1
2
3
4
5
Embedded interventions to support a young child’s social goals
1
2
3
4
5
Embedded interventions to support a young child’s communication goals
1
2
3
4
5
Assistive technology interventions in general Assistive technology interventions to promote a young child’s access to and participation in inclusive settings
1
2
3
4
5
1
2
3
4
5
Use of evidence-based practice
1
2
3
4
5
Use of evidence-based decision-making
1
2
3
4
5
Use of embedded interventions in general
1
2
3
4
5
Use of embedded interventions to support a young child’s social goals Use of embedded interventions to support a young child’s communication goals Use of assistive technology interventions in general
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
Use of assistive technology interventions to promote a young child’s access to and participation in inclusive settings
1
2
3
4
5
Circle a number to indicate your level of knowledge about:
Circle a number to indicate your level of competence in:
Briefly explain what is meant by embedded interventions
Briefly describe assistive technology intervention
123
Disagree
Neither agree nor disagree
Agree
Strongly agree
Early Childhood Educ J (2015) 43:77–88
Strongly disagree
86
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
Very useful
Written module text
Somewhat useful
How useful were these module components for learning about the topics and practices in the modules?
Neutral
I would like to complete online modules in future courses
Somewhat useless
Working through Module 1 helped me learn about embedded interventions (over & above what I learned through course activities, instruction, & readings that were not part of the module) Working through Module 5 helped me learn about assistive technology interventions (over & above…) Working through the modules helped me learn how to solve real -life problems related to working with young children and families (over & above…) Working through the modules helped me learn to integrate information from various sources in order to make evidence-based decisions (over & above…) I would have preferred to learn about embedded interventions through more traditional course instruction instead of the online module I would have preferred to learn about assistive technology interventions through more traditional course instruction instead of the online module
Not at all useful
Circle a number to indicate your response
1
2
3
4
5
Handouts
1
2
3
4
5
Video clips
1
2
3
4
5 5
Audio clips
1
2
3
4
Written module activities completed outside of class
1
2
3
4
5
Written module activities completed in class
1
2
3
4
5
Module-specific small group discussions in class
1
2
3
4
5
Module-specific whole group discussions in class
1
2
3
4
5
Briefly comment on why you do or do not recommend use of online modules in courses
123
Early Childhood Educ J (2015) 43:77–88
87
ABOUT YOU Age
ID code
years
Sex Female Male Are you of Hispanic, Latino, or Spanish origin? Yes No Race (Please check all that apply) White Black, African American American Indian or Alaska Native Asian Pacific Islander Other (please specify): ______________________________ Year (Check one) Freshman Sophomore Junior Senior In which program are you enrolled? (Check one) Early Childhood Education Major Early Childhood Minor Other (please specify): ______________________________ Did you have experience working with children ages 0-8 prior to enrolling at this university? Yes No What was your role? (e.g., babysitter, childcare assistant, summer camp volunteer, etc.)?
technology interventions, and evidence-based practice decision-making)?
Appendix 2 Focus Group Questions—Modules 1 and 5 (1)
What are your thoughts about the use of web-based modules instead of more traditional classroom instruction (like readings, lectures, and in-class discussions and activities)? Possible follow-up question, if needed to get the discussion going: Did you find the online modules more accessible, informative, and/or appealing than more traditional classroom instruction?
(2)
What specific aspects of the modules were most helpful for your learning about the topics and practices of the modules (i.e., embedded interventions, assistive
Possible follow-up question—only ask if these specific aspects do not come up in the discussion: For example, were the video clips or handouts with examples of embedded interventions especially helpful? (3)
The final question is: What additional supports would have helped you complete the online modules?
References Barkley, E. F. (2010). Student engagement techniques: A handbook for college faculty. San Francisco, CA: Jossey-Bass.
123
88 Bennett, S., Maton, K., & Kervin, L. (2008). The ‘digital natives’ debate: A critical review of the evidence. British Journal of Educational Technology, 39, 775–786. Brantlinger, E., Jimenez, R., Klingner, J., Pugach, M., & Richardson, V. (2005). Qualitative studies in special education. Exceptional Children, 71, 195–207. Buysse, V., Epstein, D., Winton, P., & Rous, B. (2012). CONNECT Module 7: Tiered instruction [Web-based PD curriculum]. Chapel Hill: University of North Carolina, FPG Child Development Institute, CONNECT: The Center to Mobilize Early Childhood Knowledge (CONNECT). Buysse, V., & Hollingsworth, H. L. (2009). Program quality and early childhood inclusion: Recommendations for professional development. Topics in Early Childhood Special Education, 29, 119–128. Buysse, V., Wesley, P., Snyder, P., & Winton, P. (2006). Evidencebased practice: What does it really mean for the early childhood field? Young Exceptional Children, 9(4), 2–11. Buysse, V., Winton, P., Rous, B., Epstein, D. J., & Lim, C. (2012b). Evidence-based practice: Foundation for the CONNECT 5-Step Learning CycleTM in professional development. Zero to Three, 32(4), 25–29. Chang, F., Early, D. M., & Winton, P. J. (2005). Early childhood teacher preparation in special education at 2- and 4-year institutions of higher education. Journal of Early Intervention, 27, 110–124. Cochran-Smith, M., & Zeichner, K. M. (Eds.). (2005). Studying teacher education: The report of the AERA panel on research and teacher education. New York: Routledge. Cohen, L., Manion, L., & Morrison, K. (2011). Research methods in education (7th ed.). New York, NY: Routledge. CONNECT. (2013). CONNECT: The center to mobilize early childhood knowledge annual grant performance report. Chapel Hill, NC: Author. Creswell, J. W. (2002). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson Education. Darling-Hammond, L., Wei, R. C., Andree, A., Richardson, N., & Orphanos, S. (2009). Professional learning in the learning profession: A status report on teacher development in the United States and abroad. Dallas, TX: National Staff Development Council. Grossman, P. (2005). Pedagogical approaches in teacher education. In M. Cochran-Smith & K. M. Zeichner (Eds.), Studying teacher education: The report of the AERA panel on research and teacher education (pp. 425–476). New York: Routledge. Heirdsfield, A., Davis, J., Lennox, S., Walker, S., & Zhang, W. (2007). Online learning environments: What early childhood teacher education students say. Journal of Early Childhood Teacher Education, 28, 115–126. Herrington, J., Oliver, R., & Reeves, T. C. (2003). Patterns of engagement in authentic online learning environments. Australian Journal of Educational Technology, 19, 59–71. Horm, D. M., Hyson, M., & Winton, P. (2013). Research on early childhood teacher education: Evidence from three domains and recommendations for moving forward. Journal of Early Childhood Teacher Education, 34, 95–112. Kennedy, G. E., Judd, T. S., Churchward, A., Gray, K., & Krause, K.L. (2008). First year students’ experiences with technology: Are
123
Early Childhood Educ J (2015) 43:77–88 they really digital natives? Australasian Journal of Educational Technology, 24, 108–122. Lee, K., & Choi, I. (2008). Learning classroom management through web-based case instruction: Implications for early childhood teacher education. Early Childhood Education Journal, 35, 495–503. Maxwell, K., Lim, C.-I., & Early, D. M. (2006). Early childhood teacher preparation programs in the United States: National report. Chapel Hill: University of North Carolina, Frank Porter Graham Child Development Institute. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage. Miller, P., Ostrosky, M., Laumann, B., Thorpe, E., Sanchez, S., & Fader-Dunne, L. (2003). Quality field experiences underlying performance mastery. In V. D. Stayton, P. S. Miller, & L. S. Dinnebeil (Eds.), Personnel preparation in early childhood special education: Implementing the DEC recommended practices (pp. 113–138). Missoula, MT: DEC. Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R. (2005). Research in special education: Scientific methods and evidenced-based practices. Exceptional children, 71, 137–148. Peisner-Feinberg, E. S., Burchinal, M. R., Clifford, R. M., Culkin, M. L., Howes, C., Kagan, S. L., et al. (2001). The relation of preschool child-care quality to children’s cognitive and social developmental trajectories through second grade. Child Development, 72, 1534–1553. Sheridan, K. M., & Kelly, M. A. (2012). Teaching early childhood education students through interactive scenario-based course design. Journal of Early Childhood Teacher Education, 33, 73–84. Skinner, D., Rodriguez, P., & Bailey, D. B. (1999). Qualitative analysis of Latino parents’ religious interpretations of their child’s disability. Journal of Early Intervention, 22, 271–285. Snyder, P., Hemmeter, M. L., & McLaughlin, T. (2011). Professional development in early childhood intervention: Where we stand on the silver anniversary of PL 99-457. Journal of Early Intervention, 33, 357–370. United States Department of Education (USDOE), & Office of Planning, Evaluation, and Policy Development. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Author. Weigel, D.J., Weiser, D. A., Bales, D. W., & Moyses, K. J. (2012). Identifying online preferences and needs of early childhood professionals. Early Childhood Research and Practice, 14(2). http://ecrp.uiuc.edu/v14n2/weigel.html. Winton, P., Buysse, V., Rous, B., Epstein, D., & Pierce, P. (2011). CONNECT Module 5: Assistive technology [Web-based PD curriculum]. Chapel Hill: University of North Carolina, FPG Child Development Institute, CONNECT. Winton, P. J., Buysse, V., Turnbull, A., Rous, B., & Hollingsworth, H. (2010). CONNECT Module 1: Embedded interventions [Webbased PD curriculum]. Chapel Hill: University of North Carolina, FPG Child Development Institute, CONNECT. Zaslow, M., Tout, K., Halle, T., Whittaker, J. V., & Lavelle, B. (2010). Toward the identification of features of effective professional development for early childhood educators: Literature review. Washington, DC: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service.
