Evaluation of an eLearning tool for training behavioral ...

Research in Autism Spectrum Disorders 4 (2010) 11–17

Contents lists available at ScienceDirect

Research in Autism Spectrum Disorders Journal homepage: http://ees.elsevier.com/RASD/default.asp

Evaluation of an eLearning tool for training behavioral therapists in academic knowledge of applied behavior analysis Doreen Granpeesheh, Jonathan Tarbox, Dennis R. Dixon *, Catherine A. Peters, Kathleen Thompson, Amy Kenzer Center for Autism and Related Disorders, United States

A R T I C L E I N F O

A B S T R A C T

Article history: Received 8 July 2009 Accepted 22 July 2009

Applied Behavior Analytic (ABA) treatment for children with autism spectrum disorders (ASDs) has been demonstrated to produce significant treatment effects across multiple studies (Eikeseth, 2009; Eldevik et al., 2009). Despite the demonstrated effectiveness of ABA, there continues to be a lack of trained clinicians to provide the treatment. Traditional training approaches include lecture, group discussion, and role-playing formats. However, alternative training approaches may need to be developed to increase the efficiency of training and/or to disseminate training opportunities to remote regions. One such possibility is eLearning, wherein trainees interact with computer training programs, rather than, or in addition to, live human trainers. Despite the potential contributions of eLearning training approaches, no previous research of which we are aware has evaluated eLearning approaches to training behavioral therapists in ABA principles and procedures for the treatment of autism. In this study, we evaluated an eLearning tool that was developed for training newly hired behavioral therapists in academic knowledge of ABA treatment for children with autism. We compared outcomes for a group of trainees who received eLearning training to a group who received traditional didactic training. Knowledge of ABA principles and procedures increased substantially for both groups, with the traditional didactic group achieving scores slightly higher than the eLearning group. These preliminary results suggest that eLearning tools may be a useful strategy for extending training in ABA principles and procedures to settings in which limited or no contact with live professional trainers may be available. ß 2010 Elsevier Ltd. All rights reserved.

Keywords: Autism spectrum disorders Applied behavior analysis eLearning Staff training

* Corresponding author at: 19019 Ventura Blvd., Tarzana, CA 91356, United States. Tel.: +1 818 345 2345; fax: +1 818 758 8015. E-mail address: [email protected] (D.R. Dixon). 1750-9467/$ – see front matter ß 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.rasd.2009.07.004

12

D. Granpeesheh et al. / Research in Autism Spectrum Disorders 4 (2010) 11–17

Applied Behavior Analytic (ABA) treatment for children with autism spectrum disorders (ASDs) has been demonstrated to produce significant treatment effects across multiple studies (Eikeseth, 2009; Eldevik et al., 2009). Despite the demonstrated effectiveness of ABA, there continues to be a lack of qualified clinicians to provide the treatment. A foundational prerequisite for the implementation of any procedure with high fidelity is good quality training of clinical staff. In a description of a researchbased service model of ABA for children with ASD, Hayward, Gale, and Eikeseth (2009) list quality staff training as one of the essential components to an effective program. Training of staff in the principles and procedures of ABA involves training in both academic knowledge of ABA and the establishment of proficiency in the actual implementation of the procedures. One component of ABA treatment for ASD is discrete trial training (DTT). Thomson, Martin, Arnal, Fazzio, and Yu (2009) recently reviewed instruction techniques for DTT. Training approaches that have been demonstrated to be effective in establishing proficiency in procedural implementation typically include written instruction, lecture, video modeling, role-playing and in vivo feedback. For example, Sarokoff and Sturmey (2004) used instruction, modeling, rehearsal, and feedback to train special education teachers to implement DTT. In addition, Lafasakis and Sturmey (2007) used a similar procedure for training parents of children with autism to implement DTT. More recently, Fazzio, Martin, Arnal, and Yu (2009) reported on the use of a self-instructional manual in conjunction with feedback and demonstration to train university students to implement DTT with children with ASD. Although these findings are related to the training of DTT techniques, they are likely applicable to the training of staff in the broader context of service delivery. For example, van Oorsouw, Embregts, Bosman, and Jahoda (2009) recently conducted a meta-analysis of research on staff training and evaluated the relative effectiveness of commonly used training components. They found that a combination of in service training and ‘‘coaching on the job’’ was required to teach skills related to changing actual client behavior. In other words, the combination of both didactic instructions along with in vivo feedback is the best approach for teaching client-related skills to staff. In addition to training in the actual implementation of clinical procedures, training in ABA (and indeed virtually any other approach) includes training in academic or conceptual knowledge of the approach. Conceptual or academic training might be particularly relevant to training in ABA because virtually all of the procedures used are directly derived from and logically linked to basic principles of learning and motivation. That is, procedures in ABA are related to basic processes in a ‘‘conceptually systematic’’ manner (Baer, Wolf, & Risley, 1968, 1987). Therefore, when procedures work, clinicians are able to account for their success in terms of the basic processes that are at play. More importantly, however, is that a solid conceptual basis in behavioral principles and procedures aids the clinician in problem-solving when clinical procedures are not working. For example, if a child with ASD is not learning a particular skill that is being taught, an analysis of the problem may be made in terms of magnitude of reinforcement, delay to reinforcement, inadvertent sources of stimulus control, stimulus over-selectivity, etc. Training in academic knowledge of ABA principles and procedures should never be presented as superior to or a replacement for practical training, but it is generally considered a critically important component of the overall process of training ABA clinicians. Despite the generally acknowledged importance of training in academic knowledge of ABA, relatively few studies have been published which examine the optimal ways in which such training may be done. Traditional approaches to training academic knowledge of ABA generally involve lecture and group discussion. Traditional approaches generally work well when an expert level trainer is available for inperson training. However, alternative training approaches may need to be developed to increase the efficiency of training and/or to disseminate training opportunities to remote regions. One such possibility is eLearning. During eLearning, trainees interact with computer programs, rather than (or in addition to) live human trainers. There are multiple potential advantages and disadvantages to eLearning instruction, relative to live in-person training. One potential disadvantage to eLearning instruction is that an instructor is not present to answer questions that trainees might have, whereas live lecture-based training may accommodate for that (assuming the trainer agrees to answer questions as he/she lectures). A potential advantage of eLearning approaches to training is that these approaches do not require an expert trainer to be physically present while staff are trained. Given the continual rise in the diagnosis of autism (Hertz-Picciotto & Delwiche, 2009) and the already-existing lack of qualified clinicians to treat them, innovations are needed that may facilitate the efficiency of training clinicians.


13

eLearning approaches might be more efficient because multiple trainees could be trained from any location that contains a personal computer, without the need for scheduling multiple personnel to be in the same place at the same time and without the need for expert trainers to travel to various training sites. Several previous studies have evaluated eLearning approaches to training a variety of skills and the results thus far have been variable. Some studies have found significant differences in outcomes when comparing eLearning and traditional approaches. For example, Wright (2008) compared traditional instruction to eLearning instruction for teaching an introductory university course in environmental sciences and found that both methods produced significant improvement in knowledge about environmental sciences but that the traditional group gained an average of approximately six points more than the eLearning group, on a 100 point test, and that this difference between groups was statistically significant. However, Schimming (2008) compared traditional in-person instruction to eLearning for teaching first year medical students PubMed skills and found that the two approaches produced identical mean post-test scores, but that student satisfaction was higher in the eLearning condition. In a comparison of traditional and eLearning instruction in the workplace, Petty, Lim, and Zulaf (2007) found that trainee self-reports of transfer of training produced by the two methods was identical. Despite the potential contributions of eLearning training approaches, no previous research of which we are aware has evaluated eLearning approaches to training behavioral therapists in ABA principles and procedures for the treatment of autism. In this study, we evaluated an eLearning tool that was developed for training newly hired behavioral therapists in academic knowledge of ABA treatment for children with autism. We compared the acquisition of academic knowledge of ABA principles and procedures for a group of trainees who received eLearning training to a group who received traditional didactic training. 1. Method 1.1. Experimental design and group assignment A randomized between groups design was used. Ten training sites across 4 states (Arizona, California, Texas, and Virginia) were included in the study. Each time a training site scheduled a cohort of new employees to be trained during the period of time in which the study was conducted, that cohort was randomly assigned to either the eLearning training group or the standard in-person training group. Randomization was achieved via a coin toss. In order to control for potential confounding variables related to particular training sites, per se, each office was required to train both an eLearning cohort and a traditional in-person cohort. The first cohort each office trained was determined by the coin toss and the second was always the opposite. 1.2. Dependent measure The dependent measure consisted of a written examination that was given prior to training and following training. The examination included 32 questions, which used both short and long answer formats. Each of the 32 questions varied in the number of points which could be earned for each question (range = 1–5), depending on the complexity of the answer required, for a total value of 75 points. Exams were scored by two staff members who were blind to group assignment. All tests were scored using a standard answer key. Interobserver agreement (IOA) was assessed for 33% of the total exams. Exact agreement was calculated by comparing how the primary and secondary test graders scored each question on the test. Agreement on a particular question was defined as both graders scoring the exact same point value for a question. Agreement for the entire test was calculated by dividing the number of questions with agreement by the total number of questions on the test and then multiplying by 100. Mean IOA across all tests scored was 88% (range = 63–100%). 1.3. Participants and setting This study was conducted within a large scale community-based service provider of behavioral intervention services to children with ASDs. All new entry-level therapists hired during the study

14


period were included as participants. Trainees were excluded from the study if they reported any significant prior history with ABA services when they were hired. A total of 97 newly hired entry-level behavioral therapists began the study and 9 participants were withdrawn, resulting in 88 participants completing the study. The 9 participants which were withdrawn were withdrawn due to voluntary termination of employment (i.e., turnover) before completion of training. A total of 33 participants completed the eLearning training and 55 participants completed the in-person training. Group size was not evenly matched due to the random assignment procedure. The average age of participants was 24 with a range of 19–46 years. Education of the sample varied, with 39 having a high school diploma, 2 an Associate of Arts degree, 31 holding a Bachelor of Arts/Sciences, and 1 participant holding a Master of Arts. Level of education was not reported for 15 participants. 1.4. eLearning training The eLearning training procedure evaluated in this study involved individual trainees interacting with a computer program which ran on a personal computer. The computer program was developed specifically for the purpose of training entry-level employees on the principles and procedures of ABA treatment for children with autism. The program was developed using Adobe Authorware1. The program consists of separate modules on the following topics: Introduction to Autism, Introduction to ABA, Introduction to Discrete Trial Training, The Antecedent, The Response, The Consequence, The Intertrial Interval, Prompting and Fading, Shaping, Chaining, Discrimination Training, Defining Behavior, Functions of Behavior, Antecedent Based Interventions, Consequence Based Interventions. Each module consisted of animated slides with vocal lecture. Trainees were able to move forward and backward through the slides at their own pace, by using a mouse to click buttons on the computer screen. Many of the slides contained video clips which demonstrated examples of correct implementation of the procedures being described. Trainees were provided a basic instruction sheet on how to use the eLearning program. An outline of all the major sections and subsections of the training was also provided for the purpose of taking notes. The eLearning training took place at personal computers throughout the office locations. Because the eLearning training was self-paced, the total duration of training varied for each participant, however, the approximate time required was 10 h. After trainees completed the eLearning, they attended a two-hour follow-up discussion session with an in-person trainer. The purpose of the in-person discussion was to allow trainees to ask any questions that they may have identified during the eLearning training. 1.5. In-person training The in-person training consisted of the standard training procedure for newly hired behavioral therapists that the service provision organization implemented on a regular basis. Training was conducted in a classroom setting. Trainers at all sites lectured using the same PowerPoint1 presentation which included all material relevant to the therapist training. Group discussion was conducted throughout the lectures and trainees were encouraged to ask any questions they had at any time. After the lecture and discussion were complete for a particular topic, the group broke up into pairs and trainees engaged in role-playing to practice the procedures which were just covered in lecture and discussion. The topics covered in the in-person training were identical to those covered in the eLearning training and included the same video clips which were included in the eLearning training. Each trainee was provided the same outline for note taking that was provided in the eLearning training condition. The duration of the in-person training was two full work days (16 h total). 2. Results The primary analysis consisted of evaluating if participants in the eLearning group improved in knowledge over the course of participation. A paired-samples t-test was conducted for both the eLearning and in-person groups. With an alpha level of 0.05, change between pre- and post-test scores were significant for the eLearning group, t(32) = 26.196, p < 0.000 as well as the in-person group, t(54) = 55.54, p < 0.000.


15

Fig. 1. Mean points earned on pre- and post-tests for eLearning and standard training groups.

A between-groups analysis was also conducted to contrast training methods. Change scores were calculated by subtracting the pre-test score from the post-test score. An independent samples t-test found a significant difference between the eLearning and in-person groups, t(86) = 3.914, p < 0.000. Fig. 1 displays the overall performance at both pre- and post-test for both groups. 3. Discussion The results of the current study suggest that eLearning approaches to training newly hired staff in academic knowledge of ABA procedures may be effective and efficient. The increase in academic knowledge of ABA was clearly apparent in the increase in test scores from pre- to post-training, which was both statistically and practically significant. This is the first published empirical evaluation, to the authors’ knowledge, of an eLearning tool for training newly hired clinicians in academic knowledge of behavioral treatment for children with autism. Similar to the results found by Wright (2008), the traditional training condition consistently produced higher test scores post-training than the eLearning condition. This is not surprising, given that the traditional training condition contained 16 h of direct contact with an expert in ABA. In addition, during the traditional training condition, trainees had the opportunity to ask questions at any time during lecture and discussion. Therefore, if a trainee identified something on the task list that they did not understand from the lecture, they could raise their hand and ask a question and receive a direct answer at that moment. In the eLearning condition, if a trainee had a question, he/she could ask it during the two-hour discussion session, but the opportunity to do so could have occurred hours or days after the question had first been identified. In addition, it is possible that the trainer utilized more examples during lecture than the eLearning module employs, thereby using ‘‘multiple-exemplar training’’ (Stokes & Baer, 1977), possibly resulting in greater generalization of learned knowledge. Despite the consistently higher post-test scores produced by the traditional training condition, the post-test scores produced in the eLearning condition were consistently high as well. Put another way, although the traditional training produced more accurate responding, the eLearning condition also produced strong gains in academic knowledge. It is notable that despite the difference in content taught, that these results are again similar to those found by Wright (2008). These results arguably reinforce what is probably already considered best practices in training. That is, if an expert trainer is available to conduct in-person training for as many hours as is needed to produce mastery of academic knowledge, then this will likely always be the best option. However, in settings and organizations

16


where this is not the case (which are likely far more common in the real world and in the many regions of the world in which ABA competence has not yet been disseminated), then an eLearning option combined with a smaller amount of in-person contact with a trainer may be a viable alternative. Perhaps the most promising potential benefit of training via eLearning is the potential for cost savings and/or decreased need for face-to-face contact with an expert trainer. There is currently a massive shortfall of experts in ABA, relative to global demand. Many states in the U.S., and indeed many countries around the world, contain few or zero behavior analysts who can train others to meet the clinical need. If an eLearning product can achieve a training goal using fewer face-to-face hours than via a traditional approach, then, all other things being equal, a larger number of staff could be trained, per unit of time. For example, instead of training one classroom full of staff all day for two days, the current data suggest that a similar (albeit not identical) result may be obtained by training the same amount of staff with only 2 h of time required of the expert trainer. Hypothetically, if the logistics of scheduling were arranged properly, approximately eight times as many people could be trained in the same period of time required of the expert trainer (e.g., four two-hour sessions per day for two days, for a total of eight groups trained, as opposed to one eighthour session per day for two days for a single group). Of particular relevance may be regions which are remote from expert EIBI trainers and therefore have the burden of paying significant travel expenses to bring experts into the country to train. If much of the academic training could be done electronically before the trainer arrived, a significantly greater number of people could be trained when the expert is on-site, therefore dramatically decreasing the overall cost of training, per trainee. Several questions remain unanswered by the current study. Among the most important questions that are asked of innovations to training format, such as eLearning, is whether it affects the quality of clinician performance when actually working with clients. The purpose of this study was to conduct an initial evaluation of a new eLearning tool for training academic knowledge of ABA treatment for individuals with autism, not to assess its impact on clinical expertise. However, the rationale for training clinicians in academic knowledge ultimately must be that it will have a beneficial impact on clinical competence, so future researchers would do well to empirically evaluate what effect, if any, innovations in training approaches have on clinicians’ ability to implement intervention procedures with clients. One potential limitation to the eLearning training format in general is the possible effect that it may have on the morale of the group being trained. No formal data were collected on trainees’ perceptions of the eLearning training process in this study but anecdotal reports were made that some therapists reported wishing they had more contact with their fellow trainees during the process. It should be noted that no eLearning training experience will replace direct, hands-on training by an expert. Indeed, hands-on training is an essential component of effective instruction (van Oorsouw et al., 2009). Therefore, although training in academic knowledge via computer may have a negative impact on group cohesion (although this is yet to be determined), it will always need to be followed by inperson and hands-on training by an expert, so the overall decrease in human contact is not likely to be great. In spite of these limitations, the eLearning program was shown to be an effective tool to train new behavioral therapists on the academic foundations of ABA for children with autism. If the limitations are understood by the trainer and sufficient hands-on training along with alternate team-building activities are included, then eLearning tools may offer a more efficient means by which new therapists can be prepared for hands-on training. While further research is clearly needed in this area, it appears as though eLearning tools such as the one investigated in this study may have the potential to increase the number of well-trained clinicians available, thereby helping to meet the global clinical demand for ABA interventionists. Acknowledgments The authors would like to thank Adel Najdowski for her contributions to the study, as well as the nearly two dozen clinical and administrative staff who assisted in conducting and coordinating the study.


17

References Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1(1), 91–97. Baer, D. M., Wolf, M. M., & Risley, T. R. (1987). Some still-current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 20, 313–327. Eikeseth, S. (2009). Outcome of comprehensive psycho-educational interventions for young children with autism. Research in Developmental Disabilities, 30, 158–178. Eldevik, S., Hastings, R. P., Hughes, J. C., Jahr, E., Eikeseth, S., & Cross, S. (2009). Meta-analysis of early intensive behavioral intervention for children with autism. Journal of Clinical Child & Adolescent Psychology, 38, 439–450. Fazzio, D., Martin, G. L., Arnal, L., & Yu, D. C. T. (2009). Instructing university students to conduct discrete-trials teaching with children with autism. Research in Autism Spectrum Disorders, 3, 57–66. Hayward, D. W., Gale, C. M., & Eikeseth, S. (2009). Intensive behavioural intervention for young children with autism: A researchbased service model. Research in Autism Spectrum Disorders, 3, 571–580. Hertz-Picciotto, I., & Delwiche, L. (2009). The rise in autism and the role of age at diagnosis. Epidemiology, 20, 84–90. Lafasakis, M., & Sturmey, P. (2007). Training parent implementation of discrete-trial teaching: Effects on generalization of parent teaching and child correct responding. Journal of Applied Behavior Analysis, 40(4), 685–689. Petty, G., Lim, D., & Zulaf, J. (2007). Training transfer between CD-ROM based instruction and traditional classroom instruction. The Journal of Technology Studies, 33, 48–56. Sarokoff, R., & Sturmey, P. (2004). The effects of behavioral skills training on staff implementation of discrete-trail teaching. Journal of Applied Behavior Analysis, 37, 535–538. Schimming, L. (2008). Measuring medical student preference: A comparison of classroom versus online instruction for teaching PubMed. Journal of the Medical Library Association, 96(3), 217–222. Stokes, T. F., & Baer, D. M. (1977). An implicit technology of generalization. Journal of Applied Behavior Analysis, 10, 349–367. Thomson, K., Martin, G. L., Arnal, L., Fazzio, D., & Yu, C. T. (2009). Instructing individuals to deliver discrete-trials teaching to children with autism spectrum disorders: A review. Research in Autism Spectrum Disorders, 3, 590–606. van Oorsouw, W. M. W. J. , Embregts, P. J. C. M. , Bosman, A. M. T., & Jahoda, A. (2009). Training staff serving clients with intellectual disabilities: A meta-analysis of aspects determining effectiveness. Research in Developmental Disabilities, 30, 503–511. Wright, J. M. (2008). Web-based versus in-class: An exploration of how instructional methods influence postsecondary students’ environmental literacy. The Journal of Environmental Education, 39(2), 33–45.