2017, American Psychological Association. This ...

© 2017, American Psychological Association. This paper is not the copy of record and may not exactly replicate the final, authoritative version of the article. Please do not copy or cite without authors permission. The final article will be available, upon publication, via its DOI: http://dx.doi.org/10.1037/tep0000155. The current citation is: Clark, S. W., Gulin, S. L., Heller, M. B., & Vrana, S. R. (in press). Graduate training implications of the Q-interactive platform for administering Wechsler intelligence tests. Training and Education in Professional Psychology.

Graduate Training Implications of the Q-interactive Platform for Administering Wechsler Intelligence Tests Sarah W. Clark, Shaina L. Gulin, Mary Beth Heller, and Scott R. Vrana Virginia Commonwealth University Author Note Sarah W. Clark, Department of Psychology, Virginia Commonwealth University; Shaina L. Gulin, Department of Psychology, Virginia Commonwealth University; Mary Beth Heller, Department of Psychology, Virginia Commonwealth University; Scott R. Vrana, Department of Psychology, Virginia Commonwealth University. The authors prepared this manuscript with the assistance of the Pearson 30-day free trial of the Q-interactive available to all professionals, as well as the free customer service for assistance with technical problems, but otherwise have no connection with the Pearson Corporation or the Wechsler assessment instruments. The authors wish to thank Rachel Wallace and Julia Brechbiel, who provided feedback on the Q-interactive from a new trainee’s perspective. Correspondence concerning this article should be addressed to Scott Vrana, Department of Psychology, 806 W. Franklin St., Box 842018, Virginia Commonwealth University, Richmond, VA 23284-2018. Phone: 804-828-1242. Fax: 804-828-2237. Email: [email protected]

Running Head: TRAINING IMPLICATIONS OF Q-INTERACTIVE

2

Abstract With the introduction of the iPad-based Q-interactive platform for cognitive ability and achievement test administration, psychology training programs need to adapt to effectively train doctoral-level psychologists to be competent in administering, scoring, and interpreting assessment instruments. This manuscript describes the implications for graduate training of moving to iPad-mediated administration of the Wechsler intelligence tests using the Qinteractive program by Pearson. We enumerate differences between Q-interactive and traditional assessment administration, including cost structure, technological requirements, and approach to administration. Changes to coursework, practicum, and supervision and evaluation of assessment competencies are discussed. The benefits of Q-interactive include reduced testing and training time and the decrease or elimination of many types of administration and scoring errors. However, new training challenges are introduced, including the need to be proficient at troubleshooting technology, changes in rapport-building with clients, and assessing and facilitating clients’ comfort with the platform. Challenges for course instructors and practicum supervisors include deciding which testing modality to use, increased difficulty evaluating some aspects of administration and scoring competency, and the potential for more frequent updates requiring additional training and updating of skills. We discuss the training implications of this new platform, and make specific suggestions for how training programs may respond to these changes and integrate iPad administration into their courses and practica. Keywords: computer-assisted testing, assessment of intelligence, training, Q-interactive, Wechsler scales


3

Introduction Computers and other forms of technology have been integrating rapidly and in myriad ways into clinical psychology practice (Eonta, Christon, Hourigan, Ravindran, Vrana, & Southam-Gerow, 2011; Maheu, Pulier, McMenamin, & Posen, 2012; Nilsen & Pavel, 2015). Until recently, individually-administered intelligence testing has remained exceptionally untouched by technological advances, having remained fundamentally unchanged since being introduced by Alfred Binet over a century ago. However, administration and scoring of these tests are undergoing a rapid change with the introduction of Q-interactive. Q-interactive, an iPadmediated administration, scoring, and interpretation platform for the Wechsler intelligence scales and other individually-administered tests, marks a paradigm shift in the structure of tests and the role of psychologists in administering them. The present article examines Q-interactive and its implications for those involved in training doctoral-level psychologists to administer, score, and interpret intellectual assessment instruments. Administration, scoring, interpretation, and reporting the results of individual intelligence tests are core competencies for clinical and school psychologists. The large majority of doctoral programs in clinical and school psychology have at least one course devoted to teaching these skills (Ready & Veague, 2014; Sotelo-Dynega & Dixon, 2014). These courses typically focus on the Wechsler family of individual intelligence tests, which have long been among the most commonly-used psychological assessment instruments (Camara, Nathan, & Puente, 2000). Although programs may opt for different levels of exposure to the Wechsler tests depending on their specific training goals, intellectual assessment remains a core domain of many clinical psychologists in practice. The Wechsler tests, despite their popularity, have a well-deserved reputation for being difficult to administer and score compared to other tests of intellectual functioning such as the Woodcock-Johnson (Ramos & Alfonso, 2009). Hundreds of hours are spent every year teaching


4

graduate students in highly selective doctoral programs to add columns of numbers, correctly use the appropriate tables, and distinguish between one- and two-point Vocabulary responses. An empirical literature has developed to investigate the errors committed by students learning to administer the Wechsler tests, and the data are not encouraging. A recent study on the fourth edition of the Wechsler Intelligence Scale for Children (WISC-IV) found that graduate students made an average of 25.8 errors per protocol administered, even though it was the second intelligence test students learned in the class (Loe, Kadlubek, & Marks, 2007). Errors were committed on 98% of all protocols, and as a result the Full Scale IQ score was inaccurate on two-thirds of all protocols. Further, students did not significantly improve over three practice administrations despite receiving individualized written feedback about errors after each administration; a more recent study found no significant decrease in WISC-IV errors over six administrations (Mrazik, Janzen, Dombrowski, Barford, & Krawchuk, 2012). Perhaps more discouraging is that these results are consistent with results reported in similar studies using previous versions of the Wechsler tests (e.g., Alfonso, Johnson, Patinella, & Rader, 1998; Belk, LoBello, Ray, G. E., & Zachar, 2002; Slate & Chick, 1989; Slate, Jones, & Murray, 1991), despite extensive changes to more recent editions that intuitively should reduce some types of errors. Difficult-to-administer subtests like Object Assembly and Picture Arrangement have been dropped as core subtests and replaced by multiple choice subtests like Matrix Reasoning and Figure Weights. Time bonuses have been eliminated or reduced on Block Design and Arithmetic, reducing the timing burden (and concomitant errors) on examiners. One intervention that presumably has eliminated many errors is computer-assisted score calculation. Over a third (37.5%) of the total errors reported by Loe et al. (2007) were “errors in the calculation of chronological age, calculation of subtest raw scores, addition of raw, scaled, or standard scores on the front page of the protocol, conversion of raw scores into scaled scores, conversion of scaled scores into IQ and index standard scores, computation of percentile ranks,


5

and computation of confidence intervals” (p. 240). Assuming the ability to accurately enter numbers from the record form into the computer scoring program (sadly, not an assumption that can be safely made), these types of errors should be nearly eliminated when using Pearson’s Qglobal scoring software, which, after examiners enter subtest raw score totals, provides subtest and index scores, percentile ranks and confidence intervals, and statistical comparisons between scores. Computer-aided scoring and interpretation is now commonly available for most assessment instruments. However, with Q-interactive Pearson has introduced a fundamental change in the administration of individual assessment instruments. Q-interactive is a platform that allows iPad-mediated administration of the WPPSI-IV, WISC-IV, WISC-V, WAIS-IV, the Wechsler Memory Scale-IV, the third editions of the Wechsler Individual Achievement Test and the Kaufman Test of Educational Achievement, and many other individually-administered memory, speech and language, and neuropsychological tests (Pearson, 2015a). Q-interactive requires two Bluetooth-connected iPads. For Wechsler cognitive ability tests the examiner uses one iPad that combines most functions of the Administration and Scoring Manual, stopwatch, Record Form, and note pad, and allows the examiner to control stimuli presented on the other iPad, which is used by the examinee to view stimuli and make responses. Pearson touts Qinteractive (Pearson, 2015b) as a system to choose, administer, score, and interpret clinical assessments, and which will allow the examiner to spend more time engaging with and observing the client. It is purported to reduce the time spent on each Wechsler assessment by about 30%. A video overview of Q-interactive WISC-V administration (Pearson, 2015c) claims that it provides “increased fidelity to standardized administration, increased ease of recording behavioral observations, and reduced clerical errors.” Given the importance of standardized administration and scoring to the reliability and validity of Wechsler cognitive ability test results, and the demonstrated difficulty of successfully training professional psychologists on these tasks,


6

these claims are quite significant. Thus, the validity of these claims and the implications of Qinteractive for assessment training need to be addressed. So far, the literature on Q-interactive versions of Wechsler tests consists of a series of non-peer reviewed technical reports comparing the equivalence of scores obtained using Q-interactive and standard administration (Daniel, 2012; Daniel, Wahlstron, & Zhang, 2014; Raiford, Drozdick, & Zhang, 2015; Raiford, Holdnack, Drozdick, & Zhang, 2014), as well as a user survey on examinee behavior (Daniel, 2013). In addition, Pearson has created several webinars on integrating Q-interactive into psychology training programs (Gabel, 2013, 2014; Noland, 2014). The present article will consider implications of Q-interactive for graduate coursework on intellectual assessment and assessment practica within university-based training clinics. We will focus on the WAIS-IV, though substance of the text is directly applicable to the WISC-V and generally applicable to many of the other instruments supported on the Q-interactive platform.1 To investigate the implications of incorporating Q-interactive into a clinical, counseling, or school psychology program’s assessment coursework and practicum training, we reviewed the Q-interactive website and training videos (http://www.helloq.com/home.html), non-peer reviewed reports published by Pearson (e.g., Daniel, 2012), and internet blogs and reviews written by practitioners. Our attempts at locating peer-reviewed materials specifically devoted to training issues were unsuccessful. In order to present information about the graduate training implications of the Q-interactive platform, we collected perceptions of people with diverse assessment experiences. The authors’ backgrounds include a professor with several decades of experience teaching an intellectual assessment course, a director of a departmental training clinic with over 25 years of experience as a school/clinical psychologist, and two advanced graduate teaching assistants whose primary responsibilities include scoring protocols, observing practice

1

The Q-interactive version we tested was downloaded on September 9, 2015. It should be noted that Pearson makes regular changes to the Q-interactive platform, so some information may be outdated by the time of publication.


7

administration, and identifying errors in scoring and administration. In addition, to obtain feedback on learning the WAIS-IV from a new trainee’s point of view, two first-year graduate student volunteers from our department’s intelligence testing course were trained on the Qinteractive platform three weeks after having learned traditional WAIS-IV administration. At the time of Q-interactive training, they had each traditionally administered and scored two full WAIS-IV tests to practice clients. These students’ perspectives on Q-interactive should be viewed within the context of being just two trainees coming from the same clinical psychology program (and being trained by) the authors; their experiences should not be regarded as definitive “data” on the experience of the Q-interactive by new graduate students. At this writing, the literature is devoid of any formal investigations regarding the use of the Q-interactive platform in university-based psychology training programs. In August 2013— approximately eight months after the official release of Q-interactive—an informal inquiry to the over 200 members of the Association of Psychology Training Clinics (APTC) yielded only five responses, from clinic directors who reported use of (or intention to use) Q-interactive in their training clinics. In preparation for the current article, in fall 2015 the current authors submitted a second inquiry to the APTC and Council of University Directors of Clinical Psychology (CUDCP) listservs, asking for feedback on their experiences with Q-interactive. Only two members of each group responded, consisting of one enthusiastic training clinic adopter and three non-adopters expressing significant skepticism about the Q-interactive platform. It is difficult to interpret this near silence from two large and typically quite interactive listserv groups. The response suggests caution, at best, among stakeholders most responsible for clinical and school psychology assessment training. Adoption of the Q-interactive platform will necessitate substantial shifts for psychology training programs. The first consideration relates to materials required; to this end, in Table 1 we present a comparison of start-up costs for Q-interactive administration with a traditional paper-


8

and-pencil WAIS-IV kit. As is evident in Table 1, start-up costs are substantially greater for Qinteractive, primarily due to the necessity of purchasing iPads and their accessories. Notably, however, the Q-interactive Classroom License ($150) provides unlimited administration for all students for the length of the assessment course and includes access to “all appropriate Qinteractive content,” not just the WAIS-IV (Pearson, 2015d). After these initial “sunk costs” are accounted for in training program budgets, longerterm costs are more difficult to estimate. With the expense of annual licensing fees, assuming stable pricing, the cost for access to Q-interactive will reach $1500 over 10 years, approximately double the total expense of one traditional WAIS-IV kit. This does not take into account, however, the need to purchase additional protocols for the paper-and-pencil version, while training courses are permitted unlimited Q-interactive administrations. On the other hand, programs would be wise to consider the likely need to regularly replace iPads, given rapid technology changes, software updates that make old hardware obsolete, and the likelihood that the devices get lost or damaged more quickly with multiple users. Assuming that programs are taking advantage of Pearson’s Training Partner Program (TPP) that offers a 40% discount, based solely on initial estimated costs, 25 administrations of the paper-and-pencil form of the WAIS-IV equate to $30.16 per administration, while 25 administrations of the Q-interactive version would be $48.24 each. Given the costs of purchasing additional sets of record forms and response booklets for the traditional administration (and assuming the 40% TPP discount), the costs become essentially equivalent at 69 administrations per test kit (or pair of iPads) per year, with Q-interactive gaining the advantage as administrations exceed 69. Obviously, the cost of traditional administration is falsely inflated, as one WAIS-IV kit is expected to be used for multiple years, until the next version is released. Another cost-related issue is that some students may own personal iPads that can be used as one of the two iPads required for administration,


9

with the installation of the free app from Pearson Education. This could reduce overall training costs for the program, but raises issues of unequal access to training materials. In addition to coursework, training programs often provide practicum training in IQ assessment. Pearson requires an Experiential Training License for training clinic and practicum use, with associated costs substantially more difficult to estimate. Although there is no fee for supervisors’ or students’ licenses, assessments are charged a discounted rate of 90 cents per subtest (Pearson, 2015d), which may be estimated and billed on an annual basis. Although conclusive guidelines regarding cost-effectiveness of each format would seem ideal, there are too many variables (e.g., number of students, number of assessments required, replacement schedules for iPads) to make definitive recommendations possible. Rather, in Table 1 we have provided a cost breakdown for traditional and Q-interactive components that training programs may use to perform their own financial analysis. How Q-interactive Differs from Traditional Administration The differences between Q-interactive and traditional administration are apparent before testing begins: WiFi is needed to configure the test battery online and then send it to the iPad for administration. Though an internet connection is not necessary during the administration, the examiner and client iPads must be Bluetooth-connected to communicate with each other. Because stimuli are presented on the client iPad, Q-interactive completely eliminates the need for a stimulus book. Certain subtests have not yet been configured for digital administration (Pearson, 2015c); for example, blocks are still needed for Block Design. At the time of this writing, Coding and Symbol Search on the WAIS-IV are still completed via paper booklet and then hand-scored, whereas digital versions of Coding and Symbol Search have been introduced for the WISC-V. Q-interactive also eliminates the need for a separate examiner manual. Rather, items are presented one at a time on the examiner’s iPad screen, with instructions. Selecting an


10

Administration icon from the top toolbar of each subtest displays instructions for start points, reversal and discontinue rules, and other subtest-specific guidelines. Q-interactive takes advantage of several iPad features to make administration easier. Response timing occurs via a digital stopwatch, eliminating the need for a handheld stopwatch. Although the examiner must manually start and stop the timer, a unique feature of Q-interactive is that the examiner is notified to prompt the examinee (“Do you have an answer?”) after a certain amount of time has elapsed; if the time does expire for a given item, the program automatically scores the item as incorrect. Q-interactive also enables automated basal and ceiling rules. The examiner is prompted when the examinee meets discontinue criteria, at which point the examiner may choose to test limits by administering additional items. The trend in recent editions of Wechsler tests has been to increase the number of subtests using multiple choice responses, such as Matrix Reasoning, Visual Puzzles, and Figure Weights. This has simplified scoring for traditional test administrations. On the Q-interactive platform, examinees may select a multiple choice response by touching their screen, which is then scored automatically. Digital implementation of Coding and Symbol Search (on the WISC-V) converts them into multiple choice tasks, allowing automatic scoring of these two subtests as well. In addition to automated scoring, Q-interactive includes a built-in audio recorder for all subtests where the client responds verbally (i.e., Similarities, Vocabulary, Digit Span, Arithmetic, Information, Comprehension, and Letter-Number Sequencing). These audio recordings are saved to the iPad and are available to the examiner following administration. For privacy purposes, audio files are not transferrable to Pearson Central, the cloud-based scoring and data storage system; therefore, examiners must listen to and capture any necessary audio data before removing the administration from the iPad. For clinicians who wish to manually record their examinees’ responses or write down behavioral observations during testing, Pearson recommends using a stylus for writing on the iPad. A small corresponding area for text is


11

available at the bottom of each item requiring a verbal response, as is a “Notes” button that, when clicked, opens into a full-screen canvas. Written responses and notes from the iPad are stored permanently in the online database. For Q-interactive Verbal Comprehension subtests, general scoring principles are available from a drop-down menu. Specific scoring guidelines for each item are available below the item prompt. Examiners select a score for each response (2, 1, or 0) based on scoring examples provided beneath the item. Thus, examiners may select which scoring example they believe best reflects the examinee’s response, rather than recording the response verbatim. Subtests are scored automatically once the end of the subtest is reached. Like Q-global scoring software, Q-interactive calculates age, adds raw scores, computes subtest and index scaled/standard scores, generates percentile ranks and classifications, computes index and subtest comparison scores, and provides subtest strengths and weaknesses. In addition, prorating and subtest substitution can be completed on the iPad. Pearson touts auto-scoring as one of the most advantageous features of Q-interactive, reducing potential errors associated with hand-scoring (e.g., Loe et al., 2007) and significantly reducing scoring time (Pearson, 2015c). Examining the Errors Committed by Novice Examiners A major aim of this paper is to evaluate the effect of Q-interactive on errors commonly committed by novice Wechsler examiners, and the implications for training. Our experience teaching intelligence assessment, supported by the empirical literature (Mrazik et al., 2012; Loe et al., 2007), is that there are certain types of errors graduate student trainees consistently make. Computation errors. Consistent with Pearson’s claims, we found that Q-interactive’s auto-scoring and interpretation system almost completely eradicates computation errors. Chronological age, for instance, is calculated automatically when the examiner inputs the examinee’s birthday. Also, because test results are saved within the application, Q-interactive improves on existing computer-aided scoring software (e.g., Pearson’s Q-global), eliminating the


12

need to manually enter raw scores from the record form to the computer. As computation errors compose over one-third of the total errors made on graduate student protocols (Loe et al., 2007), this reduces both errors and the time necessary to teach and evaluate these clerical skills in assessment courses. Because learning to complete the record form by hand and use the tables in the Administration and Scoring Manual takes substantial supervised practice, one implication of training only with Q-interactive is that students will not be able to adopt traditional testing methodology without additional instruction. Failure to record an examinee’s response. As failure to record a response on the test protocol commonly occurs with paper-and-pencil administrations (Loe et al., 2007), we hoped that the Q-interactive platform would reduce the frequency and likelihood of this error. Unfortunately, however, failing to record a response was the most frequent administration error we encountered using Q-interactive. We suspect this is because there are several steps the examiner must take for each item: read the instructions aloud, start and stop the stopwatch, and control the stimulus display on the client’s iPad. Recording a correct or incorrect response is typically the final task on the examiner’s cognitive “to do” list. Q-interactive seems to have recognized the potential for this error and automatically flags unanswered items, prompting the examiner to enter a response prior to generating scores. Additionally, assuming that the audio recording functions properly, examiners have the advantage of reviewing client responses both in-the-moment and after testing. Although iPad technology may increase the likelihood of failing to record an examinee’s response, the error is actually easier to evaluate and correct, compared to traditional paper-and-pencil administration. For verbal subtests, the audio recording may be consulted if notes were not taken. For other subtests, it is only necessary to select whether the answer was correct or not. Failure to appropriately query and score on verbal subtests. Accurate scoring and querying of ambiguous responses on Similarities, Vocabulary, and Comprehension subtests have


13

long been considered among the most difficult skills to master on the Wechsler tests (Belk et al., 2002; Slate & Chick, 1989). Further, the Verbal Comprehension Index (VCI) is the composite score most prone to examiner errors (Loe et al., 2007). The Q-interactive platform does not appear to improve VCI errors. Because examiners may select the scoring example they believe most closely matches the examinee’s response rather than recording the entire response, the examples may inadvertently encourage pattern-matching instead of careful consideration and querying of the (more accurate) general scoring principles. This was noted frequently during our novice trainees’ administrations. Such errors inevitably contributed to inaccurate Similarities and Vocabulary subtest scores, thereby affecting VCI results. Q-interactive also makes supervisors’ evaluation of querying and scoring errors on Verbal Comprehension subtests more difficult. The availability of the audio recording, along with discomfort in using the stylus to transcribe responses (detailed below), may reduce written documentation. Further, although check boxes are available to log queries, these refer to the client’s entire response rather than noting at what point and how often queries were made, making it difficult to determine if they were done appropriately. In our experience, novice examiners tend to over-query responses when a lower-point score would be appropriate. This lack of detail in response documentation means that the supervisor may lack sufficient information to identify and correct trainee administration and scoring errors. There are two potential solutions to this issue. One is to require students to write down all responses verbatim during administration and submit them to a supervisor for review. A second option is for students to record all responses verbatim using an iPad Pro and Apple Pencil, which provides a more fluid handwriting experience than a stylus. Students can then save or print a full record after the test administration. Purchase of an iPad Pro and Apple Pencil, however, adds to start-up costs. Start/discontinue/reverse errors. Novice Wechsler examiners often incorrectly apply start points and basal and ceiling rules (Loe et al., 2007). Q-interactive completely eliminated


14

this error type. Entering the birthdate automatically generates age-based start points on the WAIS-IV, although examiners may select an earlier start point if warranted. With Q-interactive, if the examinee does not provide a perfect response to a basal item, the program automatically reverses to a previous item to achieve basal. Similarly, based on discontinue rules for different subtests, the examiner is notified when the ceiling has been reached. As noted above, the examiner can then choose to discontinue the subtest, or to begin testing limits without affecting the score. Assuming that the examiner appropriately marks items as correct or incorrect, Qinteractive wholly eliminates these types of administration errors. New Training Considerations for Q-interactive Student training considerations. Those adopting Q-interactive will face a host of new training issues. Perhaps the biggest change is the need to be comfortable with the iPad, including WiFi and Bluetooth settings, and to effectively troubleshoot when problems arise. When the Bluetooth connection is lost—which occurred more frequently than we would have liked—the iPads stop communicating with each other mid-test. Further, loss of internet connection can interrupt the upload of the administration from the computer. Training will need to explicitly address the required technical knowledge, especially with trainees that are not technologicallyinclined. Pearson seems to recognize this challenge and offers a variety of online training materials. Free webinars provide an overview of the system. Two-part “on-boarding” webinars for new Q-interactive customers are offered approximately twice per month and cover topics such as correct iPad settings, Central Workflow functions, using imbedded tools (e.g., stopwatch, notes, audio recording), transferring assessments to cloud-based scoring, accessing scores, and exporting/archiving data. All webinars are recorded and could easily be incorporated into course syllabi. A related training challenge will be helping novice examiners to identify the most efficient testing procedures. In using the Q-interactive platform, this was not always as intuitive


15

as we would have liked. For example, the notepad space to record examinee’s verbal responses is quite small and generating a larger canvas for notes requires navigation away from the subtest. Pearson recommends using a stylus to record examinee responses and behavioral observations, but we found use of the stylus to be unwieldy and to result in illegible writing if not done fastidiously. Discomfort in using the stylus was further exacerbated by a lagged response (using iPad 2; response may be faster with newer iPads). Both novice trainees opted to use a separate paper-and-pencil notepad instead, thereby defeating the purpose of the Q-interactive notepad. Writing within the Q-interactive platform may become easier with practice. Nevertheless, the mechanics of Q-interactive discouraged our novice trainees from writing responses verbatim and taking extensive notes on behavioral observations, both practices we stress when teaching Wechsler administrations. Increasing familiarity with this technology to insure examiner selfefficacy will be vital. Despite feeling confident with the iPad administration, our digitally-native novice trainees stated they preferred the traditional assessment. Training should also address how this technological shift will affect examinee experience. On any assessment examiners must learn to establish rapport quickly, engage participants, and allay anxieties that may impede examinee performance, all while maintaining standardized administration. Although several reports tout Q-interactive as an effective platform for increasing engagement in child examinees (Na & Burns, 2015; Pearson, 2015d), this segment of the population has grown up with technology. In contrast, older adults use computers less often (Keenan, 2009) and generally report lower readiness and willingness to adopt new technology (Rose & Fogarty, 2010). Research on computerized cognitive testing has demonstrated increased anxiety and reduced performance for older adults (Laguna & Babcock, 1997), but at present it is unknown if iPad-based administration carries these same implications. For a time both digital and paper-and-pencil versions of the tests will be available, and trainees will need to learn how to select the optimal testing modality for a given examinee. The


16

WISC-V and WAIS-IV Administration and Scoring manuals give explicit advice about choosing a test for examinees when more than one is appropriate (e.g., for 16-year-olds), and describe this as a “suitability and fairness” issue (e.g., Wechsler, 2014, p. 17). However, there is currently no guidance for selecting between versions of iPad-administered tests. Age of the examinee is likely to be a contributing factor. In its current form, the Q-interactive version of the WAIS-IV does not include any pre-testing practice that would help familiarize an examinee with the iPad to reduce intimidation and anxiety prior to administration. Although sample items are included at the beginning of each subtest, these are identical to the items within standard WAIS-IV administration, and are similarly focused on assessing the examinee’s understanding of the task, not their comfort with the test format. Future research may empirically examine the extent to which examinees’ limited interaction with the iPad warrants additional pre-testing technology practice. Apart from consideration of age and/or level of comfort with technology, rapport building may differ between the two administration modalities. Consistent with Pearson’s claims, we found Q-interactive testing to result in faster administration than paper-and-pencil testing, even for our novice examiners. In addition, the decrease in testing materials eliminates need to reorganize materials between subtests as with traditional administration. Because these “breaks” create natural opportunities for rapport building, trainees must make a conscious effort while using Q-interactive to monitor and engage the examinee. Thus, though Q-interactive is touted as allowing the examiner to “maximize the time you spend with your examinee” (Pearson, 2015b), it may actually decrease examiner-examinee interaction. Our students agreed that they tended to focus on the mechanics of the iPad at the expense of interaction with the client. Observationally, our novice examiners appeared to check in less often with examinees, offer less encouragement, and make less eye contact when using Q-interactive. Training with Qinteractive may thus require more explicit focus on rapport-building and examinee engagement.


17

The move to this new platform appears likely to reduce examiner-examinee interaction, and future training and research should consider how this might affect performance. Course and clinic considerations. Because updated versions of software and tests can be released very easily, more frequent revisions of courses and regular continuing education will be required. For example, a student who learned the Q-interactive platform in a spring 2016 assessment course would have needed to learn the new digital Coding and Symbol Search before administering the WISC-V during a summer 2016 practicum. This is in contrast to the expectation of a new edition of the traditional test being released every 10 to 12 years. It is unclear whether the rapid iterations of Q-interactive are due to the creators tweaking the new platform and ultimately a stable version will emerge, or if frequent changes in test software, content, and administration procedures will be the new norm. A final consideration for training programs is the ethics of storing respondent information in Pearson’s database. In addition to administration data, Q-interactive permanently stores all personally identifiable information provided by the examiner (e.g. client name, date of birth, gender). These are transferred from the tablet via secure connection to an encrypted database. Should respondents be informed that their personal information and assessment results will be permanently stored by Pearson? Given the sensitive nature of intelligence test results, it is important to determine what safeguards are in place to maintain confidentiality. Training clinics will need to decide if and how to discuss this information with examinees and whether to anonymize personally-identifiable information when submitting the administration for scoring. Classes that rely on volunteer examinees will need to be particularly sensitive about identifying information, since there is no way to score tests in Q-interactive without submitting information for permanent storage on Pearson’s servers. What is Gained and Lost in Training with Q-interactive?


18

It is likely that training in administration of the Wechsler tests will be easier and less error-prone with Q-interactive, while some administration errors may be introduced and others persist. Score calculation and table usage errors are largely eliminated, as are many administration errors. Because teaching these procedures and addressing these errors takes a good deal of class time when teaching traditional Wechsler test administration, instructional time will be freed for introduction of new topics or more in-depth treatment of other issues, such as rapport building and the development of clinical acumen, score interpretation, and case conceptualization. Students may also benefit from reduced time burden for practicing tests through decreased administration time, as well as reduction of materials to be transported between classes and training sites. Hand calculation of scores and use of tables is eliminated with Q-interactive. Although this saves time, calculating scores by hand and consulting tables affords students a greater understanding of how scores are generated and complement one another. Further, as students become familiar with scoring, they learn to generate hypotheses in the context of overall case conceptualization. Although hand-scoring is still theoretically available if the training clinic has the resources to purchase both Q-interactive and hard-copy versions of the tests, scores may be generated easily online and without a thorough understanding of what they represent. Also lost is ready documentation of the administration. In traditional administration, examiners document responses, sample item administration, queries, and behavioral observations in the record form. This record form is readily available for teaching assistants, instructors, and clinical supervisors to review answers and evaluate scoring. With Q-interactive, reviewing each item is more difficult, unless additional steps are taken to record all responses on the iPad and download the complete record form as described earlier in the article. Supervisory review of scoring will be especially problematic if responses were only audio-recorded and not manually written down by the examiner. Further, whereas paper record forms can be turned in for review


19

while the trainee retains the manual and test kit for additional use, with Q-interactive the trainee would need to turn in their iPad to have the audio portion of their administration reviewed. Finally, study, practice, and preparation for testing are in some ways made more difficult with Q-interactive. Students in our IQ course and assessment practicum invariably fill the administration manual with sticky notes and scribble administration notes in their record forms, and decades of research in educational psychology has demonstrated that actively organizing information is crucial for learning. Few sticky notes fit on an iPad and, although the necessary administration instructions are in the Q-interactive program, they often exist beneath a level or two of buttons; our novice trainees noted difficulty finding the general administration instructions. Further, it is difficult for trainees to practice individual subtests, or switch the order of subtests, as each administration must be created and sent to the iPad via WiFi. This is substantially more cumbersome than simply opening the administration manual and stimulus book for a practice administration. Conclusions and Future Directions Q-interactive represents a significant shift in integrating technology into intelligence testing, a trend that is likely to increase (Vrana & Vrana, in press). Q-interactive appears to substantially reduce calculation and administration errors and increase efficiency in testing. However, some challenges remain and others are introduced. Novice examiners must be taught to understand the underlying principles of scoring, and there may be an increased likelihood of pattern-matching instead of carefully-considered scoring of Verbal Comprehension subtests. Further, Q-interactive automatically calculates index and other score comparisons, obscuring the relationships between these, so scores may be calculated without a full understanding of what they represent. The technology itself requires facility on the part of the examiner to troubleshoot WiFi and Bluetooth problems. Although reducing the material load and automatizing administration should theoretically leave the examiner more time to devote to the examinee,


20

novice examiners are still likely to focus on navigating the application at the expense of clinical observation and interaction. Q-interactive may also create new challenges for older examinees. At this time we recommend that training clinics continue to train students first on traditional paper-and-pencil administration, both for ease of practicing individual subtests and to learn and understand the basic mechanics of administering and scoring. This will facilitate the transition to the Q-interactive format. In our experience, both novice and seasoned examiners felt more comfortable beginning with the traditional format. Though it may be beneficial to learn both modalities alongside each other, the efficacy and necessity of this approach is unknown, and research is needed to formally evaluate the errors and challenges to training during the transition to Q-interactive. It is likely that technology-aided administration will increasingly become the norm, and training clinics must carefully consider the implications of these changes. Indeed, the Spanish version of the WISC-V will be released on Q-interactive only, with no “traditional” administration option (Pearson 2016), and so graduate training on the Q-interactive may soon turn from an option to a necessity. The Q-interactive platform is a relatively recent development in psychological assessment, and much is unknown about its implications for training in professional psychology. Independent research is needed comparing Q-interactive to traditional paper-and-pencil test administration to evaluate issues of feasibility, ease and accuracy of administration, trainee satisfaction, and alternative training approaches. At this time, it is unknown the extent to which professional psychologists are using the Q-interactive technology, and thus the extent to which graduate programs are obligated to train students on this technology to ensure competence in standard professional practices. Answers to these types of inquiries will assist graduate programs in determining whether to provide training in Q-interactive, and how to do it most efficiently and effectively.


21

References Alfonso, V. C., Johnson, A., Patinella, L., & Rader, D. E. (1998). Common WISC-III examiner errors: Evidence from graduate students in training. Psychology in the Schools, 35(2), 119-125. Belk, M. S., LoBello, S. G., Ray, G. E., & Zachar, P. (2002). WISC-III administration, clerical, and scoring errors made by student examiners. Journal of Psychoeducational Assessment, 20(3), 290-300. doi: 10.1177/073428290202000305 Camara, W. J., Nathan, J. S., & Puente, A. E. (2000). Psychological test usage: Implications in professional psychology. Professional Psychology: Research and Practice, 31(2), 141– 154. doi: 10.1037/0735-7028.31.2.141 Daniel, M. H. (2012). Equivalence of Q-interactive-Administered Cognitive Tasks: WAIS-IV (Qinteractive Technical Report 1). Retrieved from http://www.helloq.com/research.html Daniel, M. H., Wahlstron, D. & Zhang, O. (2014). Equivalence of Q-interactive and Paper Administrations of Cognitive Tasks: WISC–V. (Q-interactive Technical Report 8). Retrieved from http://www.helloq.com/research.html Eonta, A. M., Christon, L. M., Hourigan, S. E., Ravindran, N., Vrana, S. R., & Southam-Gerow, M. A. (2011). Using everyday technology to enhance evidence-based treatments. Professional Psychology: Research and Practice, 42(6), 513-520. doi: 10.1037/a0025825 Gabel, A. D. (2013). Q-interactive: Integrating Q-interactive into a training environment: Tips and tricks. Retrieved from http://images.pearsonclinical.com/videos/Qi_Tips_6_18_2013/ Gabel, A. D. (2014). Teaching assessment evolved: Integrating Q-interactive into your graduate program. Retrieved from http://www.pearsonclinical.com/psychology/products/ 100000773/qinteractive.html#tab-training


22

Keenan, T. A. (2009). Internet use among midlife and older adults: An AARP bulletin poll. Retrieved from http://assets.aarp.org/rgcenter/general/bulletin_internet_09.pdf Loe, S. A., Kadlubek, R. M., & Marks, W. J. (2007). Administration and scoring errors on the WISC-IV among graduate student examiners. Journal of Psychoeducational Assessment, 25(3), 237-247. doi: 10.1177/0734282906296505 Maheu, M. M., Pulier, M. L., McMenamin, J. P., & Posen, L. (2012). Future of telepsychology, telehealth, and various technologies in psychological research and practice. Professional Psychology: Research and Practice, 43(6), 613-621. doi: 10.1037/a0029458 Mrazik, M., Janzen, T. M., Dombrowski, S. C., Barford, S. W., & Krawchuk, L. L. (2012). Administration and scoring errors of graduate students learning the WISC-IV: Issues and controversies. Canadian Journal of School Psychology, 27(4), 279–290. doi: 10.1177/0829573512454106 Na, S. D., & Burns, T. G. (2015). Wechsler intelligence scale for children-V: Test review. Applied Neuropsychology: Child, 5(2), 156-160. doi: 10.1080/21622965.2015.1015337 Nilsen, W. J. & Pavel, M. (2015). Behavioral health information technology adoption in the context of a changing healthcare landscape. In L. A. Marsch, S. E. Lord, & J. Dallery (Eds.), Behavioral healthcare and technology: Using science-based innovations to transform practice (pp. 307-316). New York: Oxford University Press. Noland, R. (2014). Faculty Voices: Q-interactive in psychology training programs (Part 1). Retrieved from http://www.pearsonclinical.com/psychology/products/100000773/ qinteractive.html#tab-training Pearson (2015a). Test library. http://www.helloq.com/tests/test-library.html Pearson (2015b). Welcome to Q-interactive. http://www.helloq.com/home.html Pearson (2015c). WISC-V Digital: Overview. Retrieved from http://www.helloq.com/tests/wiscv.html#


23

Pearson (2015d). License options. Retrieved from http://www.helloq.com/pricing/LicenseOptions.html Pearson (2016). WISC-V Spanish: Comprehensive, modern, and convenient. Retrieved from http://helloq.com/tests/wisc-v/spanish.html Raiford, S. E., Drozdick, L., & Zhang, O. (2015). Q-interactive special group studies: The WISC–V and children with autism spectrum disorder and accompanying language impairment or attention-deficit/hyperactivity disorder. (Q-interactive Technical Report 11). Retrieved from http://www.helloq.com/research.html Raiford, S. E., Holdnack, J., Drozdick, L., & Zhang, O. (2014). Q-interactive Special Group Studies: The WISC–V and Children with Intellectual Giftedness and Intellectual Disability. (Q-interactive Technical Report 9). Retrieved from http://www.helloq.com/research.html Ramos, E., Alfonso, V. C., & Schermerhorn, S. M. (2009). Graduate students’ administration and scoring errors on the Woodcock-Johnson III tests of cognitive abilities. Psychology in the Schools, 46, 650-657. doi: 10.1002/pits.20405 Ready, R. E. & Veague, H. B. (2014). Training in psychological assessment: Current practices of Clinical psychology programs. Professional Psychology: Research and Practice, 45(4), 278-282. doi: 10.1037/a0037439 Rose, J., & Fogarty, G. J. (2010). Technology readiness and segmentation profile of mature consumers. Academy of World Business, Marketing & Management Development Conference Proceedings, 4(1), 57-65. Slate, J. R., & Chick, D. (1989). WISC-R examiner errors: Cause for concern. Psychology in the Schools, 26, 78-84.


24

Slate, J. R., Jones, C. H., & Murray, R. A. (1991). Teaching administration and scoring of the Wechsler Adult Intelligence Scale-Revised: An empirical evaluation of practice administrations. Professional Psychology: Research and Practice, 22, 375-379. Sotelo-Dynega, M. & Dixon, S. G. (2014). Cognitive assessment practices: A survey of school psychologists. Psychology in the Schools, 51, 1031–1045. doi: 10.1002/pits.21802 Vrana, S. R. & Vrana, D. T. (in press). Can a computer administer a Wechsler intelligence test? Professional Psychology: Research and Practice. doi.org/10.1037/pro0000128 Wechsler, D. (2014). WISC-V Administration and Scoring Manual. Bloomington, MN: Pearson.


25

Table 1. Course-related1 start-up costs of traditional versus Q-interactive versions of the WAISIV2. Item Traditional WAIS-IV Q-interactive WAIS-IV 3 Basic Kit (no case)/ Starter Kit $743.67 Included with license4 Silent stopwatch $ 9.95 2 full size iPads, WiFi enabled, at $ 959.98 (iPad Air 2) least 16gb memory, iOS 9 or higher Capacitive-enabled stylus $ 9.99 2 iPad protective cases5 $ 35.98 2 antiglare screen covers5 $ 49.98 6 Annual Classroom License $ 150.00 $753.62 $1205.93 Total 1 This table reflects Pearson pricing for a classroom license. For an experiential training (e.g., training clinic or practicum) license in which the tests are administered to “real” clients, there is no license fee for supervisors or students but there is a $1.50 (.90 with training discount) fee per subtest administered. 2 All costs reflect 40% discount through Pearson’s Training Partnership Program where applicable. Items purchased separately and not available through Pearson were priced at a national big box retailer (with exception of the silent stopwatch) and generally with the most affordable option selected. Tax exempt status is assumed. The cost of internet/WiFi access is not included, as it is assumed that training clinics already have this capacity and thus would not incur additional expense. 3 Basic Kit includes Administration/Scoring and Technical manuals, stimulus books, scoring keys, 1 set of Block Design cubes, and 25 sets of record forms and response booklets. 4 Starter Kit includes scoring keys, 1 set of Block Design cubes, and 25 sets of record forms and response booklets (Additional Starter Kits may be ordered with the Training Partnership Plan discount at $148.29.). 5 Recommended, but not required 6 Fee for professor, no fee for students; unlimited administrations for length of course