Examining Preservice Teachers Criteria for Evaluating Educational ...

5 downloads 194887 Views 368KB Size Report
Apr 30, 2017 - Full-text (PDF). Available from: Evrim Baran, Jul 13, 2016 ... Keywords. mobile learning, mobile apps, usability testing, preservice teachers .... social networking and data sharing), certain tools now present opportunities.
Article

Examining Preservice Teachers’ Criteria for Evaluating Educational Mobile Apps

Journal of Educational Computing Research 0(0) 1–25 ! The Author(s) 2016 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/0735633116649376 jec.sagepub.com

Evrim Baran1, Erdem Uygun1, and Tugba Altan1

Abstract Recent interest in integrating mobile apps into teaching will continue growing. There remains, however, a pressing need to develop methods and resources to support and educate preservice teachers about the use of these technologies. This case study aimed to examine preservice teachers’ criteria for evaluating educational mobile apps. Nineteen fourth-year preservice teachers, who studied math, science, English, and computer education at a public university, participated in the research. During the usability tests, the preservice teachers were asked to evaluate the affordances and limitations of selected educational mobile apps from their particular subject domains. Their interactions with and evaluations of the mobile apps were collected through think-aloud sessions and interviews. Qualitative analysis revealed preservice teachers’ evaluation criteria in five categories such as pedagogy, technical usability, content, connectivity, and contextuality. The results point to recommendations for future research on evaluating the capabilities of existing mobile apps with key players such as in-service and preservice teachers. Keywords mobile learning, mobile apps, usability testing, preservice teachers

1

Middle East Technical University, Ankara, Turkey

Corresponding Author: Evrim Baran, Department of Educational Sciences, Middle East Technical University, Dumlupinar Bulvari No: 1 Egitim Fakultesi, Egitim Bilimleri Bolumu Orta Dogu Teknik Universitesi, Cankaya 06800, Ankara, Turkey. Email: [email protected]

2

Journal of Educational Computing Research 0(0)

During the past two decades, digital technology has become much more portable and networked, making mobile devices pervasive in everyday life. Developments such as location-based technologies; image, audio, and video capture; and context awareness have increased mobile devices’ adaptability within authentic settings (Martin & Ertzberger, 2013). Growing use of devices such as smartphones and tablets has arguably been supported by their accessibility and the concomitant development of mobile applications or apps, enduser software programs designed to extend the capabilities of mobile devices by enabling users to perform particular tasks (Martin et al., 2011). With the widespread adoption of mobile technologies and apps in society, particularly by young people, educators are now looking for opportunities to integrate these technologies into teaching and learning environments. When mobile learning is integrated into teacher education, it enhances teacher learning with mobile tools and equips them with the knowledge and skills to incorporate mobile learning in their own teaching (Baran, 2014). Mobile tools have the potential to enhance preservice teachers’ learning experiences by engaging them within personal, contextualized, connected, and collaborative learning environments (Burton et al., 2011; Cushing, 2011). Preservice teachers’ engagement with their disciplines (e.g., scientific inquiry) can be enhanced with mobile tools, which provide opportunities to explore real-world phenomena within personal contexts (Kearney & Maher, 2013; McCaughtry & Dillon, 2008). Other advantages include connecting preservice teachers with a larger community of teachers, mentors, and teacher educators during their field experiences (Cushing, 2011) and engaging them in collaborative knowledge construction (Ja¨rvela¨, Na¨ykki, Laru, & Luokkanen, 2007). Exposing preservice teachers to mobile learning environments within teacher education increases their likelihood to use mobile tools in their own teaching practices (Husbye & Elsener, 2013). Teacher educators have used technology evaluation activities to highlight connections between technologies, pedagogies, and content areas (Angeli & Valanides, 2005). Our previous research has revealed that by engaging in evaluation activities on mobile apps, preservice teachers may develop ideas about integrating these tools into their future classrooms and further contribute to the much-needed review of these apps (Baran & Khan, 2014). Research on the development of mobile app selection with inservice teachers has provided valuable tools (e.g., Green, Hechter, Tysinger, & Chassereau, 2014), but the literature lacks research on the evaluation of mobile apps with preservice teachers who are the future users of these technologies. This research, therefore, aims to examine the criteria for evaluating the affordances of educational mobile apps as perceived by preservice teachers.

Baran et al.

3

The Characterization of Mobile Learning: Theoretical Perspectives Characterization of mobile learning is critical for developing sound rubrics and tools to evaluate the quality of mobile learning environments. An analysis of mobile learning definitions in the literature revealed five dominant concepts: portability, authenticity, contextuality, social interactivity, and personalization (Kearney, Schuck, Burden, & Aubusson, 2012; Laurillard, 2007; Pachler, Cook, & Bachmair, 2010; Sharples, Taylor, & Vavoula, 2007; Stanton & Ophoff, 2013; Traxler, 2007; Uden & Hwang, 2013). Portability as a distinguished feature of mobile learning rests on the assumption that learners are continually on the move. Mobile learning technologies allow learners to receive and send information from different places, rather than a fixed location such as a classroom (Sharples et al., 2007). Many mobile devices can be carried while learners are in motion, allowing them to choose devices according to the demands of their context and needs (Saccol, Barbosa, Schlemmer, & Reinhard, 2011). For example, visitors of a botanic garden can use handheld GPS devices to hear audio information about flowers and shrubs (Sharples et al., 2007) or students on a geographical field inquiry can utilize an app on their smartphones to discuss textual information and images of rock types and landforms (Chang et al., 2012). Because mobile devices travel easily with the user, portability creates authentic and contextual mobile learning environments (Kress & Pachler, 2007; Stanton & Ophoff, 2013). Mobile learning environments have the potential to support authentic learning activities (Kearney et al., 2012; Uden & Hwang, 2013). Learning across contexts (Cavus & Al-Momani, 2011) is one of the significant distinguishing features of mobile learning compared with traditional e-learning (Song, Park, & Kim 2011). The efficiency and popularity of mobile devices have prompted research on the practice of new technologies in supporting authentic activities in which students have access to and participate in models of real-life activities (Uden & Hwang, 2013). Learners can experience actual or simulated materials and activities of a particular learning domain beyond the classroom setting, and they are on the move while collecting, sharing, and processing data. Mobile learning can be contextually sensitive (Stanton & Ophoff, 2013; Traxler, 2007). Contexts such as location, time, individual characteristics, social expectations, resources, goals, and individual and group-based tasks play important roles in learning within mobile learning environments (Brown et al., 2010; Uden & Huwang, 2013). In their research on generating a unified mobile learning method, Stanton and Ophoff (2013) discussed three contexts influential in mobile learning environments: (a) learner status (preferences, demographics, and history), (b) situation (classroom, street, and museum), and (c) learning environment, where the learner’s status and

4

Journal of Educational Computing Research 0(0)

situation meet in the digital sphere. Mobile technologies have the ability to adapt to a learner’s status and situation, providing data anywhere and anytime. One advantage of mobile devices is their affordance of social interactivity in the form of more portable data exchange and collaboration. Such devices can be connected to data collection machines (e.g., PCs), other mobile devices, or shared networks, where social interactivity can be maximized (Klopfer, Squire, & Jenkins 2002). Mobile learning environments may afford expert–peer and peer–peer collaboration whereby students can consume, produce, and exchange information and artifacts across time and space (Kearney et al., 2012; Uden & Hwang, 2013). For example, students can use their mobile devices to collect and share online resources and visit discussion boards and chat sessions regardless of time or place via learning management systems (Ebner, 2009). They can also use mobile phones to post notes and photos on a shared Twitter account to increase social navigation and learning during a museum investigation (Charitonos, Blake, Scanlon, & Jones, 2012). With advances in mobile social features (e.g., social networking and data sharing), certain tools now present opportunities to test theories that consider learning as a constructive and social process (Sharples et al., 2007). Mobile devices also support personalization or customized scaffolding based on a user’s personal path of investigation (Klopfer et al., 2002). In their study, Kearney et al. (2012) pointed out two key features of personalization: agency and customization. Agency is defined as one’s ability to deal with and influence the world through the use of mobile devices. Learners use their expertise to make choices among appropriate technologies in terms of their cultural habits regarding media use and learning (Kearney et al., 2012). Educators can create mobile learning environments that provide the necessary structure and scaffolding while offering enough flexibility to enable learners’ knowledge building and control, as well as their self-direction (McLoughlin & Lee, 2008).

Methods The qualitative exploratory case study served as the overall mode of inquiry in examining criteria emerged from the usability tests conducted with preservice teachers. Case study helped to capture detailed accounts of preservice teachers’ evaluation of the educational mobile apps (Yin, 2003). Under the naturalistic notion of transferability (Lincoln & Guba, 2002), the research processes were described in detail to provide a rich account and offer other researchers a basis for comparison to their own unique contexts. The research followed three phases: (a) developing the initial codebook, (b) conducting usability tests, and (c) analyzing the usability data. Table 1 presents the phases of this study.

Baran et al.

5

Table 1. Phases, Purpose, and Methods of the Study. Phase 1. Developing initial codebook 2. Conducting usability tests 3. Analyzing usability data

Purpose Identify initial coding categories Generating criteria for mobile app evaluation Refining codebook categories; finalizing mobile app evaluation tool

Methods Literature survey Usability tests with 19 preservice teachers Content analysis

Developing the Initial Codebook The first phase of the research included developing the initial codebook by reviewing existing research and development work on mobile learning evaluation criteria. This review revealed that although scholars perceive mobile learning in a variety of ways, several aspects frequently appear in characterizations of mobile learning environments, including portability, authenticity, contextuality, social interactivity, and personalization. These five concepts, along with a review of recent work on the development of mobile app evaluation rubrics such as Green et al.’s (2014) Mobile App Selection for Science (MASS) rubric, helped to determine the initial categories of the codebook: contextuality, pedagogy, technical usability, content, and connectivity.

Conducting Usability Tests The second phase of the research included planning and conducting usability tests with preservice teachers. Participants were enrolled in four subject domains within the K–12 teacher education program: math, science, English language education, and computer education. The usability tests aimed to record preservice teachers’ interactions with educational mobile apps and gather their thoughts on the apps’ educational potential (Rubin & Chisnell, 2008). The usability tests followed a subject test design that required participants to follow a number of task paths for each app. To select the apps, an extensive search was conducted in databases such as the iOS App Store and Google Play, applying the following initial criteria: (a) free mobile apps that (b) covered content domains such as science, mathematics, computer education, and teaching English as a foreign language (EFL), (c) received high ratings, and (d) allowed different interactions with the content. A total of 22 mobile apps were selected (see Table 2). After selecting the apps for all subject areas, at least three tasks were determined for each to encourage the preservice teachers to explore. These tasks were

6

Journal of Educational Computing Research 0(0)

Table 2. Selected Mobile Apps. Field Science

English

Mathematics

Computer

Apps

Content

Project Noah Exoplanet Spacecraft Skeleton 3D Virtual Heart Nova Elements Monster Physics Duolingo Hello-hello Phrasalstein Toontastic GR Wonderland Preschool Math Geogebra i-Tooch Math Blaster Chicken Coop Fractions Hopscotch Kodable Cargo Bot Scratch Codecademy

Biology Astronomy Astronomy Biology Biology Chemistry Physics General English General English Phrasal verbs English storytelling Nouns, verbs, and adjectives Preschool math Geometry Elementary school math Arithmetic Fractions/estimation Computer programming Algorithms Computer programming Computer programming Computer programming

intended to evaluate the mobile apps’ functions, content presentation, and feedback, and they included statements such as “draw a line with given coordinates (x ¼ 3, y ¼ 4, x ¼ 5, y ¼ 3) in Geogebra” or “add a new spotting in Project Noah.” The selected apps were installed on an Apple iPad mini and a Samsung tablet, depending on their IOS and Android compatibility, and they were categorized by content areas. The usability tests were conducted in a reserved room at the faculty of education. The room was arranged to allow the researchers to observe and assign tasks while the participants interacted with the mobile devices. The usability room setup is illustrated in Figure 1. Each usability session lasted 60 to 90 minutes, and the following procedures were followed: (a) setting appointments with the participants, (b) preparing the tablet and video camera in the room, (c) gathering participants’ consent signatures, (d) filling out the demographics information sheet, (e) explaining testing

Baran et al.

7

Figure 1. The usability room setup.

procedures such as the think-aloud method, (f) guiding subjects in exploring the mobile apps with selected tasks, (g) conducting debriefing interviews about the mobile app, and (h) concluding the test. All usability testing was recorded. Figure 2 illustrates a usability test in session.

Participants Participants included 19 fourth-year preservice teachers (6 male and 13 female) from the departments of elementary science education (n ¼ 4), elementary math education (n ¼ 4), English language education (n ¼ 6), and computer education (n ¼ 5). The preservice teachers’ ages ranged between 21 and 24. They were selected through a nomination process carried out with instructors who taught methods and technology courses in their departments. Four instructors were asked to nominate five students who (a) were among the top three students in their class, (b) had interest in integrating technology into their future classrooms, and (c) had completed their mandatory field experiences. Such senior students were selected because of their broader knowledge of teaching methods and content, which was expected to help them to judge the educational value of the mobile apps. Preservice teachers completed two semesters of school experience with requirements of observing, reflecting, and teaching in real-classroom settings. All preservice teachers indicated that they used digital technologies frequently, with Facebook, Skype, and Twitter being most common.

8

Journal of Educational Computing Research 0(0)

Figure 2. A usability test in session.

The 17 who owned smartphones reported using them mostly for e-mail, pictures, and web searches. Most common mobile apps used by preservice teachers were Facebook, Whatsup, and Twitter.

Data Sources Data sources included the recordings of the 19 usability sessions and the debriefing interviews conducted with the preservice teachers. The data also included audio recordings of the preservice teachers’ think-aloud sessions when they completed tasks on the mobile apps. Debriefing interviews were conducted after each mobile app evaluation, which concluded with guiding questions regarding the preservice teachers’ thoughts about the pedagogical affordances, limitations, and educational uses of the apps, such as the following examples: . At which part or parts did you face difficulty while accomplishing tasks? . Would you use this app in your courses? Why or why not? . Thinking as a whole, which components—audio, video, feedback, interface—were powerful or not powerful in terms of teaching and learning? . What would you suggest to make this app more powerful educationally? . Could you provide scenarios for educational uses of this app?

Baran et al.

9

Other questions aimed to gather participants’ suggestions about criteria for evaluating educational mobile apps. All usability sessions and interview recordings (1,168 minutes total, an average 61 minutes per session) were transcribed for further analysis.

Data Analysis Data analysis followed two stages. First, tentative categories were identified for the initial codebook after a review of existing definitions, characterizations, and pedagogical affordances noted in the literature. For example, the criteria coded under technical usability were categorized using Nielsen’s (1995) usability heuristics. The first version of the codebook had five categories: contextuality, pedagogy, technical usability, content, and connectivity. In the second phase, during open coding, the transcript of each usability session was reviewed line by line to generate evaluation criteria and subcriteria while refining existing codes and adding new ones. Using NVivo 11 software, manual coding was performed through iterative reviews of the transcripts (Miles & Huberman, 1994). A unit of analysis was considered an utterance or continuous set of utterances that conveyed an identifiable idea (Aviv, 2001). For instance, “I think overall recording student activity is good for helping a student to see his/her own progress. The app can have scores, tables, dates, and graphics that show student progress” (Main category-Pedagogy; criteria–assessment; subcriteria-tracking and reporting learner progress). Once new codes emerged or existing categories and codes were revised, the researchers reviewed previous usability data for confirmation. Researchers coded entire set of transcripts together to reach consensus through extensive dialog, as the coding process required 100% agreement on coding and categories. To eliminate potential bias, codes were continuously checked to assure consistency in analysis. The final codebook included five criteria categories: (a) pedagogy (e.g., pedagogical strategy, motivation, learner, multimedia, and assessment), (b) technical usability (e.g., visibility, user control, efficient use, support, recognition, visual design, error prevention, consistency, and standards), (c) content (e.g., curricular fit, scope, validity, sequence, and language), (d) connectivity (e.g., sharing and communication), and (e) contextuality (e.g., authenticity and learning in different contexts). Categories, criteria, and subcriteria codes identified from the analysis are presented in Table 3 with selected data excerpts. After multiple reviews, frequencies under each category were calculated and, using simple percentages, the criteria were ordered.

Trustworthiness Several measures were taken to ensure the trustworthiness of the research (Lincoln & Guba, 1985). Using multiple sources such as think-aloud audio

10

User control

Up-to-date

Sharing

Technical usability

Content/Validity

Connectivity/ Communication Contextuality/ Authenticity

Different contexts

Interest

Code

Pedagogy/Motivation

Category

Table 3. Coding and Examples.

Allowing learners to share content generated by using the app with others Including activities that provide real-life practices in classrooms (via simulations, e.g., engaging with space crafts or planets, different activities such as creating a particular atom by using electrons, protons, and neutrons)

The app provides activities or content that can appeal to learners’ interest (e.g., the app can provide 3D representations related to the content or activities requiring active participation) Users should select and sequence tasks on the app Providing timely information and representations

Description

I can freely make choices in the program The content needs to be updated continuously with new species discovered It would be nice to share and watch what others shared It (the app) can be used in school when available. It can be used at home to reinforce student learning

There are so many things that can attract child’s attention

Quote

Baran et al.

11

recordings and interviews helped to present a complete picture as the criteria emerged. The context, procedures, and steps have been described in detail to help other researchers transfer findings. Conducting a pilot usability test before the actual data collection process helped to confirm the usability session setup and finalize data collection tools and procedures. Ethical clearance was obtained from the university, and participant consent was gathered before each test. Two researchers reviewed the data together to ensure the coding was reliable. Debriefing was conducted with the third researcher to confirm the code descriptions and coding procedures. All three researchers met continuously to discuss the data analysis and emerging themes.

Results Criteria for Evaluating the Affordances of Educational Mobile Apps Analysis revealed that pedagogy was dominant in the preservice teachers’ evaluations of the mobile apps; it comprised about 51% of the total criteria coded. Other prevailing evaluation categories were technical usability (about 29%) and content (about 16%). Connectivity and contextuality were least cited, comprising about 1.7% and 1.6% of the total criteria, respectively. The frequencies and percentages of the criteria categories are presented in Table 4.

Pedagogy These results show that while evaluating mobile apps, the preservice teachers mainly focused on pedagogical affordances, followed by technical and content features. Common criteria for evaluating the pedagogical value of the mobile apps included pedagogical strategy, motivation, learner fit, multimedia, and assessment, as shown in Table 5.

Table 4. Frequencies and Percentages for the Mobile App Evaluation Categories. Category Pedagogy Technical usability Content Connectivity Contextuality Total

F

%

659 379 210 22 21 1,291

51.05 29.36 16.27 1.70 1.63 100

12

Journal of Educational Computing Research 0(0) Table 5. Pedagogy Criteria and Subcriteria Criteria Pedagogical strategy

Motivation

Learner fit

Multimedia Assessment

Subcriteria Guidance Content-based pedagogy Reinforcing learning Stimulating learner interest Fun elements Interactivity Rewards (intrinsic/extrinsic) Engagement Learner level Learner needs Misconceptions Learner pace Multimedia design New representations Assessment features Tracking and reporting learner progress

The integration of appropriate pedagogical strategy in an app was considered as the most critical evaluation criteria under pedagogy, with a specific focus on guidance, content-based pedagogy, and reinforcement. Preservice teachers noted the value of guiding learners while providing specific, relevant, and timely taskrelated feedback, hints, and pedagogical instructions. Frequently referring to pedagogical strategies used in their subject domains, the preservice teachers also emphasized the appropriate use of content-based pedagogies, such as strategies for teaching programming, inquiry-based methods in science education, and communicative approaches in English language education. For example, one EFL preservice teacher commented while evaluating a mobile app on teaching EFL: “I will suggest this to all my students because it is really nice. It begins from the beginning with both writing and reading, with continuous reinforcement and repetition.” Fostering motivation emerged as the second most dominant criteria under pedagogy, with specific focus on stimulating learner interest, including elements of fun and interactivity, using intrinsic and extrinsic rewards, and providing learner engagement. Because some apps included game elements such as rewards and challenges, some preservice teachers noted the importance of these features for maintaining learner engagement, but one EFL preservice teacher expressed concern about gamification: My criteria is that gaming in the app should not go beyond the purpose of teaching language. For example, the app I examined was too much of a game. The student

Baran et al.

13

would focus more on completing the challenges in the game but not on actual learning.

Another item under pedagogy was learner fit, including subcriteria of how well the app content and design fit learners’ skill levels, targeted learners’ needs, addressed learners’ misconceptions, and allowed learners to progress at their own pace. Fit between content and design in conjunction with learners’ levels proved to be particularly important. Using pedagogically sound multimedia elements emerged as another criterion under pedagogy, with specific focus on multimedia design and providing new representations. Mobile apps’ affordances of presenting content in multiple multimedia forms (e.g., pictures, videos, text, and sounds) and providing different representations of content were noted as the critical evaluation points. Participants indicated that the apps could supplement classroom teaching by sharing topics in formats teachers typically lack, such as 3D graphics and animations. One science education preservice teacher, for example, commented: “It is hard to show in class the 3D representations of the planets. It would be easier to learn here. For example, the student can easily see the distance between the planets.” The last criteria that emerged under pedagogy was the use of assessment with specific focus on assessment features and tracking and reporting learner progress. Assessing learning via different tests was key affordances of mobile apps. For example, one computer education preservice teacher observed how mobile programming apps would bring practical value to the classroom by helping to track learner progress. A preservice teacher in English language education stated that testing students learning at different levels after each topic and unit was another good app feature. A math preservice teacher commented: The app should do the measurement well. Are the test items aligned with objectives? Also, we can find these questions in the book. The app should bring new assessment features that are not available in books.

Tracking and reporting learner progress were considered essential to mobile apps. A preservice teacher in math education noted that reports in the app could also be shared with other teachers to inform them about student progress, give feedback, and integrate measurements into the classroom.

Technical Usability Technical usability criteria refer to the practicality of a mobile app’s technical features and focus on the interface and user–mobile app interaction. The criteria included visibility, user control, efficiency, support, recognition, visual design, error prevention, and consistency and standards, as shown in Table 6.

14

Journal of Educational Computing Research 0(0) Table 6. Technical Usability Criteria and Subcriteria. Criteria Visibility User control Efficient use

Support

Recognition Visual design

Error prevention Consistency and standards

Subcriteria — — Directions Simplicity Saving Language support Tutorial Search bar Help Clarity of visual representations Limited training about usage Minimalist design Aesthetics Realistic Preventing and recovering from errors Consistent design Compatibility

While evaluating technical usability, the preservice teachers noted that the apps’ functions should be visible to the users, such that clear directions and feedback were provided when interacting with the apps. A preservice teacher in computer education noted: “I didn’t understand what to do here. It doesn’t say what to do here and I don’t know where to get additional information.” Another preservice teacher did not understand what to do at the start screen or recognize the play button during exercises. The second most common criterion under technical usability was giving students the control to select and sequence tasks in the app while also providing them choices. When the preservice teachers used the apps, they focused on their efficiency, such as having clear instructions or visuals for tools, a simple interface, the option to record progress, and familiar language. Support emerged as another important criterion. Preservice teachers frequently noted that an app should provide a tutorial, a search option, and an easily accessible help menu. One preservice math teacher, for example, stated: “Generally there are manuals that help us understand what we do with the app. But I can’t see where such a manual is in this app, if any exists.” Other notable criteria under technical usability were a focus on clarity of visual representations (e.g., tools, menus, buttons, etc., with clear icons) and the need for limited training. One preservice math teacher commented: “The themes are similar. Even if I didn’t use the app before, it is presented clearly here.

Baran et al.

15

When I look now, I can understand that I need to draw a polygon.” Visual design, with elements of minimalism, aesthetics, and realistic images, was also noted as a criterion for usability evaluation. Preservice teachers particularly valued simple interfaces without too many visuals to distract learners. Realistic elements in the interface design were also mentioned. For example, one of the preservice math teachers commented that the Geogebra app interface represented dot paper, and a preservice science teacher pointed out how the colors in the Spacecraft app echoed those in actual space. Error prevention and recovery and consistency in design and compatibility were the final criteria that emerged under technical usability, representing around 30% of the evaluation codes.

Content Preservice teachers’ evaluation criteria for the mobile apps’ content focused on curricular fit, scope, validity, sequence, and language, as shown in Table 7. One significant content criteria was the fit between the apps’ content and the curriculum or curricular goals. Scope in terms of comprehensive and detailed coverage of content was considered especially important. For example, a preservice teacher in science education stated: “The subject of space is generally left

Table 7. Content, Connectivity, and Contextuality Criteria and Subcriteria. Criteria Content Curricular fit Scope Validity

Sequence Language Connectivity Sharing Communication Contextuality Authenticity Learning in different contexts

Subcriteria — Coverage Detailed Accurate Timely Culturally appropriate — — — — — —

16

Journal of Educational Computing Research 0(0)

to the end of the year, and teachers may not have time to cover. This app is covering that, so that is good.” The validity of content in terms of presenting upto-date, culturally appropriate information was another content criterion. For example, a preservice science teacher indicated that the species database in one app should be updated regularly, and a preservice English teacher stated that cultural elements in another app should be appropriate to student contexts. Similarly, the preservice teachers evaluated the language used in the apps in terms of tone and verbiage. One preservice English teacher commented, “The videos I watched at the beginning and the animations felt artificial to me as the voice recordings sounded like robots.” The content category accounted for 16% of coding. Emphasizing mainly alignment with curricular objectives, scope, and valid, up-to-date information presented in an appropriate sequence with familiar language, the preservice teachers considered the evaluation of the apps’ content as critical to making decisions about their educational value.

Connectivity Connectivity accounted for 1.70% of the criteria coded for the evaluation of educational mobile apps. Criteria related to connectivity included sharing and communication criteria (see Table 7). The preservice teachers stated that apps should allow communication with other students and teachers while sharing content and progress within the app. For example, one of the preservice math teachers stated, “If students could share their activities with the teachers on mobile apps, teachers could help them easily.” Another preservice English teacher explained, “The mobile app I used could be used for a task altogether and students could study in groups collaboratively.” Examples of connectivity included sharing homework with others via built-in messaging systems or social media tools or allowing learners to communicate while making comments, liking, or grading content. One of the preservice science teachers stated: “For example, others can comment on the pictures and descriptions I upload to the system, or they can give points to those. In the end, there would be a user generated list.” The connectivity criteria accounted for a small percentage of the total codes. However, connection and collaboration features were still valued, particularly the potential to connect students’ mobile app activity with teachers, classrooms, and other learners.

Contextuality The context criteria accounted for 1.63% of the criteria coded for the evaluation of educational mobile apps, focusing on authenticity and learning in different contexts, as shown in Table 7. The preservice teachers valued the educational mobile apps’ potential to create authentic tasks that provided real-world

Baran et al.

17

relevance. For example, one preservice science teacher noted the importance of using science apps in museums, botanical gardens, nature, or the field to promote students’ inquiry learning processes. Another preservice teacher highlighted how mobile apps’ context-awareness capabilities may allow for the recording of information about students and their learning environments.

Discussion This study provided data from usability tests conducted with preservice teachers, including their thoughts while using the apps and their insights regarding evaluation criteria. Data analysis revealed five categories, including pedagogy, technical usability, content, contextuality, and connectivity. More than half of the total criteria coded under pedagogy suggested that the preservice teachers were mainly concerned with the pedagogical affordances of the mobile apps. These findings are in line with Green et al.’s (2014) study conducted with teachers, which indicated a strong emphasis on pedagogical elements and the integration of mobile apps into classrooms. The preservice teachers in this study frequently considered the appropriateness of the general pedagogical elements of the apps, evaluating how effectively they implemented pedagogical strategies, enhanced motivation, addressed learner needs, implemented multimedia elements, and assessed students’ learning. An emphasis on content-based pedagogy evaluation reflects preservice teachers’ focus on the use of domain-specific pedagogies in mobile apps as well, such as the integration of inquiry-based learning in science education, communicative approaches in English language education, and design-based learning in computer education. The alignment between content, pedagogy, and technology is critical to effective technology integration (Mishra & Koehler, 2006). Mobile apps’ integration of domain-specific pedagogical strategies that utilize the affordances of mobile platforms enhances their pedagogical value (Green et al., 2014). Central to the design of mobile learning environments is the pedagogical approaches they utilize, with consideration to the affordances of mobile tools (Kearney et al., 2012). Preservice teachers’ suggestions for pedagogy criteria support the value of evaluating how well mobile apps implement both domain-independent and domain-specific pedagogies. Assessing the usability of mobile tools and how they affect learning is a priority in their evaluation (Vavoula & Sharples, 2009). Investigations into mobile usability have identified it as a growing field that advocates user-centered design (Kukulska-Hulme, 2007). The emergence of a vast number of educational mobile apps and their increasing prevalence in formal and informal learning environments demand the development of frameworks and approaches for usability testing as well as criteria for evaluating their usability. Technical usability emerged as the second most important evaluation category in this research, including aspects aligned with common usability criteria (Nielsen, 1995) and similar to those proposed in mobile usability literature, such as visibility,

18

Journal of Educational Computing Research 0(0)

user control, efficient use support, recognition, visual design, error prevention, and consistent standards (Economides & Nikolau, 2008; Huang, Kuo, Lin, & Cheng, 2008; Hussain et al., 2008; Kukulska-Hulme, 2007; Lam, Lam, Lam, & McNaught, 2009; McKnight & Cassidy, 2010; Mu¨ller et al., 2009). The strong emphasis on technical usability calls for mobile interfaces and system designs that follow general usability criteria but also confirm the affordances of emerging mobile apps, such as portability, personalization, and contextualization (Kearney et al., 2012). One of the most important evaluation criteria for educational mobile apps is how accurately and efficiently content is represented, including accuracy of concepts and relevance to learning objectives (Green et al., 2014). This research revealed content as the third largest category in preservice teachers’ evaluation, focusing on features such as curricular fit, validity, sequence, scope, and language. The use of content in the evaluation category reflects stronger emphasis on correct and up-to-date content that supports learning objectives in mobile apps. Particularly, connections to the curriculum were essential to preservice teachers in guiding them to use mobile apps in classrooms to support students’ learning. The two least loaded criteria categories in this research were connectivity and contextuality. Most of the apps that were provided to preservice teachers for evaluation in this research were not suited to situated, contextualized, authentic activities. Researchers and developers are beginning to explore the affordances of mobile apps for creating contextualized and situated tasks using the new features such as context awareness capabilities. While only a limited number of apps evaluated in this research provided opportunities for context variety or the application of connectivity features, the preservice teachers still considered authenticity, learning in different contexts, sharing, and communication as evaluation criteria. Mobile app learning activities take place in authentic contexts naturally, allowing learners to process tasks that simulate real-world practices (Kearney et al., 2012; Uden & Hwang, 2013; Walker, 2011). However, it is also the activity that teachers provide and the ways teachers choose to use the technologies that can create authentic and contextual learning environments with mobile apps. Investigating learning actitivites that support authentic and contextualized learning on mobile apps with preservice teachers is critical for future research to determine how they perceive authentic learning with mobile apps. Only two works in the literature provide useful frames for evaluating mobile apps, Green et al.’s (2014) MASS and Walker’s (2011) Evaluation Rubric for Mobile Applications (ERMA). Other resources include online tools such as Schrock’s (2013) app evaluation rubrics. Walker’s rubric defined curriculum connection, authenticity, feedback, differentiation, user friendliness, and student motivation as evaluation categories. Meanwhile, Green et al.’s rubric focused on the science domain, including categories such as accuracy, relevance of content, sharing findings, feedback, scientific inquiry and practices, and navigation.

Baran et al.

19

Both of these rubrics were developed based on literature reviews and expert evaluation methods such as the Delphi method and questionnaires (Green et al., 2014; Walker, 2011). Although expert evaluation is beneficial for design, development, and evaluation studies, it only reflects the experts’ perspective on the related material. User experience, on the other hand, is valuable for studies, revealing how users actually interact with a product (Rubin & Chisnell, 2008). Therefore, usability testing as an evaluation method may allow for the collection of data as users perform realistic tasks. The limited number of rubrics in the literature, along with online app evaluation rubrics provide valuable resources, but the literature still demands evaluation activities grounded in tests conducted with end users such as preservice and in-service teachers. The preservice teachers’ criteria generated from this research are in line with previous research on evaluating mobile apps (Green et al., 2014; Walker, 2011). The categories emerged from this research such as content, authenticity, usability, and motivation also appeared in Green et al.’s (2014) MASS and Walker’s (2011) ERMA rubrics. This study further revealed detailed criteria under pedagogy such as the use of pedagogical strategy, motivation, learner, multimedia, and assessment. Because the usability of educational mobile apps played an important role in determining how well learners interact with the app, MASS and ERMA included criteria for usability such as navigation (Green et al., 2014) and user friendliness (Walker, 2011). This research further detailed the components of technical usability such as support, recognition, and error prevention. Understanding preservice and in-service teachers’ criteria for evaluation provide insights to teacher educators in creating opportunities for their engagement with educational mobile apps. The need to rethink mobile learning design and develop new methods for the evaluation of mobile learning environments has been widely noted in the literature (Sharples, 2013). Research investigating the usability of mobile learning environments has included criteria such as learnability, operability, understandability, metaphor, interactivity, and learning content (Parsons & Ryu, 2006). Despite an emphasis on the importance of usability, however, a great number of mobile learning environments have been designed without considering educational concerns or usability problems (Sharples, 2013). Kukulska-Hulme (2007) has argued that mobile usability approaches have mostly focused on technical aspects, and research that considers pedagogical perspectives would improve the quality of mobile learning experiences. This conceptualization of mobile learning gives evaluation a direction in terms of selecting priorities (Traxler, 2007). This study contributed to research by building on previous research on mobile app evaluation and further detailing evaluation criteria with preservice teachers. Conducting usability studies with preservice teachers in different domains allowed generating criteria that can be tested and validated in different subjects. The results of the current exploratory case study are limited to the selected preservice teacher population. The criteria list emerged from this

20

Journal of Educational Computing Research 0(0)

research is unique to the context of preservice teachers participated in the study and may not be generalized to other teacher education contexts. Different results and criteria may emerge from usability tests conducted with different mobile apps or populations such as in-service teachers or preservice teachers from other disciplines and contexts. While case studies are not accountable for ensuring sampling representativeness, presenting sufficient details about the context and the evaluation method can provide a base for the transferability of the findings to other teacher education settings. For example, researchers interested in conducting similar evaluation activities with preservice and in-service teachers may take the method as a model to further refine and adopt the criteria to their own contexts.

Conclusions Researchers have predicted that mobile devices will play a critical role in education in the near future, considering their prevalence among children and their capabilities for new learning and teaching opportunities (Johnson, Adams Becker, Estrada, & Freeman, 2015). This trend is reflected by an increasing number of mobile apps being developed for educational purposes. While the value of these apps is now being recognized, limited resources are available to help preservice and in-service teachers to evaluate their pedagogical affordances or to make decisions about using them (Kearney et al., 2012). Most importantly, preservice teachers, as potential future users of educational mobile apps, have limited opportunities to evaluate or examine their pedagogical affordances. This research addressed this gap in the literature, including usability tests conducted with preservice teachers on selected educational mobile apps.

Implications for Practice The educational mobile app evaluation criteria list generated from this research hold promise for implications in education. First, teacher educators may integrate similar mobile app evaluation activities to their professional development programs and methods courses to engage in-service and preservice teachers in examining the pedagogical affordances of mobile apps for specific subject domains. The list derived from this research may be refined through collaborative activities where preservice and in-service teachers test the criteria on selected mobile apps (e.g., for science learning) and adopt the list to their own context. Teachers may also conduct similar usability tests with children to examine their interaction with the educational mobile apps. Second, the criteria list may provide mobile app developers a guide to focus on specific pedagogical, technical, and contextual aspects. The research-based criteria may help designers make design decisions while considering particular criteria for usability such as user control, visual design, support, and error prevention. Mobile app developers and educators may collaborate on designing new apps with focus on pedagogical

Baran et al.

21

criteira such as the use of pedagogical and motivational strategies, multimedia use, assessment features, and the level of learner fit. Third, the criteria list may be adapted to the evaluation of apps in different contexts, such as special education, physical education, and early childhood education. Refining the criteria list with different end users would contribute to the collection of evaluation tools specialized for certain subject domains and contexts.

Recommendations for Future Research The findings warrant future testing of the educational mobile app criteria delineated in this study with other user groups. Further research is needed to gather in-service teachers’ and experts’ thoughts about educational mobile app evaluation and refine the criteria generated by preservice teachers in this study. Our future research will refine the criteria developed with preservice teachers with input from in-service teachers from different disciplines. The contextuality and connection criteria that emerged from this research have been underexamined and undervalued both in research and practice, warranting further work. The new capabilities of mobile tools such as context-awareness, collaboration, and visualizations allow for new learner interaction within a mobile learning environment. A promising research direction is to look at how educational mobile apps with these new affordances can be evaluated and tested for evaluation categories identified in this research such as pedagogy, usability, content, and contextuality. The methodology used in this study could be further tested with selected educational mobile apps that use these features, including additional examination of the criteria that emerged during this research. Considering the strong emphasis placed on pedagogical elements used in the mobile apps during evaluation, rubrics could be designed to judge their pedagogical value. Developing and testing research-based scales and rubrics would help teachers to select appropriate apps for supporting learning both in and out of their classrooms. Finally, mobile app evaluation activities could be integrated into teacher education programs. Further research could examine how similar evaluation activities may enhance preservice and in-service teachers’ development of knowledge and skills regarding mobile app integration into classrooms. Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by European Commission 7th Framework Programme (Grant number: PCIG13-GA-2013-618509).

22

Journal of Educational Computing Research 0(0)

References Angeli, C., & Valanides, N. (2005). Preservice teachers as ICT designers: An instructional design model based on an expanded view of pedagogical content knowledge. Journal of Computer-Assisted Learning, 21(4), 292–302. Aviv, R. (2001). Educational performance of ALN via content analysis. Journal of Asynchronous Learning Networks, 4(2), 53–72. Baran, E. (2014). A review of research on mobile learning in teacher education. Journal of Educational Technology & Society, 17(4), 17–32. Baran, E., & Khan, S. (2014). Going mobile—Science teacher candidates evaluating mobile apps. In C. Miller & A. Doering (Eds.), The new landscape of mobile learning: Redesigning education in an app-based world (pp. 258–275). New York, NY: Routledge. Brown, E., Brner, D., Sharples, M., Glahn, C., de Jong, T., & Specht, M. (2010). Location-based and contextual mobile learning. A STELLAR small-scale study. STELLAR European Network of Excellence in TEL (EU). Burton, E., Frazier, W., Annetta, L., Lamb, R., Cheng, R., & Chmiel, M. (2011). Modeling augmented reality games with preservice elementary and secondary science teachers. Journal of Technology and Teacher Education, 19(3), 303–329. Cavus, N., & Al-Momani, M. M. (2011). Mobile system for flexible education. Procedia Computer Science, 3, 1475–1479. doi:10.1016/j.procs.2011.01.034 Chang, C.-H., Chatterjea, K., Goh, D. H.-L., Theng, Y. L., Lim, E., Sun, A., . . . Nguyen, Q. M. (2012). Lessons from learner experiences in a field-based inquiry in geography using mobile devices. International Research in Geographical and Environmental Education, 21(1), 41–58. Charitonos, K., Blake, C., Scanlon, E., & Jones, A. (2012). Museum learning via social and mobile technologies: (How) can online interactions enhance the visitor experience? British Journal of Educational Technology, 43(5), 802–819. Cushing, A. (2011). A case study of mobile learning in teacher training—Mentor me (mobile enhanced mentoring). MediaEducation (MedienPa¨dagogik), 19, 1–14. http:// www.medienpaed.com/globalassets/medienpaed/19/cushing1106.pdfhtt Ebner, M. (2009). Interactive lecturing by integrating mobile devices and micro-blogging in higher education. Journal of Computing and Information Technology, 17(4), 371–381. Economides, A. A., & Nikolaou, N. (2008). Evaluation of handheld devices for mobile learning. International Journal of Engineering Education, 24(1), 3–23. Green, L. S., Hechter, R. P., Tysinger, P. D., & Chassereau, K. D. (2014). Mobile app selection for 5th through 12th grade science: The development of the MASS rubric. Computers & Education, 75, 65–71. Huang, Y., Kuo, Y., Lin, Y., & Cheng, S. (2008). Toward interactive mobile synchronous learning environment with context-awareness service. Computers & Education, 51(3), 1205–1226. Husbye, N. E., & Elsener, A. E. (2013). To move forward, we must be mobile: Practical uses of mobile technology in literacy education courses. Journal of Digital Learning in Teacher Education, 30(2), 46–51. Hussain, Z., Lechner, M., Milchrahm, H., Shahzad, S., Slany, W., Umgeher, M., . . . Wolkerstorfer, P. (2008). Agile user-centered design applied to a mobile multimedia streaming application. In A. Holzinger (Ed.), HCI and usability for education and work (pp. 313–330). Berlin, Germany: Springer.

Baran et al.

23

Ja¨rvela¨, S., Na¨ykki, P., Laru, J., & Luokkanen, T. (2007). Structuring and regulating collaborative learning in higher education with wireless networks and mobile tools. Educational Technology & Society, 10(4), 71–79. Johnson, L., Adams Becker, S., Estrada, V., & Freeman, A. (2015). NMC Horizon report: 2015 K-12 edition. Austin, TX: The New Media Consortium. Kearney, M., & Maher, D. (2013). Mobile learning in maths teacher education: Using iPads to support pre-service teachers’ professional development. Australian Educational Computing, 27(3), 76–84. Kearney, M., Schuck, S., Burden, K., & Aubusson, P. (2012). Viewing mobile learning from a pedagogical perspective. Research in Learning Technology, 20, 14406. Klopfer, E., Squire, K., & Jenkins, H. (2002). Environmental detectives: PDAs as a window into a virtual simulated world. In M. Milrad, U. Hoppe & Kinshuk (Eds.), Proceedings of the IEEE International Workshop on Wireless and Mobile Technologies in Education (pp. 95–98). Los Alamitos, CA: IEEE Computer Society. Kress, G., & Pachler, N. (2007). Thinking about the ‘m’ in m-learning. In N. Pachler (Ed.), Mobile learning: Towards a research agenda (pp. 7–32). London, England: WLE Centre. Kukulska-Hulme, A. (2007). Mobile usability in educational contexts: What have we learnt? International Review of Research in Open and Distance Learning, 8(2), 1–16. Lam, P., Lam, S., Lam, J., & McNaught, C. (2009). Usability and usefulness of eBooks on PPCs: How students’ opinions vary over time. Australasian Journal of Educational Technology, 25(1), 30–44. Laurillard, D. (2007). Pedagogical forms for mobile learning: Framing research questions. In N. Pachler (Ed.), Mobile learning: Towards a research agenda (pp. 153–175). London, England: WLE Centre. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Thousand Oaks, CA: Sage. Lincoln, Y., & Guba, E. (2002). The only generalization is: There is no generalization. In R. Gomm, M. Hammersley & P. Foster (Eds.), Case study method (pp. 27–44). London, England: Sage. Martin, S., Diaz, G., Sancristobal, E., Gill, R., Castro, M., & Peire, J. (2011). New technology trends in education: Seven years of forecasts and convergence. Computers & Education, 57(3), 1893–1906. Martin, F., & Ertzberger, J. (2013). Here and now mobile learning: An experimental study on the use of mobile technology. Computers & Education, 68, 76–85. McCaughtry, N., & Dillon, S. R. (2008). Learning to use PDAs to enhance teaching: The perspectives of preservice physical educators. Journal of Technology and Teacher Education, 16(4), 433–459. McKnight, L., & Cassidy, B. (2010). Children’s interaction with mobile touch-screen devices: Experiences and guidelines for design. International Journal of Mobile Human Computer Interaction, 2(2), 1–18. McLoughlin, C., & Lee, M. J. W. (2008). The three p’s of pedagogy for the networked society: Personalization, participation, and productivity. International Journal of Teaching and Learning in Higher Education, 20(1), 10–27. Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.

24

Journal of Educational Computing Research 0(0)

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. Mu¨ller, J., Cheverst, K., Fitton, D., Taylor, N., Paczkowski, O., & Kru¨ger, A. (2009). Experiences of supporting local and remote mobile phone interaction in situated public display deployments. International Journal of Mobile Human Computer Interaction, 1(2), 1–21. Nielsen, J. (1995). 10 usability heuristics for user interface design. Retrieved from http:// www.nngroup.com/articles/ten-usability-heuristics/ Pachler, N., Cook, J., & Bachmair, B. (2010). Appropriation of mobile cultural resources for learning. International Journal of Mobile and Blended Learning, 2(1), 1–21. Parsons, D., & Ryu, H. (2006). A framework for assessing the quality of mobile learning. In R. Dawson, E. Georgiadou, P. Lincar, M. Ross & G. Staples (Eds.), Learning and teaching issues in software quality: Proceedings of the 11th International Conference for Process Improvement, Research and Education (pp. 17–27). Southampton, UK: The British Computer Society. Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley. Saccol, A. Z., Barbosa, J. L. V., Schlemmer, E., & Reinhard, N. (2011). Mobile learning in organizations: Lessons learned from two case studies. International Journal of Information and Communication Technology Education, 7(3), 11–24. Schrock, K. (2013, March 25). Critical evaluation of mobile apps. Retrieved from http:// www.ipads4teaching.net/critical-eval-of-apps.html Sharples, M. (2013). Mobile learning: Research, practice and challenges. Distance Education in China, 3(5), 5–11. Sharples, M., Taylor, J., & Vavoula, G. (2007). A theory of learning for the mobile age. In R. Andrews & C. Haythornthwaite (Eds.), The Sage handbook of e-learning research (pp. 221–247). London, England: Sage. Song, K., Park, J., & Kim, J. K. (2011). Systematic design of context awareness mobile learning environment. International Journal of Control and Automation, 4(4), 157–162. Stanton, G., & Ophoff, J. (2013). Towards a method for mobile learning design. Issues in Informing Science and Information Technology, 10, 501–523. Traxler, J. (2007). Defining, discussing and evaluating mobile learning: The moving finger writes and having writ. . .. The International Review of Research in Open and Distributed Learning, 8(2), 1–8. Uden, L., & Hwang, G. (2013). Activity theory approach to developing context-aware mobile learning systems for understanding scientific phenomenon and theories. International Journal of Distance Education Technologies, 11(4), 30–44. Vavoula, G., & Sharples, M. (2009). Meeting the challenges in evaluating mobile learning: A 3-level evaluation framework. International Journal of Mobile and Blended Learning, 1(2), 54–75. Walker, H. (2011). Evaluating the effectiveness of apps for mobile devices. Journal of Special Education Technology, 26(4), 59–63. Yin, R. (2003). Case study research: Design and methods (3rd ed.). Thousand Oaks, CA: Sage.

Baran et al.

25

Author Biographies Evrim Baran, PhD, is an assistant professor in the Department of Educational Sciences at Middle East Technical University. Her research focuses on technology and teacher education, learning sciences, and human–computer interaction. Erdem Uygun is a PhD candidate in the Department of Educational Sciences at Middle East Technical University. His research focuses on technology and teacher education, curriculum, and instruction. Tugba Altan is a PhD candidate in the Department of Computer Education and Instructional Technology. Her research focuses on technology and teacher education and human–computer interaction.