Quality and Usability of Mashup Tools: Criteria and Evaluation †
Melusi Malinga / Stefan Gruner
Department AIFB Karlsruher Institut für Technologie Germany
[email protected]
[email protected]
ABSTRACT Web mashups can add “agility” to software construction by enabling end-users without programming skills to compose their own service software on the basis of existing services. All that is needed for such purposes is a good mashup composition tool. The question remains how helpful and useful such tools really are for their users. To this end we have conducted a survey in which the participants were confronted with a widely-used representative of mashup tool and a typical mashup composition task. By-and-large the participants of the survey did well and the mashup tools seem to be able to fulfill their promises.
Categories and Subject Descriptors D.2.8 [Software Engineering]: Metrics—quality measures; H.5.3 [Web]: Web Engineering—web services, mashups
General Terms Management, Measurement, Design, Human Factors
Keywords Mashups, Quality criteria, Usability, Empirical evaluation
1. INTRODUCTION Mashups are client-integrated web service compositions developed by putting together services from heterogeneous online sources in a very short time [17]. They are dynamic, adaptive and user-centered. As such they bring a high level of agility into software development [3]. As aggregation applications they combine data from different This paper is based on a hitherto un-published M.IT.project report by Melusi Malinga under the supervision of Stefan Gruner at the University of Pretoria [20]. The project was supported by Agnes Koschmider since her stay at the University of Pretoria as a post-doctoral research fellow in 2010-2011. †Corresponding: Stefan Gruner Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected]. SAICSIT ’13, October 8-9, East London, South Africa Copyright 2013 ACM 978-1-4503-2112-9/13/10...$15.00. http://dx.doi.org/10.1145/2513456.2513462 .
154
Agnes Koschmider
Department of Computer Science University of Pretoria South Africa
sources such as to cluster valuable information at clientside. Mashup developers use so-called “Web 2.0” techniques to effect these results [8]. They are built on widgets, open APIs, web services, and data sources. They are mostly composed by end-users, who are not programmers, with help of commercially available mashup tools [14]. These compositions are ad-hoc and cannot be predicted by the providers of those services or data on the other side of the line. Requiring no programming skills from their users, mashup tools must provide an easy and intuitive user interface, must be adaptable to different enduser environments, must allow for customization, ad-hoc integration and extension of information, and should also support collaborative development. Figure 1 shows how mashup tools are typically applied. In industry the development of mashups depends strongly, in quantity and quality, on the features of the mashup tools that are available and used. This motivates empirical evaluations of mashup tools, to assess whether they truly support their end-users as intended. Such empirical usability studies for mashups do not yet exist in large numbers, which is the motivation behind our work. A preliminary usability studies has been conducted by [25], which did not confirmed high usability for casual users. For this purpose we have carried out an empirical study survey with a group of end-users and IT professionals and one specific mashup composition tool. After a brief summary of related work and a brief overview of available mashup tools, the design and the results of our study are reported in this paper.
2.
RELATED WORK
Mashup engineering is a relatively new domain compared to software engineering. It might be argued that experience from usability analysis in software engineering can be transferred to mashup engineering. In fact, usability of software development tools has been widely addressed and evaluated for traditional [12] and open source software tools [22]. However, usability analysis of software engineering tools are not directly applicable for mashup
Figure 1: Mashup Composition according to [5]
engineering tools since software developers are technically skilled and have programming experience, which cannot be presumed from mashup composers. This calls for usability analysis that considers mashup tools peculiarities. Generally, usability analysis for mashup tools can be differentiated in techniques based upon inspection [10], user tests (e.g., using think-aloud [4]) and surveys [6]. For instance, [4] evaluated the usability of mashup tools using a think-aloud study and revealed a need for improvement of mashup tool. [11] evaluated the usability of datafloworiented visual programming tools based on a HCI framework, which measures the cognitive effort needed when working with such a tool. While [11] have not applied their framework for mashups, their evaluation results can be used for any dataflow-oriented programming tool, e.g., the mashup tool Yahoo! Pipes . The intention of our study was to evaluate the usability of mashups tools based upon a survey. In particular, we evaluated the usability of a process-oriented mashup tool. The evaluation of [11] is a good complement with our study and opens perspectives for future directions. Additionally, usability studies for mashups have been done for e.g., the mashup composition environment [1]. [16] evaluated the usability of mashup applications and [5] discussed metrics to measure usability of mashups. However, these approaches are only marginally related to our study since they address the usability of the final mashup and not of the mashup engineering process.
3. TYPES OF MASHUP TOOLS In [8] mashup tools are described as component software capable of accessing a large variety of resources, i.e.: source adapters, integrator components to combine data from different sources, and components for building the graphical interfaces (widgets) for the mashups to be developed. With such tools the user graphically builds the mashup by choosing, customizing, and connecting components [13]. For example, a data mashup tool would have adapters to connect to almost all known relational database source drivers such as e.g., Oracle, MySQL or MSSQL. Figure 2 depicts an example from Denodo [29].
fications of those, and also little assistance for users concerning the complexities of those tools. This makes it difficult and error-prone for end-users to obtain the right tool for their specific purposes [2]. For this reason some authors have already suggested a classification schemes for mashup tools. In [13] a classification scheme based on functionality is suggested. [9] discusses a classification in terms of presentation, data-orientation, and processorientation. The application domain of knowledge management is particularly taken into account in the mashup classification scheme of [8].
3.1
Process-Orientation
Process-oriented mashup tools allow the user to adapt a workflow design and simulate business procedures across heterogeneous system platforms [9]. The advantage of these types of tools is that they allow the user to select specific routine tasks from which to define an automated workflow. They are typically the easiest to use, depending on the process type. However, if the process includes the extraction of data from other data sources, the end-user must be aware of pertinent information such as database connection strings, usernames and passwords, which may also have safety-and-security implications. For instance, a user who generates a daily receipts report from a revenue collecting system could easily define a workflow that accesses the receipts database, loads this information into a spreadsheet, and e-mails the report to other users of the mashup. Figure 3 from Apatar [27] depicts such a scenario. Process-oriented mashup tools do not only help to automate routine processes; they can also be used to integrate assorted processes whereby they function as a consolidated resource interface for different processes in an organisation [7].
Figure 3: Workflow Design in Apatar [27]
3.2
Figure 2: Structure of Denodo’s Tool [29]
A large number of mashup tools is available on the internet, ranging from commercial to open-source tools. Typically, however, there are no helpful catalogues and classi-
Data-Orientation
Data mashup tools allow end-users to connect to large amounts of remotely hosted online data, and to extract, transform, integrate and present such information in a single view. The aggregation of data from heterogeneous sources can provide additional value to their users [8]. For this purpose, data mashup tools are equipped with components and adapters to connect to a number of data source formats, e.g.,: delimited files, websites, DBMS as well as Microsoft Office documents. These types of data mashup tools are typically applied in the domain of business intelligence. However, data mashups are not so easily adaptable by their end-users. This is because in most cases the enduser needs to understand the underlying data formats. For
155
information-technically un-trained users this will be quite difficult.
3.3
Knowledge Management
Knowledge management (KM) mashup tools can play a role in helping businesses units to share and combine information throughout the organization [26]. For this purpose, KM mashups tools support publication, search, collaboration and organisational learning. They allow developers to offer social-web features that allow members to communicate and collaborate. For instance, they enable portals to be able to integrate and connect to Twitter, Facebook and other social networks seamlessly. KM mashup tools are typically easy to use. In most cases they are built on basic “Web 2.0” platforms. Figure 4, from [3], depicts the internal structure of such a tool.
Figure 4: Structure of a KM Mashup Tool [3]
3.4
Interface-Orientation
User-interface-oriented mashup tools assist particularly in the development of interactive input menus and including also query result filters. Consider, for example, a real estate mashup showing houses for rent or for sale in a given area [30]. The mashup must return the findings as intuitively as possible (preferably graphically), such as depicted in Figure 5.
Figure 5: Mashup Interface Example from [30] To allow for comparision of the tools, a usability analysis is required.
4. USABILITY CRITERIA In [21] usability was defined in terms of the effort required to learn, prepare input, operate and interpret the output
156
of a software product. Similarly, the IEEE Standard for Software Quality Metrics [15] defines usability as the characteristic of software that bears on the effort needed for use (including preparation for use and evaluation of results) as well as on the individual assessment of such use by the users. From [21] and [15] we also know that usability cannot be evaluated in only one dimension. For instance, a tool cannot be considered of high usability if it is only easy to learn. Nielson suggests five criteria for evaluating usability [23], namely: learnability, efficiency, memorability, errors, and personal satisfaction. The IEEE Standard [15], on the other hand, suggests: stability, ease of learning, operability and communicativeness. McCall, however, suggests only three critera [21], namely: operability, trainability and communicativeness. In most of the literature seen [21] [23] [15], the most prevalent usability criteria are: learnability, operability and memorability. For the purpose of our paper, however, we consider even more, namely: learnability, operability, memorability and personal satisfaction as our criteria for assessing the quality of mashup tools. Moreover: to test whether or not there is a level of technical skill required for mashup tools users, we apply all those criteria amongst two groups of end-users, namely users with versus users without previous training in the IT domain. This would enable us to state, for example, that some tool T is user-friendly for some organisation’s IT personnel however not userfriendly the organisation’s commerce personnel.
5.
EMPIRICAL STUDY
Our goal was to assess if the level of usability mashup tools provide is sufficient for their intended end-users, and what level of skill is required for end-users in order to be able to develop their own mashups. Our assessment criteria, concerning usability, were outlined in the previous section.
5.1
Design
A survey was done with 20 participants of a mashup seminar, which was organized and conducted regionally, specifically for the purpose of our enquiry. About half of the participants had previous “IT skills” (typically IT system operators with IT-related Bachelor degrees), whereas the other half had not, (including marketing personnel, secretaries, accountants). All participants were members of the “business world” in so-called “white collar jobs”. The seminar involved the development of an exercise, which allowed the participants to operate a mashup tool to develop a small example mashup in a systematic manner. A questionnaire was then used to capture the participants’ experience on operating the mashup tool. In the seminar, an open-source mashup tool, Apatar [27], was used. Apatar is a light-weight and easily available tool. It can be used on various operating systems, (Windows, Apple, Linux) and can connect to a number of resources and many familiar DBMS. It seems thus fair to say that Apatar can be regarded as a good representative of widely-used mashup tools. The tool was downloaded together with relevant documentations and instructive videos, which were then made available to the participants of the seminar. The exercise itself comprised five basic steps: 1.) reading a few pages of tool documentation, 2.) viewing an explanatory video, 3.) installing Apatar, 4.) composing a mashup, 5.) starting the mashup composed in the previous step. Finally the participants were asked to respond to the questions of the survey questionnaire. The mashup, which the participants were asked to compose, involved the extrac-
5.2
Results
Seventeen out of twentyfive feedback forms were received, (return rate: 68%). Nine came from seminar participants without previous IT skills, eight came from participants with previous IT skills. Their answers, enumerated in Figure 6, can be summarised in words as follows. Learnability: about 94% of the responding participants claimed that they had no problems in learning how to operate the mashup tool. Operability: Looking at the questionnaire statement “it is easy to use this tool to do exactly what you want”, we see that about 82% of the responding participants claimed that they were able to use the tool in such a way as to accomplish the given exercise task. Memorability: In this category, the questionnaire statement “I have to refer to the help assistance most of the time when I use this tool” was looked at. Here about 94% of the responding participants disagreed with this statement. Thus the majority of the responding participants could easily memorise the menu operations while doing their exercise. Satisfaction: Lastly, to evaluate how satisfied the participants were with their mashup tool experience in general, we looked particularly a the responses to the questionnaire statement “I enjoy the time I spend using this enterprise mashup tool”. In this category we found the highest diversity: about 17% of the responding participants strongly agreed, whereas and about 29% had no opinion and about 6% expressed strong dissatisfaction. These diverse levels of enjoyment playing with an IT-tool might have been influenced by the different professions (IT-skilled or not) amongst the participants of the seminar.
Figure 6: Feedback after the Mashup Seminar
tion of data from a MySQL database, a data format transformation, and the transfer of those reformatted data into a Microsoft spreadsheet — a rather typical office application. The structured questionnaire had 25 “closed” questions to be answered. The questions covered awareness, usability, and assistance offered by the tool; see Figure 6. For easier evaluation, the questions had to be rated on a five point (Likert) scale (namely: strongly agree, agree, disagree, strongly disagree, and undecided). To motivate high user participation, the participants were promised to receive a copy of the research once the study had been completed.
A further evaluation, on whether the skills (IT-skilled or otherwise) of the responding participants influenced the surevey results, was done by means of the statistical Mann-Whitney U Test. This is a non-parametric test, which examines whether two samples of data could have come from the same population. The null hypothesis for this empirical study was that previous IT knowledge does not make a significant difference to the statement results or test variables, (all of which are Likert-scale statements). Indeed, the participants’ responses in a number of the survey statements did not show a major difference between those two groups. With rank 1 being assigned to answertype strongly agree, it can be seen from Figure 7 that ITskilled participants believed that the tool would indeed require previous IT training, and that there were too many steps required in order to get something to work. The IT-laymen, on the other hand, did not utter such concerns and simply accepted the given tool as it was. Presumbly the IT-skilled users, in their response, were able to “see” particular technical flaws or drawbacks which the IT-laymen could not “see” for this very reason, i.e.: due to their lack of technical expertise. Or maybe the IT-laymen did not dare do appear openly “lay-ish” in their replies?
6.
CONCLUSIONS AND OUTLOOK
The study carried out for [20] and described in this paper aimed at evaluating the practical usability of mashup composition tools. In particular, a widely-used representative of mashup tools was selected and evaluated. The
157
[6]
[7]
Figure 7: Differences between the Skill Groups
main questions were: Do those mashup tools require sufficiently little effort to learn, operate, memorise functions, and will the users be personally satisfied with their usage? According to the participants of Malinga’s Mashup Seminar [20], the answer to all those question is predominantly yes accross the board — at least as far as the one tool chosen for that seminar is concerned. Most interestingly, there was no noteworthy difference (w.r.t. the chosen tool) between IT-skilled participants and IT-laymen in that seminar. Under the presumption that all participants of the seminar answered the survey questions truly and truthfully we may conjecture that mashup tools can indeed “add value” also to those commercial organisations, which do not (or cannot) employ IT-skilled technicians or computer programmers in large numbers. On the basis of these encouraging findings, future work could be dedicated to the refinement of this study in slightly different settings, as well as to the improvement of already existing mashup tools according to those findings. Last but not least we can also foresee growing business opportunities for small-scale local IT consultants, if they are able to offer mashup training to their commercial clients, (similar to Malinga’s Mashup Seminar which was conducted only for academic purposes [20]).
[8]
[9]
[10] [11]
[12]
[13]
[14]
[15] [16]
Acknowledgments Thanks to the participants of Melusi Malinga’s Mashup Training and Evaluation Seminar, who have provided valuable feedback for the benefits of our empirical study. Many thanks also to the statistician who helped us with the Mann-Whitney test.
[17]
7. REFERENCES
[18]
[1] S. Aghaee, C. Pautasso: EnglishMash: Usability Design
[2]
[3]
[4]
[5]
for a Natural Mashup Composition Environment. Proceedings 12th International Conference on Current Trends in Web Engineering, pp. 109-120, 2012. B. Beemer, D. Gregg: Mashups: A Literature Review and Classification Framework. Future Internet 1/1, pp. 59-87, 2009. S. Bitzer, S. Ramroth, M. Schumann: Mashups as Architecture for Knowledge Management Systems. Proceedings 42nd Haw. International Conference on System Science, pp. 1-10, 2009. J. Cao, Y. Riche, S. Wiedenbeck, M. Burnett, V. Grigoreanu: End-User Mashup Programming: Through the Design Lens. Proceedings CHI’10: 28th International Conference on Human Factors in Comp. Sys., pp. 10091018, 2010. C. Cappiello, F. Daniel, A. Koschmider, M. Matera, M. Picozzi: A Quality Model for Mashups. Proceedings 11th
158
[19]
[20]
[21]
[22] [23] [24]
International Conference on Web Engineering, pp. 137-151, 2011. F. Daniel, M. Imran, F. Kling, S. Soi, F. Casati, M. Marchese: Developing Domain-specific Mashup Tools for End Users. Proceedings 21st International Conference Companion on WWW, pp. 491-492, 2012. P. de Vrieze, L. Xu, A. Bouguettaya, J. Yang, J. Chen: Process-oriented Enterprise Mashups. Proc. of the Workshop at the 4th International Conference on Grid and Pervasive Comp., pp. 64-71, 209. P. de Vrieze, L. Xu, A. Bouguettaya, J. Yang, J. Chen: Building Enterprise Mashups. Future Generation ˝ Comp. Syst. 27/5, 637U642, 2011. D. Fichter, J. Wisniewski: They Grow Up So Fast: Mashups in the Enterprise. Online 33/3, pp. 54-57, 2009. L. Grammel, M.-A. Storey: An End User Perspective on Mashup Makers. Technical Report, 2008. T.R.G. Green, M. Petre: Usability Analysis of Visual Programming Environments: a Cognitive Dimensions Framework. Journ. Vis. Lang. and Comp. 7, pp. 131-174, 1996. J. Gulliksen, B. G¨ oransson: Usability Design: Integrating User Centered System Design in the Software Development Process. Proceedings IFIP TC13 International Conference on Human-Computer Interaction, 2003. V. Hoyer, M. Fischer: Market Overview of Enterprise Mashup Tools, Proceedings 6th International Conference on Service-Oriented Computing, pp. 708-721, 2008. V. Hoyer, K. Stanoevska-Slabeva, S. Kramer, A. Giessmann: What are the Business Benefits of Enterprise Mashups? Proceedings 44th Haw. International Conference on System Science, 2011. IEEE: Standard for a Software Quality Metrics Methodology, 1993. E. Insfran, P. Cedillo, A. Fernandez, S. Abrahao, M. Matera: Evaluating the Usability of Mashups Applications. Proceedings 8th International Conference on the Quality of Information and Communication Technology, pp. 323-326, 2012. A. Koschmider, V. Torres, V. Pelechano: Elucidating the Mashup Hype: Definition, Challenges, Methodical Guide and Tools for Mashups. Proceedings 2nd Workshop on Mashups, Enterprise Mashups and Lightweight Composition on the Web, 2009. H. Lieberman, F. Patern` o, M. Klann, V. Wulf: End-User Development: An Emerging Paradigm. End User Development 9, pp. 1-8, 2006. Y. Liu, X. Liang, L. Xu, M. Staples, L. Zhu: Composing Enterprise Mashup Components and Services using Architecture Integration Patterns. Journal of System and Software 84/9, pp. 1436-1446, 2011. M. Malinga: Suggesting Quality Factors and Metrics for Enterprise Mashups. M.IT.-Proj. Report, Dept. of Comp. Sc., Univ. of Pretoria, 2012. J.A. McCall, P.K. Richards, G.F. Walters: Factors in Software Quality. Technical Report, General Electric, 1977. D.M. Nichols, M.B. Twidale: The Usability of Open Source Software. First Monday 8/1, 2005. J. Nielsen: Usability Engineering. Morgan Kaufmann, 1993. M. Picozzi, M. Rodolfi, C. Cappiello, M. Matera: Quality-based Recommendations for Mashup
[25]
[26]
[27] [28] [29] [30]
Composition. Proceedings 10th International Conf. on Current Trends in Web Engineering, pp. 360-371, 2010. W. Al Sarraj, O. De Troyer: Web mashup makers for casual users: a user experiment. Proceedings of the 12th International Conference on Information Integration and Web-based Applications & Services, ACM, 2010, pp. 239–246. D. Sharp: Knowledge Management Today: Challenges and Opportunities. Information Systems Management 20(2), pp. 32-37, 2003. World-wide web, http://www.apatar.com World-wide web, http://www.convertigo.com World-wide web, http://www.denodo.com World-wide web, http://www.housingmaps.com
159
