Texas A&M University (TAMU) Libraries serve a user population of over 44,000 students and approximately 5,000 faculty and staff members. During the past decade, the collections and services provided by the library, particularly computer based resources and services, have expanded rapidly. To gauge user perceptions of these services, the library has used periodic quantitative surveys. The survey data have provided insights into problem areas and have established a baseline against which to measure future performance of service programs. After analyzing the latest survey results, library administrators at Texas A&M determined that the dimensions of Reliability and Affect of service, particularly Assurance, warranted further study. Users had expressed lower levels of satisfaction with these areas, and administrators decided that these were the two dimensions that were hardest to understand in specific terms. If users expressed low confidence in the reliability of library service, for example, what were the specific services or components of service with which they are most dissatisfied? Were cataloging records faulty? Were staff unable to answer questions adequately? Were overdue notices sent out too often in error? What other elements were at play? While the surveys produced a snapshot of the landscape of users' assessments of service, they did not provide details regarding those perceptions. In order to understand more deeply users' views regarding reliability of services and their confidence in the services provided by staff, the researchers turned to focus groups to gain qualitative insights. Non-directive interviewing and focus groups emerged in the 1930s and 1940s as data gathering techniques that addressed the shortcomings of the close-ended questions typical in surveys and the narrow perspective of traditional one-on-one interviews (Krueger, 1994). Used successfully during the Second World War to examine the effectiveness of training materials and propaganda efforts, the technique became, for the next three decades, largely a tool of market researchers (Morgan, 1997). Since the late 1980s, focus group techniques have gained popularity among social scientists in a broad range of fields. In the 1990s, focus groups have been used prominently in the field of library and information science to investigate a wide variety of library issues, such as the effectiveness of library instruction
Probing user perceptions of service quality: using focus groups to enhance quantitative surveys Gwyneth H. Crowley and Charles L. Gilreath The authors Gwyneth H. Crowley is Head of Information Services, Levy Economics Institute of Bard College, Annandale-onHudson, New York, USA. E-mail:
[email protected] Charles L. Gilreath is Associate University Librarian for Advanced Studies, Texas A&M University, College Station, Texas, USA. E-mail:
[email protected] Keywords Service quality, Focus groups, Customer satisfaction Abstract Pencil-and-paper surveys can provide useful information about user perceptions of service quality in libraries, but qualitative methods must be used to gain more precise understanding of problem areas. Texas A&M University Libraries conducted focus group studies in spring 2001 with graduate and undergraduate studies in order to gather specific information related to their satisfaction with and confidence in the assistance provided at library service points. The sessions revealed that users were generally pleased with the assistance provided them by professional staff at reference desks and that they found librarians to be usually patient and helpful. Elements of dissatisfaction identified by the sessions included a sense of inability or unwillingness to assist at some service points and a perceived lack of knowledge of the library, particularly by student workers and staff at locations away from reference areas. The data are being used to improve library directional tools and to improve staff training for public services staff. Electronic access The research register for this journal is available at http://www.emeraldinsight.com/researchregisters The current issue and full text archive of this journal is available at http://www.emeraldinsight.com/1467-8047.htm Performance Measurement and Metrics Volume 3 . Number 2 . 2002 . pp. 78±84 # MCB UP Limited . ISSN 1467-8047 DOI 10.1108/14678040210440973
78
Probing user perceptions of service quality
Performance Measurement and Metrics Volume 3 . Number 2 . 2002 . 78±84
Gwyneth H. Crowley and Charles L. Gilreath
from the quality of cataloging records and the processes for handling fines/overdue notices to issues surrounding the shelving of materials and the training provided to staff and students at key service points. In the end, library managers realized that the expeditious way of determining what problems users had with library services was simply to ask them directly, and focus group methods were selected as the means of providing the broadest input from user groups. There are many types of group data gathering techniques available for use in social sciences research such as nominal group, Delphi technique, and brainstorming. Nominal group and Delphi studies are conducted in group settings, but the goal of these techniques is consensus building and/or prioritization of options based on individual assessments following some group interaction. Brainstorming is closer in its results to focus groups. The emphasis with this technique is primarily the generation of ideas and concepts, and the results typically provide the rich context characteristic of focus groups. Only focus group methods provide for a moderator who is able to play an active role in guiding the conversation, in providing clarification or additional information, and in probing with additional questions any specific issues that come up during the session. Focus group sessions are taped, and the basis for analysis is the set of transcripts of the taped sessions. This methodology provides both for broad input from users and for a contextual framework in which to judge their comments. Researchers drafted separate sets of questions addressing each of the two domains being studied. These questions, and a plan for advertising the sessions and selecting participants, were submitted to the university's institutional review board for human subjects research. Once the board approved the study, the library ran ads in the campus newspaper, stuffed flyers into graduate student mailboxes, and the research team asked library subject liaisons to contact faculty in their departments to help inform students about the upcoming sessions. In response to these efforts, eight focus groups were conducted: five with undergraduate students and three with graduate students. Individual focus groups ranged in size from four to nine students. Participants signed waivers allowing their comments to be recorded for transcription
programs (Carter, 2002), user perceptions of library electronic research tools (Morgan and Reade, 2000; Barnes, 1993), assessing the quality of reference services (Norlin, 2000; Massey-Burzio, 1998), and planning for future library services (Berger and Hines, 1994; Meltzer, and Maughan, 1995). Given the broad appeal of this methodology throughout the social sciences, the number of publications available to guide researchers is extensive. Morgan and Krueger, in addition to their two separate works cited earlier, have collaborated on a six-volume guide, The Focus Group Kit, which provides assistance in all aspects of the process, ranging from question development and session planning to moderation/facilitation techniques and data analysis (Morgan and Krueger, 1998).
Method Originally, focus groups were designed and used almost exclusively to gauge user reactions to specific stimuli such as films, radio or television programs, or written training material. Participants watched a film or program, listened to a radio broadcast, or reviewed a published manual and responded to questions about what they had just experienced. Now, focus groups are used to gather user reactions to and perceptions of a broad range of issues, and have proved useful in gathering user input for planning future services, for assessing current services, and for understanding user behaviors in relation to particular institutions or programs. Focus groups are frequently used, as in this study, as a follow-up tool to better understand results gathered through quantitative means such as formalized surveys. The administrations of the SERVQUAL and LibQUAL+TM instruments by Texas A&M University Libraries revealed a persistently lower than desired set of scores in the domains of Reliability and Assurance for the libraries. While the specific scores varied among the three classes of user surveyed ± undergraduate students, graduate students, and faculty ± library administrators were concerned about the overall scores. To analyze the aspects of service that contributed to the lowered perceptions of users in these two domains, administrators and department heads at TAMU University Libraries hypothesized a number of causes, ranging 79
Probing user perceptions of service quality
Performance Measurement and Metrics Volume 3 . Number 2 . 2002 . 78±84
Gwyneth H. Crowley and Charles L. Gilreath
here somewhere'' and then asking for help early in their search process. A basic confidence in the library owning needed resources is an excellent building block for patron/staff interactions. In general, focus group members responded favorably to the help they received at reference desks. Other adjectives used to describe reference staff were: ``extremely helpful'', ``very professional'', ``very useful'', and ``very personable''. The government documents reference area/desk was also identified as ``always been real good''. Typical comments include: ``they usually steer me in the right direction'' and ``I have had good experiences . . . they seem to be very helpful''. More specifically, participants remarked, ``The reference librarian pointed me in the right direction'', ``[He] told me how to search and was so patient'', ``They'll take the time'', and ``[They] went the extra mile for me''. Another participant remarked, ``I have never had problems with the research librarians''. In addition, specific librarians were named and thanked by individual focus group members for their help. When participants responded to the question of ``When you have a successful exchange with a library staff member, what particular parts of the interaction gave you confidence in the answer you received?'', their responses were comparable to earlier positive remarks about reference librarians. Participants mentioned the traits of friendliness and patience were important most often. Other attributes desired from service staff in general and received included: ``Friendly and patient when I approach them'', ``good personality'', ``understanding'', ``knowledgeable'', and ``straightforward and to the point''. The staff's confidence in their own ability proved to be another factor that inspired confidence in participants. Numerous comments from participants illustrated this point: ``for them to have a certain confidence about that gives me confidence in them'', ``They sound[ed] like they know what they're doing'', ``What gave me confidence was how quick they came up with an answer and how sure they seemed about it . . . it's more [about] how they give the answer'', ``They, point blank, told me if they'd know or wouldn't so it gave me confidence'', ``They were either able to tell me if they had it [the materials] or if they didn't'', and ``As long as they come across with confidence and good information''. Focus group participants
and further study. A moderator from the research team led the groups, using the prepared questions (see the Appendix) as a guide. All sessions were conducted from late March through mid-April 2001. The session tapes were sent out for transcription. Researchers performed content analysis of the transcripts, analyzing the texts for individual sentences and phrases that captured key points, issues, and concerns of the focus group participants. These key points were transferred onto index cards along with notations regarding either the question eliciting the comment or the context in which the comment was made. The cards were sorted into groups representing issues, problems, desires and perceptions common to participants as a whole. From this sorting process, researchers identified issues or themes related to the service dimensions of reliability and confidence and a larger number of specific problems or desired improvements.
Findings For the purpose of this paper, only the themes that relate to Assurance and confidence as measured by the instruments will be discussed. The four survey questions that identified Assurance were: ``Employees who are consistently courteous; employees who instill confidence in customers; employees who have the knowledge to answer customers' questions; and assuring customers of the accuracy and confidentiality of their transactions''. The focus group discussions, related to the concept of assurance and confidence, identified one positive theme of helpfulness and two negative themes: unwillingness to assist, and the lack of knowledge and/or training. Comments from the focus group participants will be utilized to illustrate these themes. Helpfulness Overall, focus group members indicated that library employees were nice, and that the amount of helpfulness varied with the service point. One participant summed it up by stating: [They're] ``not always helpful but [are] always friendly to me''. Some library workers were perceived as unwilling to help for various reasons as discussed later in this paper. Also, some users reported making their library work easier by adopting the attitude: ``I know it's in 80
Probing user perceptions of service quality
Performance Measurement and Metrics Volume 3 . Number 2 . 2002 . 78±84
Gwyneth H. Crowley and Charles L. Gilreath
recognized the essence of quality, but they frequently could not define it in specific terms. A participant remarked, ``It's nothing in her job description. It's the way she reacts, . . . she's positive; she's willing to go out of her way. It makes me more confident to go back to her for help every time'', and ``They seem to genuinely want to help''. Compassion for learners was also a positive desired attribute. One focus group member said, ``[Once I said to the reference librarian], I have a really stupid question, and she said, `there's really no such thing as a stupid question'. It made me feel better''. For others, they judge to whom to go for help by the physical characteristics: ``I have a tendency to first go by age''. In summary, the personality traits of patience and friendliness were most important to focus group participants for successful interactions at service points. Other traits such as self-confidence were secondary. Later in this paper, the concept of knowledge and training will be explored as an important component of a successful transaction. Unwillingness to assist While some participants were happy with the services that the library provides, other focus group member remarks cumulatively pointed out that both service point library staff and student workers were seen as unwilling to help. Staff who were inattentive to patrons, not willing to move to lead the way, and replying in a sour or grudging manner were major barriers to users obtaining the help they desired. One participant reported: ``There were two girls at the desk chatting. They ignored me and finished chatting before they would talk to me. This was at the circulation desk. There were also four people behind the desk that were doing nothing''. Another stated: ``It annoys me that the student workers have their friends come and talk to them, and then you just stand there, and stand there a little longer''. A focus group member offered the advice that: ``Being eager and polite would be a part of the interaction that makes it successful, [and] give it some validity in a way, rather than just like no big deal . . . like a student worker who doesn't try hard sometimes and just acts like it's not a big thing''. Another obstruction to helpfulness was universal at service points. Not physically standing up to help people was reported as one of the biggest barriers. Typical comments 81
included: ``[it] is not the research librarians, but practically any other area, where they don't want to get up, and help you'', ``They don't want to leave their desk'', ``I almost rarely ask the librarians. They would point it's over there, so I just try [now] to figure it [out] myself. They just don't want to help'', and ``They don't want to get up and pull the books''. Clearly, some people are not being directed to materials or being shown how to use the library in a satisfactory way, and staff, by staying ``anchored to their desk'', are being perceived as not wanting to help, thus undermining confidence in the library's information services. Specific comments regarding one of the larger service points are illustrative of user perceptions. As one user reported, ``they [staff] just sit back there, and when I go ask them, they [say] just like over there'', and ``[they said] it's over there, and I tried looking and couldn't find it again. When I asked them for help again, they said, you're right, it's not here and that was it'', ``They're abrupt'' and ``They don't want to go further with it''. This type of assistance does not build confidence in either using the library or in teaching how to use it. Offensive verbal and non-verbal behavior by staff was another major barrier reported by the focus group members. Numerous comments from participants support this premise: ``They can get so huffy and upset, as if by asking for assistance I'm absolutely putting them out'', ``Some staff I asked were nasty and not very happy'', and ``It was just the callousness of her nature . . . have a little bit of compassion''. Such a lack of civility, even if rare among staff, is a major factor in creating negative perceptions of service by users. Patrons' memories tend to be long as well; they will avoid staff members with whom they have had previous disappointing contacts. A focus group participant remarked, ``If I get the wrong answer, I'll write a note in my head, I'm never trusting that person again'', and another group member commented, ``. . . there's always been a student working there and he's so unhelpful, I just usually walk back downstairs [for help]''. All of these comments, as compared to the positive remarks on helpfulness, suggest a severe unevenness of quality service. A quote from a participant sums this up: ``I got lucky that time''. Clearly there is a problem with quality
Probing user perceptions of service quality
Performance Measurement and Metrics Volume 3 . Number 2 . 2002 . 78±84
Gwyneth H. Crowley and Charles L. Gilreath
focus group members to associate specific service areas with their complaints, they were silent. Also, it seemed as though opinions about one area of the library carried over to others. One participant said, ``When you have people blow you off, you lose a lot of respect for the whole system''. The time of the day that assistance was rendered was also a factor. Another participant stated, ``It just depends on what time you come'', and ``Early in the morning, the crowd that's there, even the Loss Prevention, they're all very friendly. I've gone there in the late afternoon and they have . . . attitude''. Communication was a barrier as well, as illustrated in the following three responses: ``I don't think it's so much the attitude as the problem we're having trying to explain what we want'', ``I couldn't understand her thick accent'', and ``Sometimes I feel they need to speak more slowly or clearly''. On the whole, focus group participants were discouraged by people not willing to stand up to help, or by staff just pointing or being verbally harsh. Other factors included time of the day, privacy concerns and communication skills by both the staff member and user.
of service if the user perceives that receiving good service is a matter of fortune. As a group, student workers were singled out for disappointing transactions. Several respondents reported instances in which they wanted help in the stacks, and when they asked book shelvers for assistance they received an array of responses including correct and incorrect information, and rude answers. ``There was a girl shelving books, so I asked her where this section was that I needed and she couldn't tell [me]'', a focus group member said. Student workers were seen as ``begrudgingly helpful, and . . . they shuffle you off'' and ``sometimes the student workers and the people at the security desk . . . are not real personable''. At the interlibrary loan desk, staff members were described as ``They will do all they can'', ``extremely competent'', and ``I have had very positive interactions with interlibrary loan services in general''. However, unfilled requests returned with paperwork stapled together was confusing to patrons, and complaints of untimely or no notification of requested interlibrary loan materials articles were reported. While these comments pointed to a problem with staff training regarding basic procedures and location information, the perception by users appeared to be that staff were simply not helpful. Responses about the circulation desks were mixed. On an affirming note, participants said, ``They're very helpful at recalling books'', and ``[we] like getting reminders in the mail''. These standard procedures seem to be working, and the following comment shows the department's flexibility: ``Kudos to [the circulation manager] who helped me very, very much when I was out of the country and had a colossal library fine''. Privacy is a problem for the patrons at the circulation desks. One group member stated that it's ``so rude and embarrassing when student workers announce [to everybody] how many times I've lost my ID''. Waiting in line to check out materials was a complaint as well. Two quotes illustrate this point well: ``It's a big old line at the circulation desk'', and ``One time, there were ten to 12 people standing in line and there was only one person to help. I see people in the back doing other stuff but you know . . .''. Other critical comments from participants tended to be more general in nature. Frequently when the moderator prompted
Lack of knowledge/training Teaching others to use a library or to do research requires having the expert knowledge and experience. Participants often sought help from people who could neither answer their questions nor refer them to someone who could. Turnover was cited as a one of the reasons. Student workers, though, bore the brunt of this assessment. They were frequently viewed as lacking in experience and knowledge. One respondent commented, ``If I have a question, I've gone to some departments and the student workers may not know what to do. Instead of saying, `I don't know', they make up something and then I get even more confused''. Other workers had the same problem as well. A participant remarked in general: ``Some just say, `I don't know.' `Do you know who could tell me?', I ask, and sometimes, they don't know that either''. A participant summed it up nicely by saying: ``If they know what they're doing, then I know what I am doing''. Sometimes patrons ask for help from people who are too narrowly trained to be of assistance. Student workers are trained to 82
Probing user perceptions of service quality
Performance Measurement and Metrics Volume 3 . Number 2 . 2002 . 78±84
Gwyneth H. Crowley and Charles L. Gilreath
areas in the library. Even though this outreach service has been in operation for a short time, statistics indicate that roving staff have been fielding over 100 questions per month. In addition, discussions are underway to revamp training in all public service units to give all staff who encounter the public in the course of their work a deeper knowledge of the library's organizational structure, major facilities likely to be needed by users, and a more thorough understanding of how to direct users to specific call number ranges. Increased emphasis on interpersonal communications will also be part of the revamped training program.
work in a specific library department. A focus group member said, ``I asked for help in searching on the first floor of the annex. [I found out] they're not trained [in that]''. Another remark was: ``When you go and ask people at other spots in the library, they say, `We can't help you; go look up the call number', and I don't know how''. Patrons expect a broad range of help and do not understand the detailed structure and roles of library staff, and resort to guessing where they should go. From the comments, it appeared that service providers were often unfamiliar with the library organization and thus could not make effective referrals, resulting in frustration for the user. As one participant put it, ``If you are in a bit of a hurry, the worst thing is to get redirected to someone else who says, who told you to come here? One day they had me running all over the library from person to person. It was just horrible." A suggestion for better assistance was: ``If [the library] had people on a certain area of each floor, you could just flag them down''. Another contributor, however, voiced, ``I don't like flagging people down because I feel like I'm an idiot so I usually just don't talk to anybody''. Differing views notwithstanding, a strong sentiment coming from the focus groups was for users to be able to ask any library staff member in a public service environment a basic question and receive at least an accurate referral to the service point where the question could be answered. Equally strong appeared to be the expectation that all public services staff be able to answer simple directional questions regarding the location of particular library facilities or call numbers.
Conclusion Surveys are closed-ended questionnaires requiring responders to answer in concrete terms. Even when responses to such surveys allow for input on multi-point scales, the nuances of individual opinions on particular issues are only roughly captured in such instruments. To be sure, surveys such as SERVQUAL and LibQUAL+TM are valuable for identifying key areas of strength or weakness in service programs. The standardization of response inherent in such surveys opens the possibilities for using the results to benchmark local services against other similar institutions. What survey instruments cannot do is to provide insight into the many facets of the service experience that color users' perceptions. Qualitative tools such as the focus groups used in this study are necessary to probe those aspects of service. Exploring the SERVQUAL and LibQUAL+TM results through focus groups allows participants to add their opinions and thoughts ± the gray matter, as it were ± permitting the dynamics of the situation to be explored more fully. In short, survey instruments assist managers best by identifying what general areas of service delivery are potential problem areas to be addressed, while qualitative tools such as focus groups help in identifying specific problems and suggesting a course of action to address the problems. Results at Texas A&M University indicated that there was a meaningful gap between user expectations and perceptions of service quality with regard to Assurance. Researchers discovered that participants generally perceived reference librarians and staff to be friendly and patient, but not always helpful, and were wary of student workers.
Outcomes Based on the information gathered from user surveys and focus groups, the library has undertaken several measures to improve service quality. A new building signage package has been purchased and installed as a means of providing users better guidance to major facilities and services. Plans are underway to deploy touch screen information kiosks throughout the library in order to help users with basic directional information such as service point and call number locations. Reference staff in the main library now rove on the main stack floors, concentrating their attention on the numerous computer bank 83
Probing user perceptions of service quality
Performance Measurement and Metrics Volume 3 . Number 2 . 2002 . 78±84
Gwyneth H. Crowley and Charles L. Gilreath
Unwillingness by staff and students to help was also a major factor. All of these findings illustrate that the library service points were providing an inconsistent quality of service and steps were taken by the library administration to better manage the barriers.
References Barnes, S.J. (1993), ``An electronic library grows'', Computers in Libraries, Vol. 13, September, pp. 12-15. Berger, K.W. and Hines, R.W. (1994), ``What does the user really want? The library user survey project at Duke University'', Journal of Academic Librarianship, Vol. 20, November, pp. 306-9. Carter, E.W. (2002), ``Doing the best you can with what you have: lessons learned from outcomes assessments'', Journal of Academic Librarianship, Vol. 28, January/March, pp. 36-41. Krueger, R.A. (1994), Focus Groups, 2nd ed., Sage, Thousand Oaks, CA. Massey-Burzio, V. (1998), ``From the other side of the reference desk: a focus group study'', Journal of Academic Librarianship, Vol. 24, May, pp. 208-15. Meltzer, E. and Maughan, P.D. (1995), ``Undergraduate in focus: can student input lead to new directions in planning undergraduate library services?'', Library Trends, Vol. 44, Fall, pp. 400-22. Morgan, D.L. (1977), Focus Groups as Qualitative Research, 2nd ed., Sage, Thousand Oaks, CA. Morgan, D.L. and Krueger, R.A. (1998), Focus Group Kit, Vols 1-6, Sage, Thousand Oaks, CA. Morgan, K. and Reade, T. (2000), ``Pioneering portals: MyLibrary@NCState'', Information Technology & Libraries, Vol. 19, December, pp. 191-8. Norlin, E. (2000), ``Reference evaluation: a three-step approach ± surveys, unobtrusive observations, and focus groups'', College and Research Libraries, Vol. 61, November, pp. 546-53.
Appendix. Assurance study focus group I would like to ask you to begin thinking about ways in which you use the library. Specifically, I want you to think in terms of those times when you have interacted with library staff members and how confident you were in the answers or assistance you received: (1) When you need to do some research (i.e. write a paper), what is the first thing you do? (2) When you think about doing research using library resources, what is the first thing you do? (3) How do you go about getting the information you need? 84
Let's narrow our focus a bit: (1) Think about when you ask the library staff for assistance (in finding information, in locating material, or in getting a library service). What were you seeking that prompted you to ask for help from a member of the library? (2) When you come to the library, how frequently do you interact with a member of the library staff (at circulation counters, at a reference desk, at some other place in the library)? (3) What kind of information do you frequently need help in obtaining ± e.g. directions in the building, help getting started with research questions, help locating particular items, library policies, etc.? (4) When you go to a library staff member for assistance, what sort of response do you expect? (At a reference service point? At another library service point such as circulation or interlibrary loan?) (5) How would you characterize these interactions? (6) When you have a successful exchange with a library staff member, what particular parts of that interaction give you confidence in the answer you received? (7) When a library staff member is not able to answer the question or solve the problem you have posed or refers you to another staff member or library department, how do you feel? (8) Have you ever received an answer from a staff member that did not meet your needs? If so, what did the staff member do or say? (9) When you have received an inadequate answer from a staff member, what do you think are the reasons for the inadequate answer? (10) When you receive an inadequate answer, what do you do? Lessons to be learned: (1) Think about all that we have talked about today. What do you think is the most important thing for the libraries to do to increase your confidence in the personal assistance provided at service desks? (2) What one change would you recommend to improve the accuracy and adequacy of staff responses in the library?
