Conducting an Online Survey in an Academic Setting

UNIVERSITY OF MAURITIUS RESEARCH JOURNAL – Volume 17 – 2011 University of Mauritius, Réduit, Mauritius Research Week 2009/2010

Conducting an Online Survey in an Academic Setting. Our Experience and Findings Moloo R K* Faculty of Engineering, University of Mauritius Reduit Email: [email protected] Pudaruth S Faculty of Engineering, University of Mauritius Reduit Email: [email protected] Paper Accepted on 22 June 2011 Abstract With the ever-increasing access to Internet in Mauritius, online surveys might soon replace traditional survey methods. If properly conducted, online surveys provide an easy, cheap, reliable and quicker way of obtaining and analysing valuable data from a wide and diverse population. In this paper, we present the effectiveness of an online survey carried out to students of the University of Mauritius to evaluate the Introduction to Information Technology (IT) module. We share our experience in designing, implementing the survey and also in collecting the required information. We show how effective, time and cost saving online surveys can be in collecting and analysing information in an academic institution. The survey further analyses gender difference, mean time taken, time of day and the effectiveness of open-ended and close-ended questions which have been sparsely considered by earlier researchers. We also present the varied advantages online surveys can bring and our experience of the techniques we used to encourage students to answer the survey forms. The survey was carried out online and targeted 1660 students. The response rate was 29.6%, covering 492 students.

Keywords: Online survey, response behaviour, ICT *For correspondences and reprints

444

Conducting an Online Survey in an Academic Setting. Our Experience and Findings

1. INTRODUCTION

The success of online surveys relies on the IT literacy of the sampled population. The National ICT Strategic (NICTSP) 2007-2011 plan [1] projects in making Mauritius a digital island. Latest statistics shows that Mauritius has a 29.6% internet penetration ratio with 380,000 internet users [2]. This figure however, includes business users as well. An estimate of the users’ age group can be estimated from Facebook [6] users demographic profile in Mauritius. We firmly believe that its demographic profile reflects the age range of internet users in our country. Around 55% of users fall in the age range of 15-24, 27% in 25-34 age group, 8% in the age bracket of 35-44, and 5% is aged above 45 years old. Therefore, it can be reasonably inferred that the majority of web users in Mauritius are in the 15-24 age group.

This can be explained since Information and Communication Technology (ICT) is rather new in Mauritius, a decade old, and a rapid expansion has been noted in this sector with the advent of the 1998 ICTA act [3], which consists of empowering people and businesses with IT skills. Pushing further our analysis, it can be deduced that online survey audience in Mauritius will be mainly within this age group.

Based on the facts and figures mentioned and with an ever increasing IT literacy ratio, we firmly believe that online surveys have a bright future over traditional methods in our country. To our knowledge, practically no methodological study on online surveys have been carried out in Mauritius to determine the impact, validity and response rates of such kind of medium. Online polls or one-question survey are often to be seen on some Mauritian websites especially media website, but these cannot be considered as full fledge surveys. All the majors firms specialized in doing surveys in Mauritius, including the Mauritius Central Statistics Office (an organization responsible for the official statistics pertaining to our country) acknowledge that response rate is very low via this method. Traditional ways of carrying out surveys in Mauritius are mainly by questionnaires and phones. 445

Moloo R K & Pudaruth S

Previously, conducting online survey necessitated a web programmer to setup a questionnaire online, to extract and report the data in a suitable format. However, nowadays lots of free Survey websites are available like, surveymonkey [7], esurveyspro [8], surveygizmo [9] to mention a few. In addition, within the academic environment e-learning platforms like Moodle [10] and ATutor [5] do provide plug-ins for the conduct of these surveys. The advantage of using them is that they also provide tools for analyzing and exporting the data in suitable format. Some of these websites provide templates, customisation of questionnaires, security measures like checking IP, stopping users from answering more than once, set time frame to complete questionnaires, specify start and end date etc. Their ease of use, flexibility and accessibility makes them unrivaled methods of conducting surveys at practically no cost.

Following a decline in performance in the Introduction to Information Technology (IT) module and requests from different faculties for change, the Computer Science and Engineering (CSE) Department at the University of Mauritius decided to revamp this module. This course, unlike traditional lectures, is dispensed on a tutorial based mode, whereby students are provided with online materials for further readings, to practice questions, tutorials and lab. Students have a one-hour tutorial every two weeks with no practical labs. The module is mainly self-learning and students are then asked to attend a one-hour tutorial based class every two weeks to discuss problems encountered in understanding the subject.

A committee was set up in this effect to look into the matter and among other things, it was decided to hold on a survey for this module to gather requirements of the different stakeholders needs and how they evaluate the current situation. This was an excellent opportunity for us to test the efficacy of online surveys at UoM. A questionnaire was devised to collect the required information. A free online elearning platform powered by ATutor [5] was used to host the survey online. The questionnaire consisted of 29 questions, out of which 22 were Multiple Choice Questions (MCQs), 3 were Multiple Choice Multiple Answers questions (MCMAQs) and 4 were open ended questions. It took one day to put and test the 446


survey online. Instructions were given on how to access and complete the survey. Tutors of this module were asked to motivate their students to fill in the survey.

This survey targeted the whole student population of 1660 doing the Intro to IT module, from the five faculties of UoM namely the engineering, law and management, science, humanities and agriculture to determine their needs. Students were required to register and take up the survey. Only one attempt was enforced to avoid bias. It is to be noted that no incentives like gifts of money were given to students upon completion of the survey. From 23rd to the 28th of November 2009 the survey was conducted. The Students were given 5 days to answer the survey form at their own free time, whether at home or on campus. An extension of 1 day was also allowed after the deadline. On day 3, a reminder was sent to them by email. A response rate of 29.6% was achieved with 492 responses. The data was collected, processed and analysed using the statistic tools of the ATutor [5] platform.

The purpose of this paper, however, is not to comment on the survey’s findings but rather share our experience on the implementation, design and conduct of online surveys in an academic environment. We comment on the surveys response rate overall and per faculty. The benefits and short-comings of an online survey are also outlined. We also comment on the answering behaviour of students.

The rest of this paper is organised as follows. Section 2 gives an overview of related work in conducting surveys. Section 3 discusses the methodology we used in devising, collecting and analyzing information. Section 4 exposes our analysis and interpretation of the results of the survey. In section 5 we share our experience in carrying out the survey. And finally in section 6, we present our conclusion.

2. LITERATURE REVIEW

The first world-wide web user survey was posted in January of 1994 [13]. This survey was done to get an idea of who uses the WWW and for what purposes. This 447


survey with 4777 respondents in one month was very successful and paved the way for innumerable number of surveys to be conducted online since then. In 1999, Bartel [14] surveyed internet users in the United States using emails. He demonstrated that email surveys can be a viable method to collect responses from users who are dispersed over a wide geographical area. He also listed several other advantages like huge penetration, cost benefits, ease of responding and reasonably high response rates when compared with postal surveys. The problems with emails are that they can be classified as spam and they expire quite rapidly, i.e., people are creating new email addresses all the time and disregarding older ones.

Kim Sheehan [15] examined five factors which could affect the response rates from email surveys since 1986. The factors are: the year in which the study was done, the total number of questions in the survey, the effect of pre-notification emails, the number of follow-up contacts and finally the survey topic itself. Quite surprisingly, the factor which affected the response rate by the largest amount was the year in which the study was undertaken. This was explained by the fact that when the technology was new, people were highly motivated to use it. However, nowadays, the technology is often misused by spammers which has thus created a mistrust among surfers to reply to emails. Response rates for many of the surveys oscillated between 10% and 70%.

In his paper [16], “Classifying Response Behaviours in Web-based surveys”, Bosnjak analysed the respondent answering process instead of the response rate. Traditional surveys have categorised responses using only three classes. Unit nonresponse is where the person does not reply to the survey at all, item non-response is where the person replies to some of the questions in the survey and complete response in which the person replied to all questions asked. However, Bosnjak identified classes of responses. In particular, he identified people who look at all the survey questions but do not answer any of them and also people who leave the survey half-way. The latter are termed as drop-outs. These classes do not exist in postal surveys. 448


Manfreda et al [17] were amongst the first researchers to evaluate whether the design of a web survey questionnaire has any impact on the measurement error and partial and unit non-response. They argue that web questionnaires are too often prepared by non-professionals who have no much idea about questionnaire design. Measurement errors result from inaccurate responses that stem from poor question wording or questionnaire design. In [17], the authors identified three main design issues which affect the visual design of web questionnaires. These are the graphic layout, presentation on questions and the number of questions on one page. Partial non-response was higher when logos were used on the web page. Partial nonresponse was not significantly affected by having each question on a different page if the survey is a short one. However, on average respondents took 30% more time to complete the survey. It was suggested also suggested that additional research was required to know the ideal number of questions to put on one page. Item nonresponse was smaller when logos were used and when questions were places on several web pages instead of one single page.

In [18], the authors undertook a simple but interesting factor which they thought could possibly impact on the response rate. An experiment was set-up to test the impact of four different salutations on response rate. Four samples of 800 alumni from Stanford University were selected. The salutations used were “Dear Stanford Alum”, “Dear James”, “James”, and “Dear Mr James”. Although on average, the personalised salutations drew more responses than the generic salutations, the differences were not statistically significant. Thus, the authors suggested that if personalisation of messages would be too costly in terms of time, effort and money, overall response rate might not suffer much with a generic salutation. They went on to give some further suggestions like format personalised salutation(Dear Mr James) may increase the response rate amongst young and old people and that the formal personalised salutation may do better with men than with women who might respond more favourably with a personalised salutation (Dear Jane). Nevertheless, they recommended that these experiments must be performed again on different types of populations as the small differences obtained in their study might be due to luck only. 449


The researchers in [19] demonstrated how invalid responses might be caught from web-survey responses. Since financial incentives are often given to people to respond to online surveys, there is the possibility that some people will submit multiple responses for the same survey. It is also very likely that some respondents will sign in only to finish the survey questionnaire as quickly as possible thereby decreasing the confidence of the survey results. However, in this study, the authors successfully implemented multiple validation protocols to remove the maximum possible number of invalid responses. While some automated tools were used, they believe that manual review is still essential at the end.

In his 2005 paper [20], Wright discusses in sufficient depth both the advantages and disadvantages of conducting online survey research. In particular, he discusses issues like accessibility to geographically dispersed populations, time and cost. The main disadvantage that he described was the difficulty to get an un-biased sample. He went on to compare and contrast different types of online questionnaire authoring software packages and web survey services. Their strengths and limitations are described to a reasonable extent. In a study [21] conducted by Apathia Inc through their website supersurvey.com, they analysed the meta-data from 199 surveys. All the surveys were paid ones. 50% of surveys received at least a 26% response rate. Generally, the higher the invitation list, the lower the response rate. About 50% of survey responses arrive within one day and nearly all arrive within two weeks. Survey invitations sent in the morning (before 9 a.m.) achieve higher response rates and quicker response times. There is a decrease in response rate but an increase in response time during the course of a working day. According to the ESOMAR Online Research 2009, 20% of surveys were carried online in 10 countries while the average for telephone research was at 18% for these 10 countries [22]. [23] analysed the factors which could influence response rates for online surveys. The factors considered were the saliency of the questions (or the survey itself), the incentives provides, whether it was done in an academic setting or to a broader audience, length of the online survey questionnaire, etc.

450


The effects of the timing of follow-ups, the types of incentives provided and the presentation of the questionnaire on the response rate and response quality were further discussed in [24]. However, these researchers felt that their results were not conclusive and their results should be applied to other types of online surveys with caution.

In [25], the suitability of online surveys against mail surveys on a national level was determined. The researchers concentrated more on response quality rather than on response rate or completion time. They concluded that both types of surveys have similar response quality. However, they suggested that their findings might not be relevant for countries with a low Internet penetration.

In their widely acclaimed article [26], the authors demonstrated the importance of evidence gathered through online surveys in U.S court cases. In particular, they outlined a few criticisms of internet surveys. They explained when the results from online surveys can be used as evidence and when they are not considered as valid.

3. METHODOLOGY

An online survey methodology was chosen over other surveys for various reasons and they were as follows: 1. As a research work, we wanted to test the effectiveness of doing an online survey, which has never been carried out at UoM. 2. Free online survey tools were available which enables the setting up questionnaires, collect and analyse information at practically no cost. 3. For economic reasons in terms of time and money, online surveys prove to be a very cheap medium. We did not have so much time to spend for face to face interviews and printing of questionnaires which would have been costly. 4. No sampling was needed since with online surveys, all the student population could be targeted.

451


5. Students doing the Introduction to IT module, which was an online, self learning module, were already familiar with the web and the IT environment. Hence, there were no needs for additional training. 6. Most of the tutors already had the mailing list of their students, and therefore it was easier to reach these students via emails. 7. Studies have shown that online surveys are a more convenient means to users, since it enables them to response at their own time when they are free.

3.1 Questionnaire Design

The questions to be set were prepared by a committee. Questions were devised according to the information required by the committee on the current Introduction to IT syllabus. Strategies to reduce non-response [11] [12] were incorporated while preparing the questionnaire. Care was taken not to make it too long since respondents might lose interest and it was estimated that a user will take on average 15 minutes to fill it. The questionnaire consisted of 29 questions, out of which 22 were Multiple Choice Questions (MCQs), 3 were Multiple Choice Multiple Answers Questions (MCMAQs) and 4 were open ended questions. Questions ranged from demographic questions (e.g. gender, faculties) to questions related to the evaluation of the current syllabus. The questionnaire included question on the mode of delivery, course contents and structure, practical and theories, things they found relevant to them, per faculty. MCQs provided 5 choices with the last option being leave blank, to understand response behaviour. Most of the MCQs were rating questions ranging from strongly agree to strongly disagree. Questions were arranged in a logical flow. In the last week of November 2009, the online survey was conducted.

3.2 Sampling

For this online survey, no sampling methodology was effected, since we targeted the whole population of 1660 students doing the Introduction to IT module. 452


Nevertheless, we asked students to enter their respective faculties to stratify the collected data for better categorization and analysis of the response behaviour of the students.

3.3 Data Collection and Response rate

We chose the ATutor [5] e-learning platform to host our online survey. It took around a day to familiarise, set up and test the system. Then tutors of the Intro to IT module were asked to motivate their students to fill in the online survey forms. It is to be noted that, no rewards were given to students upon completion of the survey form. Students were sent the instructions with the link to the website mostly via email. Students had to register on the website with their email to be able to access the survey form. Only 1 attempt to the survey was allowed. The survey took place from the 23rd to the 28th of November 2009. The students were given 5 days to respond to the survey. An extension of 1 day was allowed and on day 3 a reminder was sent to the student via email. A response rate of around 30% was achieved with 492 responses. The survey targeted each of the 5 faculties of UoM namely the engineering, law and management, science, humanities and agriculture to determine their needs.

4. DATA ANALYSIS AND INTERPRETATION

Overall, 29.6% of students responded to our survey as shown in Figure 1. The highest percentage comes from the Agriculture faculty and the lowest percentage is from the Faculty of Engineering. This is difficult to explain if we rely on the fact that Engineering students are generally the best ones in IT while Agriculture students are generally the weaker ones. Instructions to the survey were sent to the students via their emails. Tutors and program coordinators were asked to motivate their students to fill in the survey form. It is to be noted that no rewards were promised to students. The only motivational factor was the emotional appeal that this would help to enhance the course for future students. The results show that students who were regularly in contact with their tutors and responded via their 453


mails showed a much higher response rate than students who were contacted otherwise. In addition, it was noted that the email reminder on day 3 did gave a 6% boost to the number of respondents.

Figure 1 – Number of Respondents per Faculty

In terms of actual numbers, the Faculty of Law and Management was the clear winner with 203 (41.3%) responses. We got a very significant number of responses from the Faculty of Science as well. Indeed, 134 (27.2%) students responded to the survey. It was not properly understood why the response rate was so low for Engineering students. One plausible reason for this huge difference in response with other faculties might be the predominance of males in the Engineering faculty. Indeed, the ratio of male to female for the CSE1010e module for the academic year 2008-2009 was 3:1. As explained below, girls showed more willingness to fill in the online survey form than boys.

The ratio of girls to boys for the module CSE1001e is 55:45. However, for the survey we saw that the girls to boys ratio was 69:31. This difference is quite significant which leads us in believing that girls shows higher response for online surveys.

4.1 Responses per day

Figure 2 and 3 below show the percentage of respondents for each faculty and for each day that the survey was available online. Our aim in collecting this data was to get an idea of the behaviour of respondents with respect to a set deadline which the respondents were aware of. 20% of respondents filled the survey form on the 454


first day itself. Day 4 saw the largest number of responses. This can be explained by a reminder sent on the previous day and this caused an average rise of 6% in the response rate. Contrary to our expectation, only 17% of respondents filled the form on the last official day of the survey. At the last minute of the first deadline, the survey was extended by one day and the respondents were made aware of this through another email reminder. Only a minority of 4% responded after the deadline was extended. Although, we can roughly speak of a downward trend in the number of respondents as the deadline was getting closer, there are many peaks and troughs in the graph. This leads to conclude that there is very little correlation between the number of respondents filling a survey form online and the number of days before the deadline is due.

Figure 2 – Number and Percentage of Respondents on each day

Figure 3– Percentage of Respondents on each day 455


4.2 Responses at various time intervals

The day was divided into three time intervals: 23:31 – 09:30, 09:31 – 17:30 and 17:31 – 23:30.

The

first

interval

corresponds to very late at night or early in the morning. The second interval corresponds to the time during which the respondent was very likely to

Figure 4 – Percentage of Respondents for each interval The survey was conducted from Monday to Friday, from the 23rd to the 27th of be at the university or at work.

November. An extension of one day was allowed on Saturday 28th of November. And the last interval corresponds to the time during which the respondent is expected to be at home. We realised that there can be some situations where the respondent may be at home but was counted as being at work/university. The reverse might also be possible but since our data size if quite significant, we believe that the results can be trusted upon. As shown in figure 4 and 5, 63% of respondents have preferred to fill the survey form from the comfort of their home as shown in Figure 4. We argue that if all respondents had internet facility at home, this number would have been much higher. Thus, we conclude that respondents viewed the survey as being an additional task which they had to complete but which did not require immediate attention. They kept it as a secondary task to be completed at their own ease and pace.

Figure 5 – Time of day at which the survey form was filled in 456


4.3 Changing of preferences

It is good to point out that 12% of respondents modified their submissions at a later time as in figure 6. The Science respondents showed a

greater

change

willingness their

to

survey

preferences while the Law and

Management

Engineering were

not

modifying

and

respondents keen their

on

survey

Figure 4 – Number of respondents modifying their survey forms at a later time

form once they had been submitted. In fact, respondents were not told that they could modify their survey form once it had been submitted. Thus, we conclude that respondents showed a significant degree of enthusiasm to modify their survey preferences at a later time. We believe that if respondents were told that they could modify their answers even after their submission, more of them would have modified their survey answers. This in turn shows the flexibility of an online survey.

4.4 Mean time in completing the survey form

Figure 7 shows the mean time and standard deviation (in minutes) to complete the survey form. As can be seen, a respondent took 16 minutes on average to complete the survey with a standard deviation of 11 minutes. It is to be noted that respondents who took more than 1 hour or less than 1 minute to complete the form were not used in the calculation of the mean and standard deviation. These were considered as outliers.

457


Figure 5 – Mean time for completing the survey in minutes Law and Management students were the fastest ones while Agriculture students took the longest amount of time on average. Initially, we believed that 15 minutes might be a reasonable time to complete the form completely and reliably. There were a significant number of students who took less than 5 minutes but many of them did not complete the open-ended questions. Many students also took more than 30 minutes, possibly, because they had trouble with the open-ended questions which required them to generate some ideas. However, it was very encouraging to see that very few respondents have submitted completely blank forms.

4.5 Response behaviour of students while answering Multiple Choice Multiple Answers Questions (MCMAQs).

Out of 29 questions, only three were MCMAQs. Figure 8 shows the respondents for these questions. For Q4, respondents on average selected 2.56 choices from a choice of 13. For Q4, the average was 1. This suggests that respondents considered this question more like an MCQ than an MCMAQ. However, when we look more closely at the detailed results, we can see that the average number of responses for all faculties, except Law and Management, the average was higher than 1. It is difficult to explain why Law and Management respondents selected less than 1 choice on average from a choice of 6. In Q6, where respondents had to select one or more choices from a choice of four, 11% of respondents did not choose any. Again, it was Law and Management respondents who left the greatest number of blanks. It was felt that perhaps respondents had difficulty understanding this question to explain for this significant amount of blank answers for this question.

458


Figure 6 - Respondents for the Multiple Choice Multiple Answers Questions (MCMAQs)

4.6 Responses to open ended questions

Figure 9 and 10 relate to respondents

for

Open

Ended questions. 19% of respondents left the openended

questions

compared

to

4%

blank for

multiple choice questions. It was noted that the last question

which

respondents

how

asked they

Figure 9 – Number of blank answers per faculty

think this module can be improved and to offer their suggestions got the greatest number of responses. Furthermore, looking at the graph, it is seen that the number of blanks for each question for each faculty is very similar. This leads us to believe that respondents either responded to all four questions or to none. Responses such as none or nothing were considered as non-blank responses for the purpose of this survey. Engineering students were least eager to respond to open-ended questions while Agriculture and Science students offered the most number of responses in terms of percentage.

459


Figure 10 - Respondents for the unanswered Open Ended Questions

4.7 How many respondents choose the option leave blank when attempting the MCQs?

Overall, 4% of blank answers were obtained for MCQs as shown in Figure 11. In terms of actual numbers, this amounts to 411 unanswered MCQs over a total of 10824. Q18, Q19 and Q20 had drawn the largest number of blank answers. These three questions were about the practical contents for the IT module. About 8% of students were not keen on answering these questions although these were relevant to them. However, this can be explained by the fact that although all the content for the practical sessions are available online to all students, they do not have formal practical sessions at the University. This explains the unwillingness of some students to comment on the practicals.

The question which had drawn the least number of blanks was the gender question. Only 1% of respondents left this blank. Apparently, this was considered to be very important to answer. Two other questions namely, “How would you rate your IT knowledge prior to CSE1010e?” and “Which mode of delivery do you think would be most appropriate for the Introduction to IT module?” also received a 99% response rate. On average the last three questions received fewer blank answers than the middle questions. It was not clearly understood why 4% of respondents did not choose their department.

460


Humanities students showed the greatest disposition to reply to MCQs. This result is in contrast with their reluctance to answer open-ended questions where their response rate was below the average. Indeed, 41% of MCQs were answered by all Humanities respondents. This suggest that Humanities respondents were perhaps more aware of the importance of the results than other respondents because surveys are an important tool for them in many of their studies.

Figure 7 – Number of leave blank answers

461


5. OUR EXPERIENCE BASED ON THE ONLINE SURVEY CARRIED OUT. Here we share some of our experience while carrying out the online survey. Several advantages were noticed and they are enumerated below: 

Online surveys compared to mail and phone surveys are cheaper and have a faster turn-around time [27].



It is flexible. People are more likely to respond to online surveys because they can complete them at any time.



Online surveys are very easy to set up and manage. Many survey sites allow people without required programming skills to create relevant surveys very easily and rapidly.



Online surveys eliminate interviewer bias. Respondents complete the survey by themselves and there is no need for an interviewer.



Once the survey is online, it is a simple step to promote it, either through email (with a link enclosed), via a link from a website or referenced by other forms of advertising. Anyone who has the link can be connected instantly to the survey.



Multimedia content like pictures, audio, video can be included. However, these should only be used when it is certain that respondents have good broadband facilities. Otherwise, this can have an adverse effect on response rate [24] as respondents are then faced with long downloading time which reduced their motivation to complete the questionnaire and to submit it.



Statistical compilation is done in real time. Survey results can be monitored throughout the duration of the survey. This is helpful to get an idea of the response rate and to know when to send the reminders.



A user can take as much time needed to fully complete the survey. He/She can read the questions as many times as required. The respondent may also modify his/her responses at a later time if the survey allows it.



It is surely the best method to collect data if the population to be sampled is an international one.

462




Although in this particular survey, respondents had to register in the system with their real names and ID, it is possible for the survey to be conducted in an entirely anonymous way. This ensures that the privacy and confidentiality of the data are protected. We believe that this factor can indeed increase response rate for certain types of surveys.

However, online surveys do have short comings and they are as follows: 

Analysing response from open ended questions are time consuming



It is impossible to determine who actually completed the survey



It is also difficult to know whether the respondents actually understood the question being asked.

5.1 Demographic response behavior

Our survey has been performed in an academic setting whereby we had control over the respondents and this can explain the commendable response rates. Students can be easily monitored in an academic environment for the completion of the survey form. And this is because we have got the help of our colleagues in motivating and monitoring the students. In addition, in a university environment, the population can be considered IT literate since every student must do an introductory course in IT as we mentioned in the introduction. Hence it can be inferred that similar response behaviour can be achieved in an academic environment, assuming that the population is IT literate. Also the findings of the study can be extended to the wider Mauritian population in the 18-34 age range, expecting similar response and behaviour towards online surveys. Based on Facebook[6] demographic profile as mentioned in the Introduction, nearly 82% of Mauritius facebook users are in the age bracket of 1834, being digital natives of our digital island. And our academic environment can be fairly described as a subset of the digital natives with fairly the same behavior. Nevertheless, findings of this survey cannot be generalized to the whole Mauritian population, since the majority of older Mauritian population is not familiar with IT 463


and the Internet. Also, they come from various literacy group, making response to internet surveys very low. Such groups are more comfortable with paper and pencil survey or the traditional face-to-face interviews. Also we believe that the findings are restricted to countries having more or less the same characteristics of Mauritius in terms of Internet Usage, IT literacy and ICT infrastructure. Countries having IT as a luxury will not benefit from such kind of surveys. 6. CONCLUSION The results for our online survey confirm our belief that online surveys can be considered as an effective, flexible, time and cost saving medium in collecting and analysing information in academic institutions in Mauritius. The survey was carried out at practically no cost using a free e-learning platform, ATutor [5]. Time spent was only in setting up the online survey and contacting the students. It took us around 1 day to set up a survey questionnaire online using aTutor [5] and 1 week to collect the results. Analysis of the results was instantaneous due to the statistical tools provided by the platform. The whole student population doing the Introduction to IT module was targeted. This was possible since we had the mailing lists of all the different batches of students.

A commendable response rate of 29.6% was obtained without any incentive like gifts being provided. The emotional appeal that this would help to enhance the course for future students, was used as a motivational element for current students to complete the survey. A reminder proves to be useful in boosting response. A 6% increase in the response was noted. MCQs were properly answered with only 4% choosing the leave blank option. MAMCQs did confuse the students since many of them chose only one answer, though multiple answers could have been selected. Open ended questions were not popular among students of the different faculties with an average of 18.5% unanswered.

However, care should be taken while generalizing our survey findings to other sampled population. Mauritius has a sound IT infrastructure with a commendable 464


IT literacy especially among the digital natives in the 18-34 age brackets. This explains the responsiveness and the success of our survey. Also, the survey has been conducted in an academic environment, whereby students are considered to be versed in IT, are literate and whereby we had a fair control on the students. It is believed that online surveys with elderly Mauritians will be rather painful and nonresponse rates will be rather high, since they are not versed in IT and the use of the internet.

Our survey also confirmed the flexibility provided by online surveys. Indeed 55.5% of respondents completed the survey after working hours (17:30 – 23:30), showing that respondents completed the survey at a time convenient to them, compared to conventional survey methods. The mean time taken to complete our survey was 15.5 minutes with a standard deviation of 11.2 minutes.

To conclude, it is strongly felt that online surveys do have a promising future in Mauritius. A major issue with any survey is response rate. We believe proper monitoring like reminders and incentives have a positive impact on the response. Response quality depended on the proper setting of MCQs, MCMAQs and openended questions. MCMAQs should be clearly labeled as multiple answers, else students tend to choose only one answer. Unless necessary, open-ended questions should be avoided. Proper design of questionnaire, clarity of questions and proper use of MCQs and clearly labeled MCMAQs improve the quality of the collected data.

Our work was based at University level whereby student’s mailing list was available. The level of computer literacy was high and the targeted age group was between 18 to 24 years. Online surveys on a national level in Mauritius may prove to be more difficult. Care should be taken while choosing the targeted audience since IT literate Mauritians are mostly in the 15-34 age group. Older people may not find online surveys suitable. Mailing list of people should be available and proper incentives should be provided to get the desired response. 465


In the near future, we intend to carry an online survey at national level to investigate response rate and response quality. 7. ACKNOWLEDGEMENTS

The authors would like to thank the year 2009 CSE 1010e survey sub-committee for the design of the survey questionnaire and the CSE1010e committee for their invaluable suggestions. 8. REFERENCES [1] PriceWaterHouseCoopers, 2007. The National ICT Strategic plan 2007-2011. [Online] Available at http://www.gov.mu/portal/goc/telecomit/files/NICTSP.pdf [Accessed 10 Dec 2009] [2] Internet World Stats. List of countries classified by Internet Penetration Rates. [Online] Available at http://www.internetworldstats.com/list4.htm [Accessed 15 December 2009] [3] ICTA, (2004,p6). The ICT Sector in Mauritius, An Overview. [Online] Available at http://www.gov.mu/portal/goc/ncb/file/ictview.pdf [Accessed 14 December 2009] [4] LINDA J. SAX, SHANNON K. GILMARTIN AND ALYSSA N. BRYANT. Assessing, Response Rates and Nonresponse Bias in Web and Paper Surveys, Research in Higher Education, v44 n4 p409-32 Aug 2003. .[Online] Available at http://www.springerlink.com/-content/v71hp772066t1q85/ [Accessed 14 December 2009] [5] ATutor.

Learning

Management

System

[Online]

Available

at

http://www.atutor.ca/ [Accessed 10 December 2009] [6] Facebook

2009.

Advertise

on

Facebook.

.

[Online]

Available

http://www.facebook.com/ads/create/ [Accessed 10 December 2009] [7] surveymonkey 2009. [Online] Available at [Accessed 5 December 2009] 466

http://www.surveymonkey.com

at


[8] esurveyspro

2009.

[Online]

Available

at

http://www.esurveyspro.com/

[Accessed 2 December 2009] [9] surveygizmo 2009. [Online] Available at http://www.surveygizmo.com/ [Accessed 2 December 2009] [10]moodle 2009. What is Moodle. [Online] Available at http://moodle.org/ [Accessed 3 December 2009] [11]Marc I. Tillman, Amplitude Research Inc., Online Market Research Surveys Maximizing

Your

ROI,

retrieved

December

13,

2009

from

http://www.greenbook.org/marketing-research.cfm/online-market-researchsurveys [12]COOK ET AL. ,2000. A meta-analysis of response rates in web or internet based survey, Educational and Psychological Measurement, 60(6), 821-836 [13] PITKOW, J AND RECKER, M. (1994). results from the first world- wide web user survey. computer networks and ISDN systems, 27(2), 243-254. [14] SHEEHAN, K AND HOY, M. (1999). Using E-mail To Survey Internet Users In The United States: Methodology and Assessment. Journal of Computer Mediated Communication, 4(3). [15] SHEEHAN, K. (2001). E-mail Survey Response Rates: A Review. Journal of Computer Mediated Communication, 6(2). [16]BOSNJAK, M AND TUTEN, T. (2001). Classifying Response Behaviors in Web-based Surveys. Journal of Computer Mediated Communication, 6(3). [17]MANFREDA, K., BATAGELJ, Z., AND VEHOVAR, V. (2002). Design of Web Survey Questionnaires: Three Basic Experiments. Journal of Computer Mediated Communication, 7(3). [18]PEARSON, J AND LEVINE, R. (2003). Salutations and Response Rates to Online Surveys. Fourth International Conference on the Impact of Technology on the Survey Process. [19] KONSTAN, J., ROSSER, B., ROSS, W., STANTON, J., AND EDWARDS, W. (2005). The Story of Subject Naught: A Cautionary but Optimistic Tale of Internet Survey Research. Journal of Computer Mediated Communication, 10(2). 467


[20]WRIGHT, K. (2005). Researching Internet-Based Populations: Advantages and Disadvantages of Online Survey Research, Online Questionnaire Authoring Software Packages, and Web Survey Services. Journal of Computer Mediated Communication, 10(3). [21]MICHAEL B. HAMILTON, Ipathia Inc., Online Survey Response Rates and Times: Background and Guidance from Industry. [Online] Available at http://www.supersurvey.com/papers/supersurvey_white_paper_response_rates. pdf [Accessed 2 February 2011 ] [22]http://blog.vovici.com/blog/bid/22879/ESOMAR-Online-Research-2009 [23]COOK, C., HEATH, F., & THOMPSON, R. (2000). A meta-analysis of response rates in Web- or Internet-based surveys. Educational & Psychological Measurement, 60 (6), 821-36. [24]DEUTSKENS, E., DE RUYTER, K.,WETZELS, M., & OOSTERVELD, P. (2004). Response rate and response quality of internet-based surveys: An experimental study. Marketing Letters, 15, 21–36. [25]DEUTSKENS, E., DE JONG, A., DE RUYTER, K., & WETZELS, M. (2006). Comparing the generalizability of online and mail surveys in cross-national service quality research. Marketing Letters, 17(2), 119–136. [26]DALTON, J AND HICKEY, A. (2009). Proceed with Caution: Internet-Based Survey Research. Bloomberg Finance, Vol 2, No. 2. [27]DILLMAN, D. A. (2000). Mail and Internet surveys: The tailored design method. 2nd ed. New York.Wiley

468