An Approach Based on Metrics for Monitoring Web Accessibility in Brazilian Municipalities Web Sites André P. Freire
Thiago J. Bittar
Renata P. M. Fortes
University of Sao Paulo ICMC P.O. Box 668 Sao Carlos,SP, Brazil
University of Sao Paulo ICMC P.O. Box 668 Sao Carlos,SP, Brazil
University of Sao Paulo ICMC P.O. Box 668 Sao Carlos,SP, Brazil
[email protected]
thiago
[email protected]
[email protected]
ABSTRACT Monitoring and measuring the accessibility of government Web sites is an important challenge for regulators and policy makers. Moreover, over the next few years, e-government (e-gov) services are expected to expand and it is necessary to ensure access for everyone. In this paper, we present a metric based approach for evaluating municipalities Web pages using automatic accessibility evaluation tools. The sampling of the pages was done by the tool E-GOVMeter, and the accessibility evaluation and generation of the metrics was done by means of an adaptation of the tool Hera. The results show that much work should be done to improve the accessibility of Brazilian municipalities Web sites. Although it has limitations, the use of automatically generated accessibility metrics is a powerful tool for helping measuring and monitoring the accessibility of e-gov Web sites.
Categories and Subject Descriptors H.5.4 [Hypertext/Hypermedia]: User issues
General Terms Design, Human Factors
Keywords Web Accessibility, Web Metrics, E-Government, Web Accessibility Evaluation
1.
INTRODUCTION
The World Wide Web (WWW) was conceived as a technological support for publishing content using simple and powerful standards. Early in the development of the Web, Tim Berners Lee stated that “The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect”. Providing universal access to content available on the Web means to make Web content accessible for everyone. This
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SAC’08 March 16-20, 2008, Fortaleza, Cear´a, Brazil Copyright 2008 ACM 978-1-59593-753-7/08/0003 ...$5.00.
implies that developers should be aware of the needs for different types of users to enable them to fully access, interact with and contribute to Web content. The importance of the universal access is even more evident in public environments, to provide people with their citizenship rights. The lack of observance of accessibility standards would turn the evolution of the Web systems of Brazilian municipalities one more factor of social exclusion. In Brazil, the promulgation of the Statute 5.296/2004 has brought accessibility to discussion in the e-government (egov) scenario. Besides the development and spread of accessibility recommendations, it is necessary to provide means for measuring and monitoring how accessibility has been observed or not by government institutions. Accessibility metrics are good instruments to get quantitative data to monitor the accessibility of e-gov Web sites. This paper presents an approach based on a crawler to identify municipalities that have Web sites and a metric based approach for accessibility evaluation, enabling a quantitative analysis of Web accessibility in Brazilian municipalities. This paper is organized as follows: Section 2 presents the main concepts related to Web accessibility and Web accessibility metrics. In the section 3 we describe our approach for crawling and evaluating accessibility of Brazilian municipalities Web sites. In Section 4, the summary of the main results of the evaluation are presented. In the Section 5 we discuss related work and finally in the Section 6 we present conclusions and future work.
2.
WEB ACCESSIBILITY
Web accessibility has become an important issue in Web development. Making Web more accessible is related to providing access to content to people with different abilities, using different devices, including assistive technologies, such as screen readers, non-conventional input devices and others [16]. Guidelines are important tools to help the development and evaluation of Web accessibility. Although they should not be used alone, they provide good indicators to perform conformance analysis and monitoring. Each WCAG [18] guideline is divided into checkpoints, which indicate good practices for constructing accessible Web applications and help developers to avoid barriers that may prevent users to use. Checkpoints define a wide range of issues related to accessibility, such as the use of alternative text for audio and visual content, correct use of tables,
2421
appropriate use of forms and others [12]. Evaluation is a key activity to produce accessible Web applications. The main goal of performing accessibility evaluations is to identify accessibility barriers in Web sites and to report these problems to be repaired.
2.1
the available documentation, the value Wi = 0.05 is used for all barriers 1 . Following, we show the simplified formula for a single page, which does not consider the test confidence:
U W EM =
Web Accessibility Metrics
W AB =
j=1
bij i=1 ( Bij
Pn
T
(2)
b
Defining quantitative metrics for evaluating software products and processes is very important for verifying and improving software quality. The process of measurement corresponds to the assignment of numerical values or symbols to attributes or entities of the real world, which describe them according to well defined rules [8]. Metrics are very important to help the process of understanding, controlling and improving organizations products and processes [8]. Metrics may be very important to help organizations to improve accessibility of their products. Many research projects have investigated on solutions for defining and improving metrics for Web accessibility. The main goal of these works is to define means to get single values that summarize results from a guideline review based evaluation [12]. Sullivan & Matson [15] proposed a metric which considered the concept of “potential problems”. For example, an image without alternative text is considered an accessibility barrier. Thus, any image in a page is considered a “potential problem”, as it may be an accessibility barrier if an alternative text is not defined properly. The formula considers the ration between the total number of problems encountered in a page (b) and the total number of potential problems (B). The concept of “weight” for barriers is another important concept for accessibility metrics. Gonzales et. al [11] proposed an accessibility metric based on the model “WebQEM” (Web Quality Evaluation Method) [13]. In this metric, a global ratio (GP ) is calculated for a given page based on the percentual of accessibility problems (Bi ) related to their respective potential problems (bi ) for each barrier. This value is later multiplied by a weight, defined according to the impact of each barrier (Bi ). Parmanto & Zeng [14] proposed the metric WAB (Web Accessibility Barrier). This metric applies the concepts of potential problems and weights for barriers. Besides, it also takes into account the total number of pages that a given site contains (T ). The weights are defined as the inverse of the priority of each WCAG [18] checkpoint, which are defined as 1, 2 or 3. The formula for calculating WAB is as follows: PT
T Y nbpj 1 X 1− (1 − Wb ) T j=1 nBpj
Buhler et. al [3] proposed some improvements on the metric UWEM [5]. They have used some probability properties and aggregated some issues related to the complexity of the page, considering the number of violations of a given checkpoint in relation to the total number of violations (Bn ). The formula of the A3 is denoted as follows:
A3 = 1 −
b
b n
( Bi + b i )
(1 − Wi )
i
(3)
i=1
Buhler et. al [3] performed a preliminar experimental study, using the value Wi = 0.05 for all barriers. This study involved the evaluation of six Web pages by a group of six users with different types of disabilities. In the experiment, quantitative values for the use of the pages were defined. After, authors compared these values with values obtained from A3, UWEM and the metric of potential problems. The metric A3 had the best performance in this experiment. The metric WAQM (Web Accessibility Quality Metric) [17] considers the guidelines from WCAG classified according to the principles: Perceivable, Operable, Understandable and Robust. Different from other metrics, this metric also considers problems identified as warnings by evaluation tools. Two different approaches are considered for the calculus of WAQM, using a hyperbole approximation, according to formulas (4) and (5). A decision based on experimental values must be taken in order to choose which formula should be used. The summarized formula is illustrated as following. The variable Nx denotes the number of checkpoints for a given principle, or error, for example, when x = p, Np is the number of checkpoints related to the principle perceivable. The weights Wz were experimentally defined as W1 = 0.80, W2 = 0.16 and W3 = 0.04. The values a and b are also defined experimentally. The authors of WAQM have set them as a = 0.3 and b = 20. This metric ranges from 0 to 100, where 100 corresponds to accessible sites, different from the other metrics that considers high values as more inaccessible pages. X=
)(Wi ) (1)
UWEM (Unified Web Evaluation Methodology) [5] is another important metric. In this metric, the final value represents an approximation of the probability of finding a barrier in a Web site that could prevent a user from completing a given task. This metric also applies the concepts of potential problems and weights for barriers. Weights for barriers should be defined after carrying out experiments with users with different disabilities. In its full formula, the metric also considers the concept of “evaluator confidence”, which is related to the reliability of the evaluation. Although this metric has very interesting features, there is no report on experimental derived weights for barriers. In
T Y
1 X 1 T T j=1 N
X
Nx
x∈{p,o,u,r}
1 NY
X
X
Wz Axyz
y∈{e,w}Nxy z∈{1,2,3}
(4)
Axyz =
8 < ( bxyz )( −100 ) + 100, B b xyz bxyz
: −a( B
xyz
bxyz Bxyz
) + a, otherwise
