Title

Benchmarking web accessibility evaluation tools: Measuring the harm of sole reliance on automated tests

Document Type

Conference Proceeding

Publisher

ACM

Faculty

Faculty of Health, Engineering and Science

School

School of Computer and Security Science

RAS ID

17402

Comments

This article was originally published as: Vigo, M., Brown, J. A., & Conway, V. L. (2013). Benchmarking web accessibility evaluation tools: Measuring the harm of sole reliance on automated tests. In W4A '13: Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility. (article no. 1). New York, NY : ACM,. Original article available here

Abstract

The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help in reducing the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting aside further testing that entails expert evaluation and user tests. In this paper we empirically show the capabilities of current automated evaluation tools. To do so, we investigate the effectiveness of 6 state-of-the-art tools by analysing their coverage, completeness and correctness with regard to WCAG 2.0 conformance. We corroborate that relying on automated tests alone has negative effects and can have undesirable consequences. Coverage is very narrow as, at most, 50% of the success criteria are covered. Similarly, completeness ranges between 14% and 38%; however, some of the tools that exhibit higher completeness scores produce lower correctness scores (66-71%) due to the fact that catching as many violations as possible can lead to an increase in false positives. Therefore, relying on just automated tests entails that 1 of 2 success criteria will not even be analysed and among those analysed, only 4 out of 10 will be caught at the further risk of generating false positives. Copyright 2013 ACM.

Share

 
COinS