ACCESSIBLE - Applications Design and Development

Web accessibility assessment Tool – WaaT

Introduction & Information details for the WaaT Tool

The Web Application Assessment Tool is a tool (Web and Standalone version) for the accessibility verification of Web applications. WaaT supports developers and designers and accessibility analyzer in test existing, or under development, web sites and web applications. It provides users with the ability to define some input parameters, rules or constraints regarding the accessibility evaluation (e.g. evaluate a web page according to a specific disability or a particular existing guideline). WaaT helps developers and designers to find and correct errors or bad configuration which prevent impaired users to be able to correctly consume the web application. The product has been entirely developed within the research activities held during the ACCESSIBLE project.

The assessment tool, in both version standalone or web module on ACCESSIBLE portal, has been designed and developed to take as input the URL (or local) path of the web page or directly the html source code and to produce as output the evaluation results of the accessibility assessment process.

Every aspect of a web application is taken into account during the evaluation of the accessibility assessment process:

  • Complete site parts are crawled automatically by the tool (maximum number of pages can be specified by the user as a parameter of the process).
  • The markup code, both HTML and XHTML, is validated using the W3C Markup Validator which is embedded in the WaaT.
  • HTML is parsed to obtain detailed information on elements and attributes which are used to compose the pages.
  • ARIA UI components are validated in order to make sure that they are well described and that their role and value labels are correctly identified. This is very important as most of the new web pages include JavaScript UI components and to expose the accessibility information these components should include the ARIA markup.
  • The CSS are validated by the W3C CSS Validator embedded in the WaaT.
  • All the information is collected and used to evaluate the accessibility of the examined pages using the accessibility requirements and constraints specified by the user.

The WaaT consists of the following six main components which are described with more details in the public deliverables D3.3 and D5.2 and are being presented in the figure 1 below:

  • 1. A Graphical User Interface (GUI):The GUI of the WaaT tool consists of a set of forms that can be accessed by users, in order to help them with the accessibility assessment of preferable web applications. The GUI of the assessment tool is also responsible for the representation of the assessment results after the completion of the evaluation process. Via the GUI, the user is able to select the preferable tests to be executed using knowledge concerning the HAM methodology, which is included the ACCESSIBLE ontologies. Moreover, using the GUI, the user can load a Virtual User Model, on which the evaluation process will be based. After the completion of the evaluation process, the Web Accessibility Evaluator returns the results to the GUI, so as to be presented to the user.

Figure 1: WaaT architecture overview

Figure 1: WaaT architecture overview

  • 2. The Rules Inference Engine: The Rules Inference Engine is responsible for the communication between the WaaT and the ACCESSIBLE ontologies where knowledge regarding the HAM methodology is stored. The Rules Inference Engine is able to run SWRL rules as well as execute SPARQL queries, in order to extract specific knowledge from the ontology.
  • 3. The XML Storing/Loading Module: As the execution of SWRL rules defined in the ACCESSIBLE ontology is generally a time-consuming process, an XML storing/loading module is introduced. This module is able to automatically generate an XML file containing all the necessary knowledge of the ontology that is required for the assessment process. After the generation of the XML document the XML Storing/Loading module is responsible for the “virtual connection” between the WaaT tool and the ontology.
  • 4. The Core Web Applications Assessment Module: The Core Web Applications Assessment Module is the core component of the WaaT tool and includes all the required algorithms and methodologies for the execution of preferable user’s selections. This module consists of the following six sub-components:
    • i. The Web Crawler, which returns the single pages of a web site, when an entire web site is being selected to be evaluated.
    • ii. The W3C Markup Validator , which evaluates the conformance of a web page against the technical specifications of HTML and XHTML.
    • iii. The HTML Parser that parses the web page source code and returns the necessary information concerning the desired elements/attributes of the HTML/XHTML.
    • iv. The W3C CSS Validator, which evaluates the conformance of a CSS file against the CSS 2.1 Specifications.
    • v. The CSS Parser that is responsible for the parsing of all the CSS files that are connected with the examined web page.
    • vi. The Web Accessibility Evaluator, which is the core component of the Core Web Applications Assessment Module performing the accessibility evaluation of a web application, according to the user’s preferences.
  • 5. The Reporting Module: The Reporting Module of WaaT is responsible for the generation of various types of reports including the results of the evaluation process. The supported report types include the following:
    • A PDF report containing all the errors and warnings of the evaluated web page(s). For each error/warning, a description, a tip for its fixation and the corresponding HTML source code (where the violation appears) are provided. Moreover, the success criterion and the technique of WCAG 2.0 (where the error/warning refers to) is also provided.
    • An EARL-based report of the detected errors and warnings. EARL is a standard format for support of the evaluation of Web applications. It contains a vocabulary to describe the results of a test’s execution, in order to facilitate its processing and interpretation by different tools. EARL is expressed in RDF, which can be extended easily and be adapted to any domain, as in this case accessibility.
    • A PDF version of the EARL-based report, containing all the information of the EARL-based report in human readable format.
    • A report containing all the HTTP content transferred between the WaaT and the server that hosts the evaluated web application, using the HTTP Vocabulary in RDF. This report can be used as input to other web accessibility evaluators, in order to perform a similar use case scenario and compare their results with those of WaaT.

Owner/Main Developer (including also key partners contribution)

CERTH/ITI, FORTH (implemented the User Interface of the online tool)

Intelectual property Rights (IPR issues, Licences)

MIT licence

Nature of the Code

Open source

Published Papers

  • Lopes, R., Votis, K., Carriço, L., & Likothanassis, S. A Service Oriented Ontological Framework for the Semantic Validation of Web Accessibility. In M. Cruz-Cunha, E. Oliveira, A. Tavares, & L. Ferreira (Eds.), Handbook of Research on Social Dimensions of Semantic Technologies and Web Services, IGI Global, Hershey, 2009, 49-67, DOI=10.4018/978-1-60566-650-1.ch003
  • Lopes, R., Votis, K., Carriço, L., Tzovaras, D., and Likothanassis, S. Towards the universal semantic assessment of accessibility. In Proceedings of the 2009 ACM symposium on Applied Computing, (Honolulu, USA, 2009), ACM, 147-151, DOI=10.1145/1529282.1529311
  • Lopes, R., Votis, K., Carriço, L., Tzovaras, D., and Likothanassis, S. The semantics of personalised web accessibility assessment. In Proceedings of the 2010 ACM Symposium on Applied Computing, (Sierre, Switzerland, 2010), ACM, 1440-1441, DOI=10.1145/1774088.1774394
  • Votis, K., Lopes, R., Tzovaras, D., Carriço, L., and Likothanassis, S. A Semantic Accessibility Assessment Environment for Design and Development for the Web. In Proceedings of the 5th International Conference on Universal Access in Human-Computer Interaction. Part III: Applications and Services, (San Diego, USA, 2009), Springer-Verlag, 803-813. DOI=10.1007/978-3-642-02713-0_86
  • Partarakis, N., Doulgeraki, C., Antona, M., Oikonomou, T., Kaklanis, N., Votis, K., Kastori, G.E., and Tzovaras, D. A unified environment for accessing a suite of accessibility evaluation facilities. In Proceedings of the 6th international conference on Universal access in human-computer interaction: design for all and eInclusion - Volume Part I, (Orlando, USA, 2011), Springer-Verlag, 267-275.
  • Fernandes, N., Lopes, R., and Carriço, L. An architecture for multiple web accessibility evaluation environments. In Proceedings of the 6th international conference on Universal access in human-computer interaction: design for all and eInclusion - Volume Part I, (Orlando, USA, 2011), Springer-Verlag, 206-214.
Strengths Weaknesses
  • Automated & Semi-automated accessibility checking of Web applications (URL and source code) according to the selection of different categories of disabilities and combination of them, impairments and personas
  • Automated & Semi-automated checking of conformance to WCAG 2.0 guidelines, ARIA 1.0 and other well known accessibility standards
  • Web and standalone supporting versions of the tool
  • Evaluation of all pages of a web application is done in a very short time
  • The evaluation method is a combination of automatic and manual evaluation methods
  • Identify some actual accessibility problems (errors) in Web applications
  • Identify pages containing elements that may cause problems (warnings)
  • Annotated code view where the problem/error identified
  • Checking the CSS and Html syntax of the applications’ source code
  • Supporting the accessibility assessment of secured Web applications (Https protocol support)
  • Report Generation: Human and machine readable (PDF and EARL-based) accessibility reporting results
  • User-centricity
  • Real-time, accurate, reliable, and easy to use
  • Provide customized and manageable accessibility assessment services
  • Simplification of accessibility assessment procedure
  • The adopted evaluation method does not require a high level of proficiency and experience
  • To train developers in accessibility and usability issues
  • To provide evidence of the accessibility and usability for designers and developers
  • WaaT improves flexibility as it is possible to modify or add checkpoints and tests
  • Integration of Waat to the NetBeans IDE
  • Automated evaluation fail to fully examine all the indicators of accessibility since evaluation of some indicators requires manual examination and judgment by experts
  • Missing of an integrated Web browser for improved presentation capabilities to the standalone version of the WaaT tool
  • Non integration of the tool with Content Management Systems (e.g. Wordpress), other IDEs (e.g. IBM Eclipse), Web browsers as a plugin (e.g. Mozilla firefox)
  • The Tool is unable to provide interpretation of the results and expert suggestions during the assessment procedure
  • The assessment performance needs more improvement when a large Web application must be evaluated
  • The WaaT tool doesn’t support the accessibility evaluation of non European standards such as the Section 508 accessibility standard

Contribution to the state of the art

There is a large number of software tools performing accessibility evaluation of web sites based on the guidelines of popular accessibility standards, such as WCAG 1.0, WCAG 2.0 and Section 508. Recently some tools partially supporting the WAI-ARIA guidelines have been also appeared. The most common technologies that are checked include Cascading Style Sheets (CSS), XHTML, PDF, images, Synchronized Multimedia Integration Language (SMIL), and Scalable Vector Graphics (SVG). The automated checking on a single web page is the most common feature supported. However, some tools support evaluation of groups of pages or entire web sites. The report of the evaluation results may include step-by-step evaluation guidance, displaying information within web pages or more formal report types, such as EARL-based reports [1]. Some accessibility evaluators provide also repair functionality by changing the source code of the web pages, helping with captioning audio or video content, or converting document into accessible mark-up.

Many tools, such as the Foxability, WAVE, HERA and Hera-FFX have been developed based on the WCAG 1.0 and 2.0 guidelines. Also AChecker is an open source web accessibility evaluation tool developed by the Adaptive Technology Resource Centre at the University of Toronto. It supports a variety of international accessibility guidelines like Section 508, Ley Stanca (Italy), WCAG 1.0 (levels A, AA and AAA) and 2.0 (levels A, AA, and AAA), and BITV 1.0 (Germany). AChecker presents results in three categories: known problems, likely problems and potential problems. Worldspace FireEyes is a free web accessibility evaluation tool introduced by Deque Systems, Inc evaluating the compliance of a web site according to standards such as WCAG 1 (Priorities 1, 2 and 3), WCAG 2 (levels A and AA), Section 508 and contains some dynamic rules that test for WAI-ARIA compliance. The FireEyse also includes features such as: color contrast analyzer, dynamic report filtering, interactive issue remediation and transcripts of all pages visited in a session. Worldspace FireEyes is fully JavaScript aware and handles event-based page content. It works as a complement of the Firebug Firefox extension.

Total Validator is another accessibility validator supporting WCAG 1.0, WCAG 2.0 and Section 508 standards. It includes a HTML validator, an accessibility validator, a spelling validator, a broken links validator. There is a web version, a Firefox extension, and a desktop version of the tool available. TAW3 is an accessibility validator developed by the Spanish Fundación CTIC. It is available in two versions: a plug-in for Mozilla Firefox and in a standalone version. TAW3 analyses websites according to WCAG 1.0 and WCAG 2.0 guidelines by providing fixes and recommendations. TAW results are presented with different representation of violations (problems, warnings, and not reviewed).

Although there are many existing tools performing accessibility evaluation of web sites, there is a missing point: the personalized accessibility evaluation according to the specific needs/preferences of a user and/or a disabilitiy. This is extremely valuable considering that people with disabilities have often special needs varying from person to person, even if they are having the same disability. The current WaaT tool performs personalized web accessibility evaluation of web application using personas and virtual user models of people with disabilities and elderly.

Technical Testing Results from the developer and beneficiary validation

WaaT tool has been tested in pilot tests with developers and beneficiaries and several important usability issues occurred. In the following table we outline the most important issues that were identified during the testing, and how they were addressed for the improvement of the tool.

Issue description Priority Solved Solution / Explanation
For Colour-blindness, the colours of the images must be checked as well. 1 YES Fixed.
For VERITAS website, there were 6 elements without title attribute, and only 4 were recognised. 1 YES Fixed.
It is unclear when a long description for an image is needed or not, as the tool seems to randomly identify it as errors. 1 YES All the images are examined for the existence of "longdesc" attribute. In the latest version of WaaT, an image without a long description is considered as a warning (not as error as it was previously considered).
Keyboard accessibility issues were raised, however keyboard access keys are not considered a good practice. 1 YES The developed approaches regarding the keyboard-triggered event handlers have been based on the WCAG 2.0 technique G90 (http://www.w3.org/TR/2012/NOTE-WCAG20-TECHS-20120103/G90.html). The objective of this technique is to permit individuals who rely on a keyboard or keyboard interface to access the functionality of the content. According to this technique, the WaaT tool examines if all event handlers triggered by non-keyboard UI events are also associated with a keyboard-based event, or provide redundant keyboard-based mechanisms to accomplish the functionality provided by other device-specific functions.
No ARIA assessment 1 YES WAI-ARIA assessment is included.
The analysis of images and the forced presence of links with images. This is wrong. 1 YES As you can see in the following references, the "longdesc" attribute is poorly supported and instead of using "longdesc", it is suggested the D-Link technique to be used. According to this technique, when the developer wants to give a long description of an image, he must put a link with the letter "D" in brackets ("[D]") before or after the image. http://www.w3.org/TR/WCAG-TECHS/G73.html http://www.htmlcodetutorial.com/images/_IMG_LONGDESC.html http://www.w3schools.com/tags/att_img_longdesc.asp http://reference.sitepoint.com/html/img/longdesc#longdesc__fig-ff-longdesc http://en.wikipedia.org/wiki/HTML_element#longdesc So, we've changed a bit the specific approach, according to the following: 1) The specific output result is now considered as a warning (not as an error as it was considered previously) 2) Search for "[D]", not for hyperlink in general.
Unsure if the alt="" for background images is detected correctly; unsure if longdesc attribute missing should be considered an error; it is not always clear how 1 YES The "alt" attribute is checked for all the images of the web page. The lack of "longdesc" attribute is now considered as a warning, not an error.
For VERITAS website, <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr" lang="en-US"> is provided, but the error states “3.1.1 Provide an "xml:lang" attribute to the Web page, in order to identify the default language of the document”. This seems to be an error. 1 YES Fixed in the latest version of WaaT.
I become a NAN result by testing the site with accessible approach as option. 1 YES Fixed in the latest version of WaaT.
line numbers seem to be set -1 always, which makes it difficult sometimes to find the error. In the portal version the line numbers seem correct. 1 YES Fixed in the latest version of WaaT.
Not always clear how an error is corrected. 1 YES The descriptions of the errors as well as the hints have been corrected/improved.
Number of onclick event handlers without redundant onkeypress event handlers is indicated as an error, while it is not an error at all, and an alternative approach is possible, while also keyboard can be used. So this should not be an error (VERITAS website success criterion 2.1.1 line 238). 1 YES The developed approaches regarding the keyboard-triggered event handlers have been based on the WCAG 2.0 technique G90 (http://www.w3.org/TR/2012/NOTE-WCAG20-TECHS-20120103/G90.html). The objective of this technique is to permit individuals who rely on a keyboard or keyboard interface to access the functionality of the content. According to this technique, the WaaT tool examines if all event handlers triggered by non-keyboard UI events are also associated with a keyboard-based event, or provide redundant keyboard-based mechanisms to accomplish the functionality provided by other device-specific functions.
Sometimes not so clear description of the error. 1 YES The descriptions of the errors as well as the hints have been corrected/improved.
The line of problematic elements is hardly ever correct. So difficult to find the error. 1 YES Fixed in the latest version of WaaT.
While success criterion 4.1.1 is listed as consisting of errors, it is enlisted as errors in the earl reporting. (VERITAS) 1 YES Regarding success criterion 4.1.1, the assessment of the VERITAS web site returns 68 CSS warnings, which are also listed as warnings in the EARL report.
E.g. for VIPI-Project.eu, it indicates under Guideline 1.1 - Level A Number of <IMG> elements without adjacent <A> element, and this is erroneous. This is NOT an error at all. In fact, no <A> tag etc. is needed since the image is not even linked or needs to be linked. This happens for almost every single image being used. The term “erroneous” is also not correctly spelt, it should be “erroneous” 1 YES As you can see in the following references, the "longdesc" attribute is poorly supported and instead of using "longdesc", it is suggested the D-Link technique to be used. According to this technique, when the developer wants to give a long description of an image, he must put a link with the letter "D" in brackets ("[D]") before or after the image. http://www.w3.org/TR/WCAG-TECHS/G73.html http://www.htmlcodetutorial.com/images/_IMG_LONGDESC.html http://www.w3schools.com/tags/att_img_longdesc.asp http://reference.sitepoint.com/html/img/longdesc#longdesc__fig-ff-longdesc http://en.wikipedia.org/wiki/HTML_element#longdesc So, we've changed a bit the specific approach, according to the following: 1) The specific output result is now considered as a warning (not as an error as it was considered previously) 2) Search for "[D]", not for hyperlink in general.
Errors about <A> associated with images is often wrong. Seems an image always needs to be associated with <A>. This is not correct. 1 YES As you can see in the following references, the "longdesc" attribute is poorly supported and instead of using "longdesc", it is suggested the D-Link technique to be used. According to this technique, when the developer wants to give a long description of an image, he must put a link with the letter "D" in brackets ("[D]") before or after the image. http://www.w3.org/TR/WCAG-TECHS/G73.html http://www.htmlcodetutorial.com/images/_IMG_LONGDESC.html http://www.w3schools.com/tags/att_img_longdesc.asp http://reference.sitepoint.com/html/img/longdesc#longdesc__fig-ff-longdesc http://en.wikipedia.org/wiki/HTML_element#longdesc So, we've changed a bit the specific approach, according to the following: 1) The specific output result is now considered as a warning (not as an error as it was considered previously) 2) Search for "[D]", not for hyperlink in general.
For VERITAS website, <html xmlns="http://www.w3.org/1999/xhtml" dir="ltr" lang="en-US"> is provided, but the error states 3.1.1 Provide an "xml:lang" attribute to the Web page, in order to identify the default language of the document. This seems to be an error. 1 YES Fixed in the latest version of WaaT.
It seems Guideline 4.1 - Level A makes no sense as it indicated A[attributes={name=header_page}; value=[]] which I could not even find back in the source code. (VERITAS) Equally, 4 elements <A> were identified as having no title attribute. None could be found. 1 YES Fixed in the latest version of WaaT.
Legends, as asked for by the <fieldset> element are not common used today anymore, as far as I know. Creating a legend would not be a part of this tag. The "value" and "title" attributes of an INPUT-tag of the type "text" are mostly the same value and might not have to be displayed as an error. 1 YES This approach is based on the H71 technique of WCAG 2.0 (http://www.w3.org/TR/2012/NOTE-WCAG20-TECHS-20120103/H71). As it is mentioned in the specific technique: "Check that each fieldset has a legend element that includes a description for that group."
Export As Pdf Failed One Time. App Crashed. 1 YES During lots of tests, never managed to reproduce such an error.
I was not able to select more than one disabled person for evaluation assessment. 1 Partially It is not supposed to support the combination of personas. The sequential assessment of two or more personas is supported instead.
It seems to work properly but when testing the same URL more than one time under the same conditions every result (percentage of accessibility) is different!? 1 YES Fixed in the latest version of WaaT.
The standalone application had a deadlock during the initialization. 1 YES It was fixed in the latest version.
No evaluation for people who cannot move any of their body parts. 2 YES Evaluation for upper limb impaired users can be performed.
Blue letters on blue background are difficult to read 2 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
In some cases more details are needed 2 YES The descriptions of the errors as well as the hints have been corrected/improved.
Large amount of warnings not expanding correctly. 2 YES The portal provides better presentation mechanisms.
Windows size don´t adjust to the computer screen. You can´t maximize the window. 2 YES Fixed. The GUI of WaaT was made resizable.
Number of onclick event handlers without redundant onkeypress event handlers is indicated as an error, while it is not an error at all, and an alternative approach is possible, while also keyboard can be used. So this should not be an error (VERITAS website success criterion 2.1.1 line 238), rather a warning. 2 YES As shown in http://www.w3.org/TR/2012/NOTE-WCAG20-TECHS-20120103/G90.html, this is actually an error.
The information is not always sufficient. 2 YES The descriptions of the errors as well as the hints have been corrected/improved.
By clicking on the red/black/blue numbers, you get more detail per error. However this should be better indicated, as otherwise people will in general not use it. Same with possibility to do manual checks and check checkboxes etc. in the options provided when selecting success criteria; but waiting time was sometimes almost 10 minutes for 1 page. What then for a larger website? 2 YES The portal provides a better GUI. The duration of the assessment process depends on the content of the examined web page/site. However, the WaaT offers two different types of evaluation: 1) a light one, which is faster and 2) a full version that includes all the supported tests (even these that are time-consuming).
"Choosing approaches manually" does not clearly explain what you do then. Also, the HAM methodology should be explained via a help, etc. 2 YES It is better explained through the portal.
EARL reporting needs to be improved. Current version is difficult to use due to lack of overview. 2 YES Overview tables are included in the PDF version of the report.
Scrolling by mouse wheel is too slow in the standalone-application. If the program would tell me the line in which the warnings occur, it would be really easy to fix the code. 2 YES The mechanism that calculates the line in the source code where the error/warning occurs has been improved in the latest version of WaaT. Regarding the slow scrolling and other GUI problems, they are addressed in the portal.
The gui is crowded 2 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
The gui needs redesign. 2 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
The reporting should indicate on what basis the assessment was carried out. E.g. if it is based on a persona, that should be shown in the EARL reporting. As is now, the reports created do not make any mention of the assessment procedure. 2 YES Fixed. Now the PDF report contains information about the persona or impairment that has been selected.
The results should be displayed in a more understandable way, for example, grouped. (Portal) 2 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
Why is WCAG 1.0 not supported? This is still used a lot by organisations and public websites. Also, when doing it manually, you are not able to only check on one single level only, unless you start unchecking the different principles in each level. 2 YES WCAG 1.0 was initially supported and according to the reviewers' comments during the 1st review, WCAG 1.0 was removed. Checkboxes enabling the check/uncheck of all the approaches belonging to the same priority level have been added.
You need to type "http://" otherwise an error occurs - but there is no feedback regarding this - it's just a test and trial error 2 YES Fixed. Now, the "http://" can be omitted.
Red textlinks that are not underlined are not recognized. 3 YES The portal, which is the main tool for web accessibility assessment, offers a much better GUI as well as better presentation mechanisms.
Some of the error messages have spelling mistakes. This should be corrected. E.g. success criterion 2.1.1 3 YES The descriptions of the errors as well as the hints have been corrected/improved.
Window is getting closed after choosing an URL - that is confusing. After checking the URL which is displayed in a small window the Main-Window opens again. Why can the Mail-Window not stay open in the background? 3 YES A much better interface is available through the portal.
it is not clear if selected impairments in one category are remembered when the category is changed (missing united list); this works much better in the portal. 3 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
The "Use personas"-Button should be placed at the top-left corner because of overseeing it when intuitively focusing the top-left corner similar to reading a book. 3 YES The "Use personas" button has been placed at the top-left corner.
The main problem is that the reporting afterwards does not indicate what disability or persona had been selected. This makes analysis afterwards very difficult, unless you make notes separately on what you are assessing and how. 3 YES Fixed. Now the PDF report contains information about the persona or impairment that has been selected.
There is no way to customize the reporting results. 3 YES The portal offers better reporting mechanisms.
The main problem is filtering and selecting different result sets from the report. If I can do that I will be able to effectively focus and fix major problems. 3 YES The portal offers better reporting mechanisms and categorization of the results.
Green codes are appearing as black codes in the reporting, confusing. Also difficult to find errors as there is no table of content or headings to ease navigation. 3 YES The portal includes various presentation mechanisms supporting different categorisation types. A table of content is included in the PDF report.
Hierarchical presentation of the results according to the seriousness of the guidelines 3 YES The results are presented by priority level, which refers to the seriousness of the guidelines. Moreover, the portal offers better reporting mechanisms and categorization of the results.
I could not export results to pdf. 3 YES The results can be successfully presented in a PDF report.
It´s necessary for the application to allow you to check the accessibility problems within the tool. 3 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment.
My suggestion is to offer some sort of highlighting option for problems of the specific page as all most common W3C validators do. Highlighting the error directly in the source and a suggested solution would be very helpful. 3 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment.
Non attractive. 3 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
Spelling errors in the reporting, e.g. 4.1.2 Number of <A> elemenets without "title" attribute. It also seems Guideline 4.1 - Level A makes no sense as it indicated A[attributes={name=header_page}; value=[]] which I could not even find back in the source code. 3 YES The descriptions of the errors as well as the hints have been corrected/improved. The specific result of Success Criterion 4.1.2 includes a hyperlink (<A> element) without "title" attribute.
The problematic element as reported is often not traceable in the source code. Seems it is "restructured" or stripped. 3 YES Improvements have been made in the latest version of WaaT.
The reporting should link directly to the source files where the errors and warning points. For example if there is an error in file a.html line 135, when I click on the error the file a.html should be open with the cursor in the line 135 3 Partially As the WaaT tool does not support direct editing of the problematic elements in the source code (because when evaluating a URL the specific web page/site is stored on a server), it can only report the file and the line where the error appeared.
Too much information provided regarding the warnings (I would prefer a more briefer content). 3 YES The amount of information regarding the warnings (as well as the errors) depends on the content of the examined web page. However, the portal includes better and more attractive presentation mechanisms.
More details about how to fix errors 3 YES The descriptions of the errors as well as the hints have been corrected/improved.
More information and examples are needed 3 YES The descriptions of the errors as well as the hints have been corrected/improved.
A progress bar showing, while the site(s) are assessed with the estimated time till completion based on the number of tags to analyse and how time-consuming they are (if the task should take longer than a few minutes) 3 Partially The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
No "any" checkbox 3 YES Three new checkboxes have been added, allowing to select/deselect all the approaches of priority A, AA and AAA, respectively.
It needs more help tags; multi selection possibilities. 3 YES The descriptions of the errors as well as the hints have been corrected/improved.
Needs refinement in interaction. 3 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
Portal version has a higher setup time through project creation; the urls in a project can't be modified later, there is no link to the project overview site, sometimes got redirected to start page for no reason. Standalone version seems to have trouble interacting with AdobeReader, and the copy/paste functionality of the URL line is not good. No control of which 10 pages are tested; manual copy/paste of source code overrides this. In the portal I don't see where to specify the number of pages to assess 3 YES The PDF report has been improved in the latest version of WaaT.
Some refinements needed. 3 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
Sometimes one of the results does not expand, even when clicking on the "expand all" button, all will expand except one. 3 YES Fixed in the latest version of WaaT.
The actual reporting analysis is quite difficult and tedious to use as one has no concrete overview, and the pdf has no headings included for easy navigation. 3 YES The PDF report has been improved in the latest version of WaaT. It includes overview table with hyperlinks to the errors/warnings for each guideline.
The explanation of the full version is not entirely visible. The icons for the reports also jump depending on being in A, AA or AAA. 3 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
The portal is more user friendly than the standalone version. There should be a "select all" and "deselect all" to facilitate to choose the guidelines. 3 YES The accessibility assessment of web applications will be mainly performed via the portal. The standalone version of WaaT is mainly for debug purposes and can be used by developers as a secondary tool for web assessment. The GUI of the standalone version is not supposed to be fully accessible and attractive.
There is no "any" checkbox when you try to select the WCAG2 Evaluation approaches. 3 YES Fixed. Three new checkboxes have been added, allowing to select/deselect all the approaches of priority A, AA and AAA, respectively.

Position in the theoretical development process

The WaaT evaluation tests have been implemented according to the defined ACCESSIBLE Harmonized methodology (HAM) which is described in deliverable D3.1 "ACCESSIBLE harmonizes methodology".

The Web Accessibility Assessment Tool (Waat) supports the accessibility assessment of Web applications according to WCAG 2.0 and WAI-ARIA implemented tests. The following tables present all the trivial tests that have been implemented in the WaaT, according to the WCAG 2.0 guidelines and the WAI-ARIA.

Implemented tests in WaaT following accurately the test procedure defined by the guidelines of WCAG 2.0

WCAG2 Guideline Success Criterion Technique(s)
1.1 1.1.1 (Level A) H24, H45, H46, H35, H37, H36, H44, H2, H67
1.2.8 (Level AAA) H46
1.3 1.3.1 (Level A) H71, H63, H44, H39, H43, H51, H71, H44, H73, H65
1.4.2 (Level A) G170, G18
1.4.4 (Level AA) C12, C13, C14, C28
1.4.6 (Level AAA) G17, G18
1.4.8 (Level AAA) C21, C24, C12, C13, C14
2.1 2.1.1 (Level A) H91, G90, SCR2, SCR20, G202, H71
2.2.2 (Level A) G11
2.2.4 (Level AAA) G75, G76
2.3 2.3.1 (Level A) G15, G19, G176
2.3.2 (Level AAA) G19
2.4 2.4.1 (Level A) H64, H50
2.4.2 (Level A) G88, H25, H24
2.4.5 (Level AA) G126, H59, G185, G126
2.4.6 (Level AA) G130
2.4.8 (Level AAA) H59
2.4.9 (Level AAA) H30, H24
3.1 3.1.1 (Level A) H57, H54
3.1.4 (Level AAA) H28
3.2 3.2.2 (Level A) G80
3.2.5 (Level AAA) G76, H76, SVR1, H83, G110, SCR24
3.3.2 (Level A) G162, H65, H44, H71, H44, G162
3.3.5 (Level AAA) H89
4.1 4.1.1 (Level A) H93, G134, G192, H94, G134, G192, H74, H75, H88
4.1.2 (Level A) H91, H65, H64, H88, H44

Tests implemented in the WaaT according to the WAI-ARIA specifications

WAI-ARIA Test Implementation details
Check for role attribute presence Identify elements with "role" attribute and value equal to one of the following: "status", "tree", "row", "alertdialog", "navigation", "option", "menuitem", "application", "definition", "list", "tabpanel", "treeitem", "contentinfo", "combobox", "separator", "note", "region", "tablist", "tootltip", "log", "search", "link", "checkbox", "math", "dialog", "heading", "document", "radiogroup", "rowheader", "banner", "img", "group", "progressbar", "main", "marquee", "menubar", "listbox", "presentation", "radio", "treegrid", "directory", "complementary", "button", "menu", "rowgroup", "alert", "toolbar", "grid", "menuitemcheckbox", "menuitemradio", "spinbutton", "tab", "textbox", "scrollbar", "timer", "slider", "gridcell", "form", "columnheader", "listitem", "article"
Check for elements with missing required WAI-ARIA properties Examine the following (based on WAI ARIA roles:
  • Elements with role = spinbutton without attribute aria-valuemax
  • Elements with role = treeitem whose parent is not group or tree
  • Elements with role = tab whose parent is not tablist
  • Elements with role = application including more than one elements with role = banner.
  • Elements with role = row whose parent has not role grid or row group or tree grid
  • Elements with role = document including more than one elements with role = banner.
  • Elements with role = tree whose children are not groups or tree items
  • Elements with role = application missing title and aria-labelledby attributes.
  • Elements with role = option whose parents are not elements with role = combobox or role = listbox or role = menu or role = radiogroup or role = tree
  • Elements with role = progress bar missing aria-valuemax attribute
  • Elements with role = tablist whose children are not tabs
  • Elements with role = radiogroup whose parents are not elements with role = radio
  • Elements with role = combobox not including elements with role = listbox or role = textbox.
  • Elements with role = list whose children are not elements with role = listitem or role = group
  • Elements with role = spinbutton without attribute aria-valuenow
  • Elements with role = navigation missing the aria-labelledby attribute
  • Elements with role = dialog missing aria-label and aria-labelledby attributes.
  • Elements with role = alertdialog whose aria-describedby attribute is missing
  • Images with role = presentation having also non empty alt attribute
  • Elements with role = tabpanel without aria-labelledby attribute and without aria-controls property on the corresponding tab
  • Elements with role = menuitem whose parent has not role = menu or role = menubar
  • Elements with role = checkbox without attribute aria-checked
  • Elements with role = rowgroup whose children are not rows
  • Elements with role = scrollbar without attribute aria-valuenow
  • Elements with role = rowheader whose parent has not role = row
  • Role document incorrectly applied on element other than body.
  • Elements with role = scrollbar without attribute aria-valuemin
  • Elements with role = rowgroup whose parent has not role = grid
  • Elements with role = column header not contained or owned by an element with role = row.
  • Elements with role = content info without aria-labelledby attribute
  • Elements with role = gridcel not owned by elements with role = row
  • Elements with role = complementary without aria-labelledby attribute
  • Elements with role = combobox without attribute aria-autocomplete.
  • Elements with role = application containing more than one elements with role = main
  • Elements with role = slider without attribute aria-valuenow
  • Elements with role = menuitemradio whose parent has not role = menu or role = menubar
  • Elements with role = menuitemcheckbox whose parent has not role = menu or role = menubar
  • Elements with role = spinbutton without attribute aria-valuemin
  • Elements with role = scrollbar without attribute aria-valuemax
  • Elements with role = slider without aria-valuemax attribute
  • Elements with role = menu whose children are not elements with role = group or role = menuitem or role = menuitemcheckbox or role = menuitemradio
  • Elements with role = slider without attribute aria-valuemin
  • Elements with role = progress bar missing aria-valuemin attribute
  • Elements with role = listbox whose children are not elements with role = option
  • Elements with role = radio without aria-checked attribute
  • Elements with role = scrollbar without attribute aria-orientation
  • Elements with role = toolbar missing aria-labelledby property while there are more than one toolbars
  • Elements with role = menuitemradio without aria-checked attribute
  • Elements with role = form missing aria-labelledby attribute
  • Elements with role = row whose children are not column header or gridcell or row header
  • Elements with role = menuitemcheckbox without attribute aria-checked
  • Elements with role = region missing aria-labelledby attribute or having aria-labelledby attribute but not referencing a heading
  • Elements with role = document including more than one element with role = contentinfo.
  • Elements with role = combobox without attribute aria-expanded
  • Elements with role = search without the aria-labelledby attribute
  • Elements with role = document containing more than one elements with role = main
  • Elements with role = treegrid whose children are not rows
  • Elements with role = application including more than one elements with role = content info.
  • Elements with role = scrollbar without attribute aria-controls
  • Elements with role = document missing title and aria-labelledby attributes
  • Role application incorrectly applied on element other than the body.
Check for problems in aria properties Examine the following:
  • Elements with id ref specified in more than one elements’ aria-owns attribute
  • Elements with aria-labelledby property referencing an ID that does not exist
  • Elements with aria-describedby property referencing an ID that does not exist
  • Elements with aria-owns property referencing an ID that does not exist
Check for elements with invalid role attribute Elements with invalid role, which is not one of the following: "alert", "alertdialog", "application", "article", "banner", "button", "checkbox", "columnheader", "combobox", "complementary", "contentinfo", "definition", "dialog", "directory", "document", "form", "grid", "gridcell", "group", "heading", "img", "link", "list", "listbox", "listitem", "log", "main", "marquee", "math", "menu", "menubar", "menuitem", "menuitemcheckbox", "menuitemradio", "navigation", "note", "option", "presentation", "progresbar", "radio", "radiogroup", "region", "row", "rowgroup", "rowheader", "search", "separator", "scrollbar", "slider", "spinbutton", "status", "tab", "tablist", "tabpanel", "textbox", "timer", "toolbar", "tooltip", "tree", "treegrid", "treeitem"

A path to obtain the required quality to integrate the tools in mainstream development environments (e.g. Netbeans, JDeveloper)

The Waat tool has been developed with the usage of open source technologies and open standards and for that reason there is the possibility of integrating the WaaT in existing IDE environments (e.g. NetBeans, JDeveloper). Currently, the AEGIS FP7 EU project has implemented an appropriate WaaT plug-in for the support of the NetBeans IDE environment. This implemented plug-in can be used by interested developers and designers in order to evaluate their Web applications during their software development process.