Milestone: Automatic Accesibility (A11Y) testing

January 18, 2022 — GNU social development team

We use pa11y-ci to run our accesibility testing on all GNU social pages. The process of setting this up was a bit of a ride, but the results are quite useful:

Accesibility report generated by Pa11y

We also take screenshots of all pages and compare them with the previous, allowing us to spot (possibly) unintended changes in the UI.

The way we do this is by using a docker image derived from node, with pa11y-ci installed, which doesn't have an entrypoint. This then allows us to spin up a webserver, PHP and Redis containers, mimiking a staging environment. pa11y-ci uses a JSON config file to tell it what to do, and handles visiting all pages with the given resolution, logging in and registering any accesibility violations found.

Accesibility violations found by Pa11y

Note that most of these errors are actually the same.

The saved screenshots are then used to detect changes, with ImageMagik's compare, to generate a report on visual changes.

Screenshot comparisons

We were already striving for making GNU social accesible, which is clear from the really low number of violations found, but there's always room for improvement :)