ASSISTments Accessibility Audit

Summary

This case study covers an accessibility audit completed on the student facing product at ASSISTments. I cover how I developed each test case against the WCAG criteria, the audit itself, and the resulting accessibility conformance report (ACR). Finally I discuss the steps we took to fix the shortcomings testing found and how we advocated for the importance of accessibility to the larger organization.

Introduction

ASSISTments is an math ed-tech non-profit which supports students’ understanding of math and gives teachers detailed reports on where students are struggling. They are a grant-funded and as a result of recently passed legislation, they need to be accessible for students in order to be adopted by school district partners. I led this project, which included a team of designers and software testing engineers.

The goals of this project were to:

  1. Conduct a complete accessibility audit of our student-facing product

  2. Complete a publicly available Voluntary Product Accessibility Template (VPAT).

  3. Develop a roadmap to remediate critical accessibility gaps.

  4. Educate our internal team on the importance of implementing changes to make our product fully accessible.

Task Development

Defining Use Cases

Leading the Team

After analyzing the success criteria and writing the tasks, I began to assemble my team. My team consisted of a team of designers and a software testing engineer. We began to learn to use the assistive technologies we needed to conduct the tests and divided the task list. As we divided up the task list, it was important for me to have checks and balances for each task, so often more than one person did the same tasks. Doing this helped drive discussion and ensured we were in agreement about the outcomes of each of the tests. The responsibilities of the team were:

  1. Conduct and video themselves doing each of their assigned tasks

  2. Discuss and agree on the outcome of each of the tests

  3. Help lead a company-wide seminar on web accessibility

After analyzing the guidelines, I created a list of 90 tasks covering all ways a typical student might use the product, including unreleased tasks I’d need to test later. Many of the tasks I developed could test multiple success criteria. I then grouped these tasks by personas of users with different disabilities. While developing these personas, I also learned to use assistive technologies like screen readers to understand how they access the internet.

WCAG Standards Analysis

I analyzed the WCAG criteria for both A (basic accessibility) and AA (industry standard) levels. Through this, I learned the requirements and their intent. This helped me confidently report issues to engineers, ensuring we stayed focused on the purpose of the standard, not just the details. I also reviewed the tests, techniques, and exceptions for each requirement.

Accessibility Audit

Testing Process

To complete the tests, I used different sets of problems to ensure accessibility of our site as a whole and to not limit our audit to only code-related issues—because many guidelines deal with the way the user interacts with content. As I documented each standard, I tagged each department in the company who would need to address these issues. This helped me later as I prepared a company-wide session on addressing accessibility issues in our product.

As I conducted each test, I made sure to video the test. This allowed me to:

  1. Test in as “real” a way as possible

  2. Write Jira tickets to include examples and WCAG source criteria

  3. Design solutions (when necessary) to solve any accessibility issue found

I made the design decision to keep the current experience the same as much as possible. I did this to not disrupt our current users and to minimize the development time for proposed solutions like the one pictured below.

Right place, Right time

When I was in the middle of testing, a colleague forwarded me a bug from one of our users who specifically works with students with vision impairments. I took this opportunity to speak to this user about their issue and our in-progress accessibility efforts. We spoke at length about current issues their students faced and and they sent me several videos of their students using the software—this helped to validate our internal testing and gave us real-world examples to share at a company-wide accessibility training we were prepared to do.

Report on Findings

Accessibility Conformance Report

After I conducted the testing, I met and discussed the results of our testing. We did this to ensure we covered as much content as possible and to cross-reference each other’s findings. I wrote the ACR or Voluntary Product Accessibility Template (VPAT) based on our testing and worked with my manager on the language for the published report.

Engineering Roadmap

After the report and testing was finished, I wrote JIRA tickets to serve as a roadmap for our engineering team to fix critical accessibility issues. Along with writing the tickets, I designed solutions when I needed to that would appropriately address the accessibility issue to meet the WCAG success criterion.

DEI Seminar

After our report was published, My team and I led a company-wide seminar on our findings. The purpose of the presentation was to build on the momentum our report had generated and to build empathy for users with disabilities.

Lessons Learned

This was a very different project from many of my other day-to-day tasks. I had to not only develop the testing tasks, but had to lead the team to carry out those tests. A big lesson I learned was to trust my team to do the job. I also learned that advocating for accessibility in product is something that doesn’t just happen once, it is a constant battle with internal stakeholders to be sure our product remains accessible.