How I fell in love with screen reader usability testing

The Intro

Few things in our careers stick out to us and usually, those are the most powerful moments. My first screen reader usability testing sessions are still vivid years later. It was an embarrassingly long period into my career before I was on a project with enough budget and backing for proper accessibility focused usability testing. I was excited and nervous and honestly didn’t really know what to expect. It left enough of an impression that I still think about it and tell the story regularly several years later.

The Build-Up

This project was a long, complicated, and extremely important form. Important enough that just the name of it was anxiety-inducing for users, well the name and the fact it had been such a poor experience for years. Early on we knew WCAG AA was a requirement so my team and I planned screen reader testing from the start.

This was also new to me, up until this point we had used this HTML Code Sniffer and called it a day.

I had never used a screen reader before so that was my first step. Which one should I test with? I ended up bouncing between JAWS and Apple’s Voice Over. On my iMac I covered the screen with paper, our test PC laptop I turned the screen off and started working through the form with keyboard only. Having the commands handy helped a ton with the learning curve. Here are the Apple Voice Over Commands and JAWS version.

This was humbling, it was much more challenging and frustrating that I realized and my bug/issue list was growing. I put together a sizeable list from my testing, shared with the team so we could fix the glaring issues, then asked our client to help schedule sessions with a few screen reader users.

The Sessions

These usability testing sessions were eye-opening, exciting, humbling, and I had that glowing buzz feeling the rest of the week.

Comfort zone

It’s worth noting that this put me out of my comfort zone. Am I going to look clueless? Will they be frustrated with me? Can I help them be relaxed and comfortable?

The cliche follow up is something along the lines of “true growth comes when you are out of your comfort zone”. That cliche is 100% true in this case.

Challenging all my assumptions

I quickly realized 70% of my assumed issues were nonissues and based on my lack of knowledge experience with screen readers.

It was immediately obvious to me they were on a whole different level with their tools. The most obvious one was the speech rate (the speed the screen reader speaks the text). It was blazingly fast and hard for me to follow at first. The speed they were skimming a page was impressive.

Their reactions to participating

After my standard session intro and profusely thanking them for their time, I was ready to dive into the session.

Before I could each participant stopped us to express their gratitude, specifically for working with them on this particular project but also in general. For all of them, this was the first time they had been invited to a usability testing session. Again it was eye-opening. This tiny slice is part of a bigger portion of users that for a long time have felt like their experience isn’t as important as others.

Here are a few quick tips on setup

• Lean on your client to identify participants with the organization
• If this is for the public lean on your network, friends, etc. You will find people much easier than you think
• Have multiple note-takers. Don’t go crazy with this one but more than one observer/note-taker helped me catch a lot more than I would have (see again the point about speech rate speed)
• Record — Another obvious one but if they give you permission recordings are extra helpful with screen reader usability testing.
• Don’t be afraid to ask for explanations. Getting clarification at the moment helped a lot. Don’t get carried away with this and make sure to keep the session flowing after the explanation.

The Learnings

Contextual aria labels

Contextual aria labels are important. Picture an interface with two sections where you can add a new item. One section is for references and the other is for previous employers. If both aria labels read “Add new” that isn’t too helpful.

Always give these context, “Add a new reference”. Your aria labels can be more specific than your button text

Modal windows can be tricky.

None of our modal windows were set to take the focus when active. This was painfully obvious when users completely missed them and continued through the form.

Shift the focus to these modals when they are active.

Contextual error/warning messages need extra attention.

Similar to modals our error messages were only relying on eyesight to tell users there was a problem. This was extremely frustrating to one participant who was stopped from moving forward to the next form section but had no idea why. Only after they skimmed the page again did they see the errors.

Shift the focus to these errors as soon as they occur.

Think about the different user flow

Screen reader users skim pages differently. Differently from us and each other. Some will just skip to links, some will skim headlines first, some will read every piece of text on the screen.

Make sure your interface supports all these different skimming methods.

Test on your own

It’s so easy and quick to do, cover your screen, and get hands-on with the screen reader of your choice.

The Feedback

Presenting the report was eye-opening all over again. We shared this with others in the company who weren’t on this project, my immediate team, and to stakeholders in a sprint review.

Most people were interested. Some people were fascinated and just as excited as I was. A small handful was mildly entertained to not interested at all. That group of fascinated folks (myself included) continued learning more and have successfully gotten others excited about accessibility in other projects.

Excitement perspective

My biggest take away was reframing how I think about accessibility. Up until this experience years ago, accessibility was just a requirement of some projects. After this experience, it was been an exciting, fun, and integral part of every project I work on.

Addendum

I started this article months ago then set it aside. “No one is going to care about this,” I thought. Then a friend/fellow designer called me on one of our regular catch up chats. He was so excited to tell me about a usability testing session he did with a couple of screen reader users.

The similarities between our stories were oddly amazing and he was just as excited. I could tell that just like my experience this had reframed his perspective and was going to shift his workflow in the future.

“Well maybe at least two people will want to read this,” I thought when I decided to finish the article.