How to set up a realistic testing environment

This method is a true stress-test of a service

Earlier this year, we did “day in the life” usability testing for a service we’re working on. We were inspired by the work that the DWP did on their bereavement service.

We found that observing an end user of a service in a realistic environment is one of the most effective ways to get feedback and insights.

Being able to observe how someone uses a new service to carry out their every day tasks means that you’re not testing in isolation, you’re testing the circumstances that a service or tool operates in. Our “day in the life” testing was for a Ministry of Justice (MoJ) service to help staff record their decisions and outcomes for people on probation.

As it’s a tool designed for staff, we thought this type of testing would be beneficial to how a user’s environment might affect their use of a service. We learned a lot by doing this so we’ve shared some ideas to help others try it.

Decide what your measures of success are

When you’re testing a new service with someone, you need to think about how you’re going to measure its success. Depending on the type of service, you’ll need to consider how you’ll meet the needs of your users. Some of these questions might help:

Our service will replace legacy technology and for our testing, we wanted to observe how intuitive our service is and if it was easier to use than the current system.

Create a similar physical environment

It’s good to think about how your users will interact with your service and what their physical environment might be like. It’s important to think about:

On the MOJ service, the user researchers spent a day shadowing people who worked in probation and this was invaluable to understanding how the physical environment affects someone’s work. One of the observations they made was that probation practitioners get interrupted regularly by phone calls and visitors when they’re in the office. So for our testing, we knew we had to add in interruptions to test how this would affect how they interact with our service.

A member of our team phoned the probation practitioner we were testing with during a task to see how they would respond and how the interruption affected them using our service.

Involve users in creating the testing environment

It was really useful for us to have shadowed our users before we tried to recreate a realistic testing environment. We got to see and were aware of constraints on how they currently work and what a typical day is like for them.

One of the main insights for us was that our group of users often don’t use any devices when they have appointments with people. This means that they often react to certain situations and then record what happened later, but not during meetings.

For our testing, participants were emailed their schedule for the day and used our service to review the details of people on probation they were due to meet. At the appointment time they went to a meeting room where they found a written dialogue of what had been said. This reflects their every day activity and we observed what they would do next with our service and how they wanted to use it (rather than prescribing them tasks).

Using fake but realistic data or scenarios

For our service, we created fake personas for people on probation and we had to think about their lives and what probation practitioners need to know about them. For obvious reasons, we couldn’t use real people and their personal information so we had to create this information from scratch.

We populated our prototype with these details to help the probation practitioners complete the scenarios we created as if they were real people on probation. We had to think about their personal circumstances, employment history, why they were on probation, the conditions of their sentence, how they were engaging with their probation officer, and many other aspects of their lives that probation practitioners need to be aware of.

We also knew that we didn’t want to just test how they completed tasks with our prototype, we wanted to see how they found information on it too. So using realistic scenarios was essential for us to understand how they would use our tool. As this blog post outlines, the real world is complicated, so use real data in your designs.

Think about how to record the testing and/or have observers

Doing testing remotely makes it much easier to record sessions than in-person recording. This is something to consider when you’re thinking about the physical space and where the testing’s going to take place. It’s also worth thinking about other people in the team who want to observe the session and take notes. How can you share what happens during usability testing with others?

For our project, we combined the physical set up with the remote. We limited the people in the physical office to 3 and the participant joined a Teams call which ran for the full session (4 hours). This allowed us to record their interactions with the service and allowed observers from the rest of the team to join for some or part of the session.

Plan properly

We learned loads after doing our first “day in the life” testing and we made improvements for each session that followed. We also spent several weeks planning for the first session. Here’s what we learned when planning our testing:

This method is a true stress-test of a service and provides a low risk and high return way of building confidence before moving into private beta. It bridges the gap between usability testing and launch, surfacing any issues so that they can be dealt with before release. This is especially important when working on a service that could impact vulnerable people or have safety implications.