Thoughts on keeping user research going in difficult times

Three shapes from dxw brand balanced on each other

We all agreed that it’s more important than ever that we work in an evidence based and user centred way

Last week, I hosted a panel discussion with others working in UK public services to share what we’re all doing to keep user research going in these difficult times.

On the panel we had:

And around 120 people joined the session online.

I want to thank the panelists for their thoughtful contributions, and everyone who joined for and contributed to the discussion.

In this blog post, I’ve collected together the different things we discussed and some conclusions we reached.

But I’ll start with links to the resources that people are sharing.

Useful resources

Dr Salma Patel, Head of User Research at Ofsted, and Samatha Davies, Head of User Research at Monzo Bank, have both shared their notes from the discussion. Contact them on Twitter for access.

Jonathan Nicholls and the team at GDS are preparing guidance for the user research section of the Service Manual. We’ll update this blog post to reference that guidance once it’s published.

FutureGov has published an excellent guide to making design research work remotely.

Scottish Government are crowdsourcing and co-producing a guide to doing user centred design work remotely.

Supporting rapid changes to guidance and services

An obvious question was what user research we should be focussing on.

Many of our organisations are making significant and rapid changes to existing guidance and transactions, and even creating and rolling out completely new services.

We all agreed that at this time it’s more important than ever that we work in an evidence based and user centred way. It’s vital that our services work for everyone who needs them, and that we don’t make mistakes that might harm people who are coming to us for help.

Everyone is reviewing their priorities and considering the risks of doing and not doing user research in each case. In some cases that risk will be high, and we have to find a way to continue doing effective research, often very quickly. In other cases the risk will be much lower and we can learn more later.

We also discussed the need to consider the potential burden on hard pressed colleagues, and on participants who’ll have many other more important concerns. Although where changes are clearly important, we’re finding that colleagues and the public do understand the value of our research, and are willing to give their time.

We agreed that we need to continue working closely with colleagues, with clinicians in the NHS, and with social researchers and policy analysts in education. We’re going to have to do some really rapid research. And we need to triangulate data so we can be more confident about our decisions, to make sure what we do is clinically and educationally sound.

As services come under new pressures, we may need to quickly investigate problems in existing services caused by changes in demand, as lots of people switch to accessing a service online for the first time. And where we’ve needed to make rapid changes, we must make sure we can monitor the effectiveness of those changes to see how well they’re working for everyone.

Some organisations may be working with user groups that they’ve had little engagement with before, for example providing teaching advice and resources directly to parents who are educating their children at home. This may become the focus for new research.

Using additional time and space to improve our knowledge and our practices

We also noted that some organisations, like the courts, will have suspended some of their activities. And this may create opportunities to conduct research with colleagues who are normally very busy and difficult to get time with.

In those organisations we may have time to bring colleagues back into the problem space, to reflect more on people’s circumstances and context, and how they might want to experience the service.

We agreed that this can also be a time to focus on knowledge management. Consolidating and sharing things we’ve already learned about different user groups, about their circumstances, needs, common challenges, and barriers. And making sure we have good catalogues of findings from previous work.

We can work to make sure that this knowledge is more visible and accessible, and share it with new staff and with colleagues across our organisations.

There can also be an opportunity to improve our research operations, to improve and document our practices and internal resources.

Many of us are setting up additional video learning and sharing sessions to make sure that our communities continue to develop and connect with each other.

Looking after participants

We discussed ways that we need to take care of our participants.

Many will be experiencing significant emotional and practical stress and strain. Some will be ill, or have family members or housemates who are ill. Some will have been isolated on their own for some time and will just want to chat.

This will impact how we do the research and the results we’ll get. And we’ll need to carefully consider the power dynamics between the researcher and the participants.

More than ever, we need to respect our participants’ time. Is now the right time to be talking to people, especially on a subject that might be a burden for them? What’s the risk of getting this wrong? We don’t want to put someone in a worse position by spending time with us, that they could be spending elsewhere.

There will be additional ethical and safe-guarding challenges to work through. Particularly with participants who lack the privacy they would get from a face to face research session in a separate, neutral space.

We’ll need to check in more with people. To make sure they are really okay to go ahead with the research and don’t feel like they are forced to continue. They might feel they have to, or be more financially driven by getting an incentive than usual.

We’ll need to put more effort into establishing the context of the research. To explain to participants the value of the research we’re doing, even with everything else that’s going on.

And that will take more time. So we’ll need to recalibrate how much we expect to get from research sessions.

We also need to be honest with our participants that there’s a risk the research will be disrupted by problems at our end, with children for example. And that we understand that they may have the same issues.

Looking after ourselves

We will also need to look after ourselves and be kind to ourselves.

Doing remote research is hard for those of us who share homes with several other people, who are also trying to work from home. And is particularly hard for researchers who are balancing child care with remote working – often working ‘shifts’ with a partner.

We need to be honest with participants and colleagues about what’s going on. For example, that we are using some less than perfect equipment or tools, and that the 3 year old might pop in.

We know that some colleagues are feeling the pressure to show that they are ‘present’  and working. We need to let ourselves off the hook.

As managers and leaders, we need to give people permission to take breaks, take exercise, take care of themselves. To show understanding that we aren’t all working in ideal conditions with perfect setups. To accept that we may be less productive than usual.

To help stay in touch, many of us are having more regular catch ups and virtual coffees and lunches for user researchers.

Adapting our practices

We also looked at how we need to adapt our practices when we can’t work with participants or colleagues face to face. When we know that going and seeing the lived reality of peoples’ lives and work is so important.

We all agreed that it’s important not to try to simply replicate what you were doing before, but somehow making it all remote.

We should step back to our objectives, what are we trying to learn with a particular piece of research, what are we trying to achieve in a workshop. Then look for the sequence of activities and the supporting tools that will produce the best result.

And we should set the expectation that things won’t be the same as they have been. Research might take longer, and methods won’t be perfect, and that’s understandable given the circumstances.

We also need to take care when rushing to pick up and use new tools. Especially ones that we and our colleagues are not familiar with. In particular, for user research, what are the data and privacy implications of using the new tool?

Wherever possible, we should adapt existing tools that colleagues know how to use and have already had data privacy impact assessments. For example, using a shared document in Office 365 or Google GSuite for collaborative analysis.

There is also an opportunity here too. We are being forced to think really carefully about what we’re doing, and may come out of this as better researchers.

But we also need to think about how this will look after several weeks when the novelty has worn off and we’re all struggling with continuing lock down.

We all recognised how valuable our service manuals, design systems, prototype kits, and communities of practice have been in helping us to make changes to our services and to our practices at pace.

Doing research remotely

We’ll obviously need to switch almost all our research to happen remotely, and to do that rapidly.

We know that some kinds of research are easier to do remotely than others, for example doing moderated usability testing using a video meeting.

And we know that some organisations are better prepared and more familiar than others for remote research and remote working general. We want to make sure those organisations are not left behind.

Many of us are running remote research training and practice sessions to help upskill people and to give them confidence.

Several people have been working on guidance – see the useful resources at the top of this blog post.

Of particular interest is safeguarding for both the researcher and the participant. For example, what are we showing as a researcher (what’s around us on a video call) and what influence does that have on the research.

And how are we supposed to be interacting with each other? What is polite? If we are doing research with people who are disadvantaged, what do we need to consider?

We also need to be careful with the tools we’re using and issues with privacy and data protection. Yes, we need to respond as quickly as possible, but we need to be thinking about potential longer term problems.

And don’t forget the humble telephone call. We can all get a bit obsessed with trying to get an HD video call to work, when we can get a lot done on a regular phone call.

Keeping research inclusive when we can’t visit or see people face to face

We are all concerned about how we keep our research inclusive when we know that we reach many people through visits and face to face approaches.

How do we keep doing research with people who might not have access to the internet or to an appropriate device. For example, delivering educational content to children through the BBC Sounds app might seem like a good idea, but children might not be able to consume it.

For some things like accessibility testing we may be able to rely more on a combination of expert audits and remote usability testing. But there’s a risk that we focus on the ‘mechanical’ usability of the screens, and less on the wider context of use, which often creates more barriers and is more relevant to the decisions we make.

We discussed different possibilities for continuing research with the help of other people in a household. When we do research we often ask participants if there’s anyone they would like to support them. We can work through the remote equivalent of this. For example, working with carers and local grassroots organisations who are supporting people.

And we may be able to work with proxies, people who have experience of working with people with particular circumstances or needs. It may not be ideal, but may be the best we can do at the moment.

We will have to acknowledge the research debt we may be building up, and make sure we don’t forget to deal with it in future.