Moderated User Testing Newbie: What I learnt

by Olivia Gruetter

posted on: April 14th 2018

Last month I completed my first moderated user test since finishing university in September, as well as my very first test that were run in a user lab for Mamas and Papas. Here's a summary of what we learnt.

USER TESTING NEWBIE

Before this, I had only done guerrilla moderated user tests for my master’s dissertation. Armed with a backpack filled with notes, papers, a laptop, video camera, and tape recorder, I had rushed between participants, spending hours on various forms of public transport in order to get them all done.

When there, I spent the time one on one with participants, holding the video camera, making sure all the recordings were working properly (sometimes they weren’t), while trying to take the participants through the tasks, and make mental notes of interesting actions and behaviours. Despite this, and despite the days that followed I spent transcribing 12 hours’ worth of tests, it was one of the most enjoyable and useful experiences I had on the entire course.

Needless to say, I was extremely excited to have the opportunity to run moderated user tests in a real lab as a team member of User Conversion, with all the equipment I had been wishing for just a few months before.

Throughout the process of organising the sessions beforehand, being the moderator in the room with participants, taking notes in the observation room, and analysing the findings both myself and in group sessions, I was able to experience the tests from many different angles.

From this, I learned a lot about what worked well, and a lot more about what I would do differently when I ran more moderated user tests in the future.

I wanted to share those tips with you as a moderated user testing newbie.

WHO TO TEST WITH

One of the first things I learnt on the lead up to the tests is the importance of considering every aspect of recruitment to make sure that you get exactly the right participants for the site you are testing.

We hadn’t considered how small differences in participants circumstances could alter the way that they look for different items, and their general shopping behaviour.

For these sessions specifically, we assumed that any mother who fit into the category of pregnant, or recently having given birth to their first child, would have similar shopping habits. What we found, however, was that even after having only recently given birth and having bought some higher priced items (e.g. car seats, pushchairs, furniture) for their baby already, their process of selecting items and shopping was completely different.

We also hadn’t taken into account users’ budgets, and the fact that not all users shop for new items on retail sites, as some instead look for second-hand items which can be bought for lower prices or swapped for other items.

TIPS

  • Really think about who your target audience is – be very specific
  • Be picky when selecting participants
  • Try and select users who would usually be able to afford to buy items from the site you are testing

AIMS AND TASKS

During testing and analysis, it became clear that to get useful data it’s crucial to know exactly what you are aiming to find during these sessions.

We had created tasks that were centred around three main areas which were left quite broad. This was a good place to start, but we found that as we began running the sessions it was important to adapt the tasks to the individuals we were testing with and change the way we ran them as we progressed.

For example, we gained some interesting insights from first letting some users navigate the site themselves, without having a specific task in mind, as it allowed us to see their natural browsing behaviours. It also became clear that users acted in a much less natural way when they had already heavily researched or previously purchased the items we were prompting them to look at.

TIPS

  • Have very specific aims in mind, and base your tasks around these aims
  • Rehearse the sessions multiple times, until you get the tasks right
  • If possible, rehearse the sessions with people who match your desired participants as closely as possible
  • Leave time and room for adaptation based on individual participants

PRACTICE

I found that moderation is a particularly difficult skill to learn, and getting used to it requires a great deal of practice.

It’s not easy to stop yourself telling the participant exactly what to do, especially as you feel like helping them, or asking them accidentally leading questions that will confirm a thought you have had yourself, or even shutting off further discussion completely with a ‘yes or no’ question which doesn’t gain any interesting insights.

For someone like myself, who has terrible memory when under pressure, I immediately forget any useful information for making the participant feel more comfortable with me; their name, their baby’s name, their likes, and their dislikes.

In future, it would be useful to have a list of basic information about the participant in front of me (name, age, circumstance etc.), which could be updated by those watching from the observation room as the participant shares more personal information.

TIPS

  • Practice, practice, practice!
  • Be careful with leading and yes/no questions
  • Have notes visible throughout the session

NOTE TAKING

Note taking during the sessions could be even harder than moderating the sessions themselves.

Keeping a note of everything that is happening requires a lot of concentration, and some very speedy writing skills!

I initially put my focus on the moderator and user, but soon realised that I should be concentrating on their screen more, so I could see what was happening on the site. It wouldn’t be easy for one person to catch every possible insight during a session, so having more than one note taker present is preferable.

During the sessions, we had agreed to make note of the observation, a time stamp, and how happy the participant was feeling (in the form of a quick drawing of a sad, neutral, or happy face), but we found that during the analysis, we were missing some very important information – where the user was on the website.

In future, I would try to ensure that at a minimum each post-it note showed a time stamp, a face showing the participant’s happiness, a note of their position on the site, as well as the actual observation.

It would also be much easier to have enough colours of post-it notes that one could be used for each participant, as during these sessions we had to use the same colour for a few participants, which made the note taking during the session and analysis afterwards a bit trickier.

TIPS

  • Make sure each post-it note contains a time stamp, a face showing the participants happiness, a note of their position on the site, and the observation
  • Use a different coloured post-it note for each participant
  • Concentrate on the screen, and not on the participant
  • Have more than one note taker where possible

THE MORE THE MERRIER

In all aspects of these session I found that the more people present, the better.

Having more than one moderator allows the sessions to be split between multiple people, giving them each a rest and change of scene throughout the day. This is especially important when sessions are running back-to-back for a full day, with little time in between.

The same was true for note takers, as when there were multiple people present taking notes, we got much richer insights from them than when there was only one. Having a range of people with different backgrounds was also helpful, as each person concentrates on different areas of the session, and notices different things.

TIPS

  • Where possible, have more than moderator
  • Get everyone involved in the sessions who may benefit from them
  • Get a group of people with mixed backgrounds to observe and take notes, as they will bring different viewpoints and expertise

IMMEDIATE ANALYSIS

Analysing moderated user tests takes a very long time. Whatever amount of time you plan in to get it done, it will probably take longer than that.

We conducted the sessions for a full day on Thursday and on Friday morning. This left us some time on Friday afternoon to do some initial analysis, but it wasn’t enough time to properly get started.

We then had to wait two days over the weekend before analysis could properly start again, which meant that the sessions were not as fresh in our minds as they could have been.

I think ideally it would be best to leave time at the end of each day to have a brainstorm with everyone present, and make some initial notes so that nothing is forgotten. If possible, the sessions would also take place earlier in the week, which would allow the analysis to begin without a couple of days’ break. This would likely limit what was forgotten or missed over time.

TIPS

  • Perform the analysis straight away, not after the weekend
  • Leave some time free at the end of the day (if it is running over multiple days) to discuss and write up notes about the sessions
  • Have lots of people present in the observation room

MORE SPACE, LESS NOISE

Analysis of the notes after the sessions have ended takes up loads and loads of physical space. You need to be able to spread your analysis out as much as possible, otherwise it’s far harder to analyse.

For these 9 sessions, I made use of a whole wall of the office, but still found that by the last few sets of post-it notes I was running out of space and ended up having to put post-it notes on top of each other.

This makes rearranging them later on much more difficult. This mess of post-it notes all over each other also makes it hard to show others what your thoughts are and limits the group brainstorming sessions that are possible.

Having a space where you can work without too much noise and distraction may also be beneficial, though this may not be possible in every office as you’d need to have the space free for several days.

A good tip I picked up (at my own expense) when choosing a space that was not to do the analysis above a radiator that will blow the post-it notes off when turned on, or near an open window with a breeze coming in!

This will mean a lot of picking up post-it notes from the floor and trying to figure out where they were supposed to be. Also, make sure you buy the best quality post-it notes that are available, otherwise you will have to watch, as I did every day, your post-it notes fall down while you are trying to finish the analysis.

TIPS

  • Find a room or area where you can do the analysis undisturbed
  • Try and make sure the room isn’t too breezy, and the analysis isn’t done directly above a radiator or next to a window
  • Make sure you have a massive wall or whiteboard to keep all your notes and thoughts on
  • Make sure you have good quality post it notes, otherwise they will fall off the wall

IT'S GREAT!

Moderated user tests can be a great way to get a lot of rich qualitative data over a short period of time, and with a small sample of participants.

For example, we completed a total of 9 sessions over two days, with a nice variety of participants, which allowed us to get plenty of rich qualitative data in a short period of time.

This lead to a lot of insights into user behaviour, and over 60 potential test ideas for Mamas and Papas.

It’s also a fantastic opportunity to get a mixture of stakeholders involved, as well as showing those who wouldn’t usually be involved in research how valuable it can be. We were fortunate enough to have the chance for lots of our team members to come along and experience moderated user testing themselves (and help us out with some note taking!), as well as have members of the company whose site we were testing come along and see users on their site first-hand.

Overall it was a brilliant opportunity to both practice user testing skills, and learn new ones.

avatar for author

Olivia Gruetter

Moderated User Testing Newbie: What I learnt

by Olivia Gruetter Time to read: <1 min