Do website polls harm conversions?

by David Mannheim

posted on: December 1st 2015

We’ve had experience before where clients ask whether the inclusion of polls or surveys on websites can harm conversion rates.

It’s a natural response. Polls or surveys can often be seen as intrusive and annoying therefore harming the user experience and ultimately conversion rates. Wait! But feedback polls are so unobtrusive, especially those offered by HotJar that they barely get in the way of the user experience, right? Uh…well…I sometimes get responses to questions about how we can improve the site like this:

“Less pop up windows”

“Get rid of this red box”

“Stop these annoying pop up windows”

…or my personal favourite

“Get rid of these fucking popups”.

The fact that someone was commitment and angry enough to tell my insight generation tool to F-off must have meant this guy was really pissed off. That being said the overall benefit of the insight generation tool far outweighs that of one angry user. I don’t need to preach about how much feedback polls can assist us in understanding customer insight - they’re invaluable, especially in facilitating hypothesis generation for A/B testing. In fact, we’ve often used the phrase “it’s like a plaster” with such feedback polls. It might hurt a little bit but the benefit far far far outweighs the disadvantages 20:1.

That being said we’re optimisers. We’re in the unique position where we can test this. So we bundled the HotJar code (our feedback tool of choice) into VWO and ran an A/B test; 50% of users saw the poll and 50% didn’t. Tip: At the time of writing, HotJar don’t quite have the facility to allocate a portion of traffic to view their polls / heat maps etc but it is on the road map.

Irrevocable proof:

In a variation we ran the inclusion of HotJar code was shown to have ~0.20% better conversion rate at a statistical significance of 91% on over 60,000 users.

Please note: results may differ on different websites this was just run on a specific site over a 27 day period.

To back this up, Brian from Backlinko spoke about his experience and commented “NO ONE cared about the nano-second interruption. I’ve had the form on my site for almost a month now and no one has said a word about it” (although talking about an email popup)

The effect of this has rarely been tested. Email subscription popups have been tested previously as they have a similar concept; they’re seen as incredibly annoying but yes, they work. For example, Dan Zarrella tested this experience on his site both with and without a subscription popup. The conclusion he came to is the same as his blog article “email popups work and don’t hurt” where with the popup the subscription rate was 3.08% and without 1.52%.

OK so there might be an element of context in this. I suppose popups can harm the overall user experience which is directly linked with conversion rates but the way in which they are presented and framed to the user is important. For example, it’s seen that using exit intent instead of a direct popup can increase subscriptions by 600%. Considering we’re discussing small, unobtrusive popups this shouldn’t really come into the fold.

O, but it does.

These little popups are still just as annoying despite how unobtrusive and non-committal they are. The context in which they are presented is important and can affect your user experience. Here are some tips to improving, not just your response rate, but also your user experience.

1. Set expectations around the commitment of the user

By using terms like “quick question” we are setting the user’s expectation that this feedback won’t take up much of their time. I wrote an article about this after doing some research for the guys at HotJar about this. Since then, I’ve found that using the term ‘question’ can be improved further by reducing the commitment even further and using the term ‘poll’. From experience, ‘quick poll’ gets a higher response rate than ‘quick question’ at the start of every question yet there is no statistical evidence behind this (yet!)

2. Using radio buttons instead of open text questions

Again, more about reducing the level of commitment required from a user, by offering limited choices we are increasing the response speed required from the user. We are also ensuring they don’t swear at us in an open text field!

3. When the pop up appears is important

The key is not to disrupt the user’s journey and make the popups appear in as least a disruptive was as possible. Using exit intent or ensuring the popup appears after a certain time on the site may reduce overall impressions, but adversely may increase response rates (again no statistical evidence to suggest this).

HotJar also make various recommendations about the best way to use polls in their Action Plan, as does Scott Bolinger.

avatar for author

David Mannheim

David is an experienced conversion optimiser and has worked across a series of core optimisation disciplines including web analytics, user experience and AB & MVT testing.

Do website polls harm conversions?

by David Mannheim Time to read: 3 min