Odd A/B Test Results

Posted by Matt on Thu, Mar 03 2011

A few months ago I got a Grasshopper 800 number. At the time I just stuck it in the footer. A week ago I decided to make the number more prominent, figuring it would help earn the trust of new users. I was going to just make the change, but decided to A/B test it for kicks. Turns out the change wasn't very good. Anyone care to take a stab at explaining these results?

 

  Control Variant
     
Impressions 1310 1310
Conversions 119 92
Conversion Perc 9.08% 7.02%
Posted in Startup

12 Comments

Basti said on Mar 03, 2011
I would guess the number should generally make people trust you more, but in this case it lessens the determination of clicking the button by making people think if they might have any objections to signing up. As a consequence they come up with something they might want to ask you before signing up.

Basti
(would have put a name/email in if i could have)
Reply
Matt said on Mar 03, 2011
Thanks for the comment...The name thing is a caching issue I haven't figured out yet.
Mike Greenberg said on Mar 03, 2011
I would consider moving the phone number to a different part of the page. I agree with Basti that it might be distracting them from the Call to Action here. Would be curious on the results comparing "no-number near button" with "a number at the top of the page but still not near the button".
Reply
Matt said on Mar 03, 2011
Added this to my testing queue.
Marc Grabanski said on Mar 03, 2011
People want to know that you are there to support them BEFORE they decide to sign up, not after LOL.
Reply
Jeremy said on Mar 03, 2011
For me, having the number forces me to reconsider a "I'll just sign up and try it." to an, "Hm, others must have had questions so maybe I should research this more." It's almost like putting the number there takes the impulse away.
Reply
Matt said on Mar 03, 2011
Great point. Thanks.
Copongcopong said on Mar 04, 2011
The toll-free number makes the customer to stop signing up immediately (clicking the button). What stop them is the fact that they can call first before signing up, which most of the time, they are too busy to call your toll free. And forgetting to get back to the site and signup.
Reply
ob said on Mar 04, 2011
with that number the site looks more serious and less annonymous -> service more intimidating. Parhaps it makes people think, they are getting into something that's not that easy to quit. Phone number implies there are sales people on the other side.
Reply
Stephen said on Mar 15, 2011
this is exactly what I was thinking too...
Alexander said on Mar 05, 2011
"If they will let me call them, maybe they will want to call me. I don't want them to call me!"

Or, maybe it's the "sales people are involved" bit.

In any case, I think it should be taken away from signup since it isn't naturally a part of that process.
Reply
Aaron said on Aug 26, 2011
They called instead of clicked the ad?