Odd A/B Test Results
A few months ago I got a Grasshopper 800 number. At the time I just stuck it in the footer. A week ago I decided to make the number more prominent, figuring it would help earn the trust of new users. I was going to just make the change, but decided to A/B test it for kicks. Turns out the change wasn't very good. Anyone care to take a stab at explaining these results?
| Control | Variant | |
|---|---|---|
![]() |
![]() |
|
| Impressions | 1310 | 1310 |
| Conversions | 119 | 92 |
| Conversion Perc | 9.08% | 7.02% |
Remembering Site A Month 5 Years Later
On March 1st, 2006 I started my first blog. It was called Site A Month (siteamonth.com - domain spam now). The goal was to produce a new "site" each month. Some of the projects didn't end up as actual sites, so the name ended up not being that great. The goal was to try new technologies, so not actually producing a site for a month didn't mean failure.
All the posts have been moved to this blog. You can see them if you scroll back far enough in the archives. They are truly awful, so don't do it. Here's the first post which covers the "Mission Statement".
Rules
- I had a calendar month to design, build, and launch the project.
- I had to use at least one new technology for each site.
- I didn't want to have to manage/contribute to the site when I finished, so it had to be low maintenance once launched.
March 2006 - RSStalker.com
Status
Idea
The originally idea was a tool where a user could enter an Amazon product id and it would generate an RSS feed for price changes. Halfway through the month I expanded the scope to track an entire wishlist.
Lasting Memory
I meant the name to be a witty combination of "RSS" and "Stalker". Everyone called thought it was supposed to be "RSS Talker."
My first time working with
Ajax, RSS, Amazon's Web Services, Javascript effects, Apache Mod Rewrite
April 2006 - RSStalker.com Widget
Status
Unknown. The widget is still out there, but I have no idea if it works or if Yahoo Widgets still exist.
Idea
To some using RSS was a little too complicated, so I made a version that could run from the desktop using Yahoo! Widgets. It had a frontend for creating feeds and showed the latest updates.
Lasting Memory
Nothing special.
My first time working with
Yahoo's Widget Engine
May 2006 - Stadium Traveler
Status
Gone
Idea
A site where users could post "insider" tips for attending sporting events. Best places to park, drink...
Lasting Memory
I learned to never do a user generated content site again. This was also my first ever CakePHP project.
My first time working with
CakePHP
June 2006 - Site A Month redsign
Status
Gone
Idea
When I launched the blog I used some generic theme. I wanted to create a custom one.
Lasting Memory
I suck as design.
My first time working with
Wordpress Plugins
July 2006 - Nothing.
Status
Gone
Idea
I took the month off.
Lasting Memory
I was exhausted.
August 2006 - Shoot4It.com
Status
Never finished.
Idea
A browser-based multi-player ajax rocks, scissor, paper game.
Lasting Memory
I failed because this one was hard to code in small chunks. At the time I'd get like 15 minutes a day to work on these. I only spent half the month on this so that gave me more time for the next project.
August, September 2006 - FantasyLifeGame.com (later FantasyLife.net)
Status
Gone.
Idea
I loved (and still do to some extent) this concept. My friends used to talk about keeping a giant scoreboard on the wall (I think they even did this at one point) where you would get points for doing stuff in real life, such as getting a date, a raise or writing an A paper. The site was a way for groups to define goals and give point values, then compete with their friends.
Lasting Memory
I thought this one would make millions.
My first time working with
I think I cheated here. I don't really remember working with anything new.
October 2006 - The Netflix Prize
Status
No output.
Idea
The Netflix Prize was a contest to see if you could generate better movie rating predictions then Netflix's in house algortihm. The first person/team that did 10% better would win $1M.
Lasting Memory
Working with large datasets on a shitty computer sucks.
My first time working with
Math
November 2006 - DealCrew.com
Status
Gone.
Idea
This was an existing site of mine. It aggregated coupon codes for various online stores. I scrapped the version that was there and made a simpler OriginalSignal/PopURLS style site.
Lasting Memory
My friend wanted to make this a group deals site with a local focus. I told him that idea was retarded.
My first time working with
It's pretty much impossible to build a meaningful site and learn a new technology in a month. I'd pretty much given up on the new technology thing at this point. I figured I was learning a lot anyway even if I didn't new a new technology.
December 2006 - PseudoCoder.com
Status
Rose up and slain it's creator and before going on to world domination (ok, that second part is still to come).
Idea
A blog that profiled bootrapped startups. Like TechCruch before it started slurping the Facebook/Apple serum. Here's the mission statement and the form to submit a site.
Lasting Memory
I met some cool people and was inspired by their projects. Amazingly most the sites I profiled are still online: 88 Miles (original post), 16bugs (original post), AdGridwork (original post)
My first time working with
Interviews.
January, February, March 2007 - CamXip.com
Status
Gone.
Idea
YouTube for IP Cameras. I really liked this one a lot, which is why it ended up dragging on for three months. I even had some partners and paid for professional design.
Lasting Memory
Using Google to finding some crazy IP cameras that were just left open.
My first time working with
Flash
Deprecated
On June 10th, 2007 Site A Month was officially done. I kept the site up for awhile, but eventually I moved the content over and shut down the domain.
Supporting Over 2000 Customers On The Side
One of the questions I'm frequently asked is how I support 2000 paying customers for my business while working a full time job. In general it actually doesn't take much time. During my busy season (Aug to Oct) I get around 15 emails a day. It can be a bit overwhelming. The rest of the year it's closer to 1-2 emails a day. Here are some of the tactics I use.
My System
Most of the support requests come in over email. I don't use a service to handle or track these other than Gmail. My system is to leave an email in the inbox and archive it once I've responded. The "Send & Archive" button, which can be turned on in Gmail Labs, is helpful here.
Response Time
I try to reply to emails as soon as possible, but that can be tricky when I'm at work. Pretty much anytime I have a few minutes I'll check my phone and respond if it's something I know off the top of my head.
Swype makes this a ton easier. With Swype I can bang out an email one handed while walking to a meeting or waiting for the elevator.
If the email is something I need to look up or fix I generally have to wait until lunch or when I get home. I don't think I've ever received a complaint about my response time. In fact people seem to be impressed when I reply late at night or on the weekends.
Phone
Get One NOW!
I used to get emails asking for phone support and would think "Are you crazy? You want phone support for a $20 a year product?". I should have been thinking "Awesome I can get a real customer on the phone and not only help them, but use it as an opportunity to learn what they value and how I can grown my product. I should be paying them $20."
Grasshopper
I'm using Grasshopper to handle the 800 number. Don't even waste time thinking about this, just get one. It's dirt cheap, no commitment and absolutely priceless when it comes to connecting with your customers.
Who Calls
Although most of the calls come from individual customers with miscellaneous questions there is a certain type of user who always calls. I'm talking about school systems who ask things like "I have a ton of teachers and we want to get a group account and pay you lots of money. How do we set this up?" These calls are awesome.
Frequency
Right now (the slow season) I get around 3-4 calls a week. It will be interesting to see what happens during the busy season, especially since I'm running an A/B test right now that will likely result in the 800 number being more prominent on the landing page. More on this in a future post - make sure to sign up for the feed or follow me on Twitter.
My System
I can only get to maybe 10% of the calls when they come in. This kills me, but there isn't much I can do since most of the call come during the day when I'm at my job.
The caller will leave a message, and I'll try to call them back when I get a moment and it will lead to an endless game of phone tag.
I'm trying to get to more of the calls "live". I have Grasshopper ring my home number for 10 seconds, then switch to my mobile for 10 seconds. After that it goes to voicemail.
My Secret Weapon
I have an unfair advantage here as well. My wife is a teacher and user #1 of the service. She's also on maternity leave. If the baby is napping she can pickup. If the caller leaves a voicemail she can call them back much quicker then I can.
Questions?
Feel free to leave any questions in the comments or email me at matt@pseudocoder.com.
Where My Customers Came From in 2010
In previous posts I've shown how much I revenued last year and how much was a result of AdWords spending. Since AdWords only accounted for 23% of revenue we need to figure out where the rest came from.
Organic
Search engine traffic accounts for a big portion of the remaining revenue. I'll break it up into two groups. One for searches from people who were searching for the site by name and one for people searching by keywords.
| Type | Orders | Revenue |
|---|---|---|
| By Name | 185 | $3700 |
| By Keyword | 516 |
$10,320 |
| Total | 701 | $14,020 |
By Search Engine
For fun here's the numbers broken down by search engine.
| Search Engine | Orders | Revenue |
|---|---|---|
| Google |
579 | $11,580 |
| Yahoo | 54 |
$1080 |
| Bing | 42 | $840 |
| Other (AOL, Ask...) |
26 | $520 |
Links
Inbound links were another source of new users last year. There were a total of 108 new orders ($2016 revenue) as a result of a user clicking a link from another site. I generally don't do a lot of link building, but was fortunate enough to be mentioned in two articles on high traffic sites in 2010.
Direct
This means a user opened their browser and typed the domain directly into the address bar. How they actually learned about the site is impossible to tell. I generally chalk it up to word of mouth. Logically this group is pretty much the same as people who typed the site name into Google.
In 2010 there were 285 users who came to the site directly and eventually subscribed, generating revenue of $5700.
Summing It Up
Here's what the complete picture looks like for new individual orders in 2010. This doesn't include group orders or renewals.
| Type | Orders | % of Orders |
|---|---|---|
| AdWords | 331 | 23% |
| Organic By Name |
185 |
13% |
| Organic By Keyword |
516 | 36% |
| Link | 108 | 8% |
| Direct | 285 | 20% |
My AdWords Spending for 2010
This post is a follow up to How Much Money I Made From Side Projects In 2010 from earlier in the week. In that post I talked about how much revenue was generated from one of my side projects. Clearly those numbers are worthless without looking at how much it cost to generate that revenue. Let's take a look at that.
AdWords
Although I tried some other ad platforms as well (mentioned later), I spent the most with the Google AdWords network. In the HN thread for the previous post I estimated $5k. Turns out I was pretty close. Here's a month by month break down.
| Month | Clicks | Impressions | CTR | Avg. CPC | Cost | Cost / conv. | Conv. rate |
|---|---|---|---|---|---|---|---|
| 1/1/2010 | 334 | 21100 | 1.58% | $0.18 | $60.72 | $1.17 | 15.57% |
| 3/1/2010 | 126 | 12289 | 1.03% | $0.18 | $22.60 | $2.26 | 7.94% |
| 4/1/2010 | 428 | 42810 | 1.00% | $0.22 | $93.83 | $1.95 | 11.21% |
| 7/1/2010 | 369 | 22621 | 1.63% | $0.22 | $80.88 | $2.45 | 8.94% |
| 8/1/2010 | 4195 | 158202 | 2.65% | $0.27 | $1,116.61 | $1.99 | 13.37% |
| 9/1/2010 | 5126 | 193727 | 2.65% | $0.28 | $1,444.94 | $2.14 | 13.17% |
| 10/1/2010 | 2494 | 123450 | 2.02% | $0.27 | $671.19 | $2.09 | 12.87% |
| 11/1/2010 | 1166 | 71018 | 1.64% | $0.26 | $304.34 | $1.98 | 13.21% |
| 12/1/2010 | 1154 | 67267 | 1.72% | $0.26 | $298.40 | $1.94 | 13.34% |
| Total | 15876 | 921912 | 1.72% | $0.27 | $4,215.75 | $2.10 | 12.65% |
I was about $800 over on my estimate.
How I Read These Numbers
I'm far from an AdWords expert and I don't pay nearly enough attention to my campaigns as I should. The number I generally look at the most is cost per conversion ("Cost / conv" above).
In 2010 I paid $2.10 for every user that signed up for a free account. Since paid accounts at $20 to be profitable I would need to convert free users to pay users at a rate slightly better then 10% (10.5% to be exact).
Granted some of these users may renew which means in the long term I can afford to do less then 10% and still come out ahead.
Free To Paid
I didn't always, but for at least the last year I started keeping track of how new users came to the site. This allows me to check how well various customer acquisition methods are working at converting to paid accounts.
This is just looking at users that came in from AdWords.
| Month | Signups | Paid | Conversion Rate |
|---|---|---|---|
| 2010-01 | 57 | 11 | 19.3% |
| 2010-03 | 10 | 1 | 10% |
| 2010-04 | 50 | 3 | 6% |
| 2010-07 | 31 | 6 | 19.35% |
| 2010-08 | 561 | 102 | 18.18% |
| 2010-09 | 677 | 119 | 17.58% |
| 2010-10 | 318 | 56 | 17.61% |
| 2010-11 | 150 | 20 | 13.33% |
| 2010-12 | 145 | 13 | 8.97% |
| Total | 1999 | 331 | 16.56% |
What This Means
That I'm an idiot who should be spending more on AdWords.
I spent $4215 and generated revenue of $6620. That's +$2405 (not including PayPal's cut). My free to paid conversion rate is 16.5% - well above the 10% break even point.
Those Who Don't Learn From History...
...something something. Last year I came to the same conclusion. I'm not spending enough on AdWords.
Last year I spent $762 on AdWords and made $1720 for a difference of +$958.
Although I increased my AdWords spending 5.5x my revenue only increased 3.8x. This seems to indicate that my return will diminish as I throw more money at AdWords. There still seems to be room to push it higher though and still do well.
Facebook Ads
I also gave Facebook Ads a try last year. I spent $120 on ads over a 5 day period in September. No one even signed up for a free account. I killed that campaign pretty quickly. I'm not saying Facebooks Ads can't work, but I clearly needed to change my approach. I haven't revisited this, but may at some point.
Coming Up
In my last post I showed that I made $30k in new orders. If only $6600 came from AdWords, where did the other $23.4k come from? I'll take a look at that in an upcoming post so, make sure to sign up for my feed or follow me on twitter.


