Conversion Teardowns: Cratejoy (Subscription Box Marketplace)

Posted By Devesh Khanal | 16 Comments


Note: I’m trying a new style of post called Conversion Teardowns. I’m going to pick a site and break down how I would approach optimizing their conversions rate. We’ll cover SaaS, ecommerce, content marketing, and whatever else readers ask for.

Free Bonus Video: Get a user testing video where I walk through the site with my friend Justin Simon, ask questions along the way, and get his insight into what works and what doesn’t. Click here to get the link to the video, free.

Conversion case studies are great. They show real results from real sites (instead of cliche top 10 lists). But one issue with them is that you only see the nice, shiny, glorious end result.

You don’t often see the thinking and strategy that led to those results. That’s what I’m going to be presenting here (and in future conversion teardowns).

I’ll take a site that I haven’t optimized, haven’t seen the analytics of, and don’t have an affiliation to, and I’ll break down how I would think about optimizing their conversion rate.

Here’s my goal: This will show you how to start thinking about your own site’s conversion strategy — from the beginning. Not just “try a red button color”, but actual processes for doing user research, learning about customers, finding good test hypotheses, and where to prioritize your efforts.

Let’s get started.

The Site:

Subscription boxes are boxes of stuff that arrive at your place regularly, and you pay for it every month (or so). Cratejoy is like the farmer’s market of subscription boxes. You can go there and find all kinds of stuff that you can have delivered to you every month: food, books, shaving supplies, cosmetics, whatever.

For example here’s a box of candy and snacks you can get for $12.95/month:


So you get a bunch of snack food delivered to your door for a low price every month. All you have to do is pay $12.95/month and watch out for diabetes.

And (this is key) you can also sell your own subscription boxes.

cratejoy conversion optimization case study

So they have an interesting conversion optimization problem: optimize for two different sides of a transaction.

We’ll discuss that in a bit, but let’s first discuss the CRO process.

My 5 Step CRO Process

You’ll see this as a pattern in future conversion teardowns, so let me outline how I approach the CRO process:

1. Goals

2. Browser Testing

3. User Research

4. Your Own Walkthrough

5. Hypotheses

Since this is the first conversion teardown, let me explain the process in one sentence before we go step by step through the process for Cratejoy:

First, figure out what you should be measuring and measure that (1. Goals), then make sure stupid mistakes aren’t killing conversions (2. Browser Testing), then begin getting a deep understanding of what users want, what they’re trying to do, and where they’re getting stuck (3. User Research), then walkthrough the site yourself and think about what is clear, what is not, what works, what doesn’t (4. Walkthrough), and at the same time, with all the prior info in hand, make hypotheses of how you can more easily give users what they want better, faster, or with less barriers (5. Hypotheses).

If you follow this process instead of just testing random ideas that folks around the office have, you’ll get more sustainable results from your conversion optimization efforts.

Let’s go through the 5 steps for Cratejoy.

1.The Conversion Goal: Buyers or Sellers?

With marketplaces like this, the first thing you have to identify when starting to “optimize” a site, is the goal: Are you optimizing to get more sellers selling, or to get more buyers buying?

Don’t overlook how important this is for a marketplace site. There are two totally different visitor types and thus conversion goals:

1. Customers who want to buy subscription boxes.

2. Vendors who want to sell subscription boxes.

You have to be very careful on every page about who you’re speaking to. If you mix and match messages too much, you’ll confuse both sides.

Why B2C marketplaces focus on buyers first and sellers second

Notice that most established B2C marketplace businesses have their main homepage speak to buyers above sellers.

For example, here’s the homepage of Airbnb:

airbnb conversions

The entire hero unit is dedicated to you starting your search to book a place, and one button on the top is about becoming a host.

In a totally different industry, WordPress themes, here’s’s homepage:

themeforest homepage

Again, the most prime screen real estate is about finding themes (for the buyer), and there’s one totally non-descript link in the navbar for selling themes.

Reason #1: Buyers are the source of cash
One reason Airbnb, Themeforest and a bunch of other marketplace companies do this (Ebay, Amazon, Etsy, even Craigslist) is because buyers are the foundation for the entire business. They are the source of cash. The demand they generate is the motivation for sellers to sell in that marketplace.

Reason #2: There are a lot more buyers
The second reason is that these are consumer based businesses (B2C), so the number of visitors of each type are heavily swayed in the buyers favor, so it makes sense that the site would focus on buyers’ needs. In a B2B marketplace business (say, a business that hooks up companies who need heavy machinery with companies that sell heavy machinery), the number of customers is far lower, so the ratio isn’t as buyer heavy.

So, buyers make the whole business run, and there’s a lot more of them. Makes sense to have the website focused on them, right?

Cratejoy’s Homepage: Unclear Focus

Unlike Airbnb, Themeforest, and others, Cratejoy uses premium homepage real estate to appeal to sellers, and it’s sandwiched between copy and functionality for buyers, these leaves the user distracted and unsure on the best next step:


Here are the main goals they should measure in Google Analytics (GA):

New to Google Analytics? Check out this great guide from Justin Cutroni: Google Analytics E-Commerce Tracking.

1. Purchases (through the ecommerce integration of GA)
2. Popular subscriptions
3. Clicks on these elements of the homepage (among others):
4. Setup a funnel through the “subscribe now” step.

1. Clicks on the homepage to
2. Funnel through the store creation process.

If you do not have an ecommerece site I highly recommend you check out the article: Conversion Goal? It Depends on Your Business Model from Crazy Egg’s blog.

2. Browser Testing

Browser compatibility issues exist in almost every site. There are people likely viewing your site in a browser that are seeing errors and that’s drastically reducing your conversion rate.

For example, look at this GA breakdown of conversion rate vs. Internet Explorer version number from a client of mine:

IE 8 and 9 have a seroius conversion problem compared to other versions. And when we multipled the reduced conversion rate by traffic and average value of each conversion, we learned this was costing them $150,000 a year in revenue.


So, let’s look at Cratejoy.

I did these tests in (as usual, no affiliation. In general, I don’t do affiliate links on my site, so just know when I link to something it’s not cause someone is paying me 10 cents for you to click it or some nonsense).

Here’s a sample of OS/Browsers where I saw obvious UI issues. Since I don’t have access to their GA, I can’t see how many people arrive every month from each browser, but they can. So they should do this browser testing for all of their major browser types according to GA.

Windows 8/Chrome 22
Some search bar/nav issues

Windows 7/IE 8
Totally incompatible.

Windows 7/IE 9
Only a few subtle issues compared IE8.

No need to belabor the issue, but the general principle here is that you should test on all browsers and devices where you see non-neglible traffic.

Simply use a service like Browserstack to walkthrough a good chunk of your site.

3. User Research

Now we’re getting to one of my favorite steps: figuring out what people want (and what irritates them).

On page surveys

I’d definitely put a Qualaroo style survey on Cratejoy’s homepage, and their “start selling” page (

What I (and others) find works best is open-ended qualitative questions, with a few more pointed yes/no’s thrown in. As Dave Chaffey, founder of Smart Insights, explains in an article by Raphael Paulin-Daigle:
“We’re fans of asking open questions on specific pages in the journey to get additional insight you can’t get from analytics tools.”

So on the homepage, I’d ask:

  • What kind of subscription boxes are you looking for?
  • Have you ever bought a subscription box before?
  • Are you finding what you’re looking for?
  • What else do you want to know before trying a subscription box?

In terms of non-qualitative questions, I’d also ask:

  • Are you looking to buy or sell subscription boxes?
  • Have you ever thought of selling a subscription box?

People will then start answering the open-ended questions by basically telling you their problems and desires. Some will be short; others will be long.

Pro Tip: After I get 100 or more responses, I categorize them by subject. Then I make a simple histogram of the responses, so we can all clearly see the subjects users are asking and talking about the most. Here’s an example for a recent SaaS client:

conversion survey

Most people wanted to know more about how the software works, setup, etc. (We did a deeper analysis of all the how it works questions as well). And a ton of people just wanted to know price, which was not mentioned on the homepage.

Note: Just because users ask for something in the qualitative survey, it doesn’t mean showing them exactly that will improve conversion rate. It just gives you insight into how the majority of them are thinking. For example, we showed pricing on the homepage for the SaaS company mentioned above, and conversions stayed the same.

So on page surveys will give you good insight into what live users want to ask for, complain about, and compliment you on.

Screen recording

For ecommerce type sites like this (I know this isn’t “traditional” ecommerce, but site functionality is similar: lots of product options, search, browse by categories, etc.), screen recordings of real users are a great way to figure out how users actually browse, add items to card, and check out.

They require patience. Watching people browse your site over and over again can get tedious, but they can reveal interesting behavior that you haven’t noticed before.

For Cratejoy, I’d be curious about a few things:

  • How do you users actually find a subscription box they like? Trending? Similar? Search? Categories? Reviews?
  • Do they use the trending subscriptions?
  • What about sidebar items?
  • Do they linger on reviews?
  • Do they click through the different photos?
  • Do they use the “people also bought” section?
  • How many boxes do they typically browse before making a purchase?

Some of these are things that you could measure with careful click and event tracking in Google Analytics, but most people don’t do that. Other things (how long they linger on reviews or certain descriptions) are not easy or possible to measure with GA. (page time is not the same as time on a certain section of a page).

Live User Testing

Finally, an invaluable technique is to use actual live, paid users to go through your site and give you feedback while they browse and after they’re done. According to marketing consultant Kristina Allen, “There is no better way to find out how visitors behave on your site and what makes them convert at a higher rate than through user experience (UX) tests.”

You can do this with and competitors (I have no affiliation). Or, you can just put an ad on craigslist and ask for certain type of people and offer how much you can afford per video.

For a wide reaching B2C business like Cratejoy, it’s easy to find users that fit their target customers. You can watch them browse and have them give you feedback about what they like and don’t like.

For Cratejoy, I’d be very curious about the above questions from live users AND what else they ask for that I haven’t thought of.

  • Are they asking for more photos?
  • Are they curious why I have to go to a different site to actually purchase (I am!)
  • Do they even notice that you can browse by category way on the bottom of the homepage?
Free Bonus Video: Get a user testing video where I walk through the site with someone who has never seen Cratejoy before and is asked to find a gift for his parents. I ask questions along the way, and get his insight into what works and what doesn’t. Click here to get the link to the video, free.

4. Your Own Walkthrough

Your own walkthrough is the last step after you have all the user research in hand, so you already have some ideas of things that could be hampering conversions.

Now, it’s time to slowly go through the site, page by page (or page template by page template) and list:

  • Things you can do better to answer user concerns
  • Ways you could express value more clearly
  • Places to remove distractions and clutter
  • Ways to help users navigate more easily to what they want
  • Other ways to add elements of persuasion

Check out my sample walkthrough of Cratejoy, here:

Obviously, a full site walkthrough will take a lot longer than 12 minutes, but you get to see the kinds of questions I ask that I’d end up testing.

Here are some of the questions I’ve asked myself when walking through Cratejoy’s site:

  • Are trending items the first things you should see on the homepage? Is that really how users browse for subscription boxes?
  • What about categories. I might be interested in subscription services, but I’m totally overwhelmed with what I’m seeing.
  • Clarity clarity…am I shopping for a subscription or am I starting a business??
  • I want the best ones, not the newest ones or hottest ones.
  • I want to know what the box has in it. Not just a photo.
  • Why do I have to go to a seperate looking page to buy? Can I trust them?

5. Hypotheses

Finally, after all the information from the above steps is gathered. You should list them out into hypotheses. That is, variations to certain pages or page types to test.

Test them, of course, with your favorite A/B testing software like Optimizely or VWO.

Watch the video above to see some hypotheses that naturally fall out of the walkthrough.

Get a bonus user testing video plus more teardowns like this free

I’ll be walking through more sites on a regular basis. If you want to get future teardowns in addition to other conversion tips, as well as the recording of my user testing walkthrough with someone who has never seen the video before, click here and let me know where to send the video links and future teardowns. It’s totally free.

Have you gone through yourself? If so, what would you test that I haven’t thought of? Let me know in the comments.


  1. Chris Von Wilpert
    December 16, 2015

    This is a nice beefy post Devesh. Looks like you put a ton of effort into it. Loved seeing your whole process and how CRO is not just about changing something based on gut feeling and hoping it works. And I never knew you could have click Goals until reading this. Just to let you know the first minute of your teardown video seems to repeat itself.

    • DeveshKhanal
      December 16, 2015

      Awesome, glad you liked it. Thanks for the feedback on the video, I’ll definitely use that to make them better going forward.

  2. Kate
    December 16, 2015

    This is brain picking love it! You mentioned that we need to be very clear who we are talking to on the homepage, in the case of CrateJoy it would be optimal to gear towards the buyers because they are the income source. Now what about a site like Odesk? who should be a priority here, the freelancers or clients?

    • DeveshKhanal
      December 16, 2015

      Thanks Kate! For Odesk/Upwork, I’d still go with gearing the homepage to the buyers (and they do). Although for upwork, there are a LOT of freelancers, the buyers are the ones that run the business because they generate demand. If there is a LOT of demand, freelancers will naturally want to come and join.

      • Kate
        December 16, 2015

        Ahhh that make sense. My original thinking was that freelancers and clients are weighted equal in Odesk case. Clients come to the site, see a limited numbers of freelancers then the clients would leave. Same thing goes for freelancers, they see little jobs available, they leave. That left me wondering which side I should talk to on the landing page.

  3. Steffen Kaufmann
    December 16, 2015

    Just wanted to let you know that I really appreciate your material Devesh. I follow a lot of blogs but yours is really top notch. Please keep doing awesome stuff – I appreciate you!

    • DeveshKhanal
      December 16, 2015

      Steffen, thanks so much for those kind words. I’m glad you like the material and appreciate this comment. Curious, what specifically makes my material stand out to you?

  4. Emils_W
    December 17, 2015

    Great article. A more effective way to do live user testing is to use actual visitors instead of random people. Tools like make it fairly easy to invite your traffic to participate in user testing.

    • DeveshKhanal
      December 17, 2015

      Great point Emils! Indeed that is a great technique and I mention it briefly in the screen recording section above.

  5. Rhys Kilian
    January 13, 2016

    Really interesting read @DeveshKhanal:disqus ! I found your blog through Bryan’s post (

    This is a really cool experiment you are doing here. Do you have any more data about the suggested user testing? I’d love to see what Cratejoys’ customers have to say.

    Anyway, looking forward to the next post in this series!

    • DeveshKhanal
      January 13, 2016

      Thanks Rhys. I don’t have any actual customer data for them (as I mentioned, I have no affiliation with them), but perhaps in a later company we’ll get something like that going. Next post is in the works…

      • Rhys Kilian
        January 14, 2016

        It’d be really interesting to see what those numbers turned out to be. Would be an interesting little experiment.
        And I’m looking forward to it!

  6. Jay Nguyen
    September 20, 2016

    Hey Devesh – really enjoyed this teardown mate. Some really great insight and stuff that I could apply straight away to sites I’m working on. Keep up the great work!

    By the way – note sure if there was something wrong with your ‘Confirm Opt-in’ page, but it looked a bit weird to me with the placement of the ’email graphic’ and steps 3&4. ure


Leave a Reply