Our Blog

When to Stop Interviewing and Start Writing Code

So you’ve done a bunch of interviews. How do you know when to stop, and then start building? Use this article to learn a couple of different methods.

Robert Graham of WhiteTail Software, and this awesome guest post on cold calling asks:

@whitetailsoft https://twitter.com/#!/whitetailsoft When do you stop #custdev efforts and build the product? I’ve been wrestling with the details of #leanstartup.

I talked to 30 people before I realized that a certain idea of mine was a crappy idea, and about 40 people before starting WP Engine. Here are the details of both of those customer development experiences. 

But there’s no one “number.” Food on the Table — a now-famous lean juggernaut in Austin run by IMVU alum Manuel Rosso — talked to 120. Capital Factory 2011 alum GroupCharger talked to 50 before building and another 50 after that. At AppSumo, another Austin startup with startling growth, Noah Kagan talked to 0 people initially, but maintains ruthless pressure on a tight and measurable product.

There’s two ways to decide when you should stop talking and start building.

Way #1: Go until boredom.

Recently at WP Engine I did some brand new customer development for a new project that we think will revolutionize WordPress blog management. I spent 30 hours talking to WordPress consultants, but I didn’t have “30” preset in my head. I knew to stop when the process got boring.

The first dozen calls were a blizzard of activity — my note-taking fingers furiously trying to keep up with new information being revealed, theories getting alternately validated and blown away, unexpected customer segments arising, and new ideas recombining from the primordial soup generated by introspective, honest, provocative conversation with thoughtful people who were “living the pain” we’re trying to solve.

This is what it feels like to learn, remember? If it feels like learning, you’re probably, you know, learning, and that’s the point. If you’re not learning, what are you accomplishing?

But then as the conversations wore on, I felt a change. As I emphasized what I (now) thought was interesting, sure enough the person on the other end of the phone gets excited. As I attempt to push old ideas which just two weeks ago seemed so fresh and smart, sure enough they fall flat again.

The change is exactly like those charts from biology showing the rate that a chemical imbalance diffuses into a cell — fast at first when the imbalance is high, then lessening asymptotically.

With my recent interviews, it got to that flat place in the graph. I could segment the customer from the first two questions, and then I could predict almost perfectly what the answers would be to the other questions.

No more learning is happening. Now you’re bored. Time to stop.

Does that mean there’s nothing else to learn about the market? Of course not — it means that this particular exercise is no longer useful right now.

So now it’s time to do something else to progress the company and peel back the next layer, and that means coding up and releasing your MVP.

Way #2: Get ten paying customers.

Remember this is before you’ve built the product, so if ten people give you cash anyway, that’s the best possible signal that the world wants this product to exist.

If it takes talking to 200 people to get those 10, that’s not a good sign. You might think “5% conversion is pretty good,” but not when you’re getting face-time. If the founder sits down with what ought to be the perfect customer and chats for an hour and cannot convince more than one in twenty of the value of the project, it’s too difficult to sell.

But with WP Engine I had lunch with 40 people and 30 said they’d give me $50/mo as soon as I built it, and 20 actually did before launch. I should have asked for the $50 on the spot, or at least had them sign something committing to it; then I would have focussed my remaining efforts on making those 20 people happy instead of diluting it with all 40.

“But it’s impossible to get non-technical people to pay for ‘software’ that’s nothing more than sketches on a napkin,” I hear you say.

Bullshit. Both Food on the Table and GroupCharger did it, one with busy moms (i.e. consumers), the other from small associations (i.e. business). Two of the other Capital Factory companies did it successfully too.

In fact, almost all the people I tell this to are initially incredulous, but you should see the look on their faces when it works. It separates the polite interviewees from the truly interested, and the truly interested are frequently up for a reasonable pre-pay.

And once you get 10 of those, that’s enough opinions. Every customer wants something different, all see a different direction for your company, each have their pet features they want, each will find some part of the product confusing or unnecessary or critical. More cooks in the kitchen — at this very early stage — is not better, because design is not a field for the Wisdom of the Crowds http://blog.asmartbear.com/ignoring-the-wisdom-of-crowds.html.

Fewer than 5 and you’re designing an idiosyncratic product, not much better than “building it for myself.” More than 15 and you’ll have too many voices not converging on simple truths.


“When to Stop Interviewing and Start Writing Code” was originally published by A Smart Bear on June 12, 2012.
By: Jason Cohen, Founder of WP Engine & Smart Bear Software.