Keunbo Hur

“He paid you, in cash, in a parking lot. And now you’re only 4 weeks behind?”

You guessed correctly. That was when I confirmed with a coworker that the CEO had paid her in cash in a parking lot, causing her to only be 4 weeks behind on pay. I was envious, I was 8 weeks behind.

Just a year earlier, we’d all been in the office, on the 8^th floor of a building at the most central intersection in downtown San Jose, latest tranche of funding had hit, we were going back and forth from eating $19 personal pizzas at San Pedro market and $20 California rolls at Sushi Randy’s, a surprisingly terrible Japanese restaurant, even with the name accounted for. Maybe we had the debacle coming.

Or maybe it could have been avoided had we not switched our “strategy” every 3 weeks. But alas, our MO was rooted in a second problematic interpretation of the mantra ‘Fail Fast’:

Labeling inconclusive results as failures for the sake of ‘speed’

If you owned a pizza shop and were trying out a new dough recipe to boost sales, you’d probably want feedback from more than one customer before deciding to keep it or scrap it, practically, you’d want feedback from many more customers than one before you make a call when whether the dough is a success or failure (even more practically, you’d want to know that a new dough recipe has a good chance of boosting sales, as per my first post, that often doesn’t happen). This makes intuitive sense even without a background in statistics.

Alternatively, if you’ve taken a stats or a research methods class at some point, you might recall the heuristic ‘p-values smaller than 0.05 are good’, where the p-value is one of the common measures that quantifies how much less likely due to chance the conclusion drawn becomes as the number of trials increases (assuming there is a true effect, a fair assumption for most revops initiatives). Almost every revenue leader I’ve worked with also understands this principle of ‘more trials more conclusive’ either intuitively or explicitly, yet many still tend to make one type of judgment call against this intuition in one direction – that is, to call an experiment a failure and moving on to the next thing.

So why does this happen?

In my experience, it’s been largely due to underestimation of or altogether failing to account for the ramp time and resource intensity of a new initiative. At our office in San Jose, the CEO had a tendency to label an idea as a failure and quickly move on to a different experiment, which wouldn’t have been an issue had he made reasonably sure that a given idea truly didn’t work.

Furthermore, this habitual underestimation of time and resources needed to properly reach a conclusion often compounded and led to amplified detrimental effects through a propagation of error of sorts.

For example, one of the first weeks after I joined, we started targeting new customer segment, above $20M in revenue or above 500 employees per LinkedIn. The first few weeks of conversion metrics looked similar to the following:

The conversion rate calculation for the 3 weeks looks like the following:

(7 qualified first 3 weeks) / (120 leads engaged first 3 weeks) =  5.8% conversion rate. Which was deemed not ideal and abandoned, worse yet, most of these ideas were revisited some time later and we’d have to start over again.

Had we seen the idea through a few more weeks, we’d likely have observed better numbers for weeks 4 through 6. This is because the efficacy of a new idea will improve with time as the organization as a whole ramps up. Roughly accounting for training/ramping volume looks like the following:

Recalculating the qualification rate:

(7 qualified first 3 weeks) / (120 leads engaged first 3 weeks – 60 leads engaged for training) =  7/60 = 11.7%

At our org, the threshold was somewhere between the 5.8% and 11.7%, and the accounting of ramp would have deemed the experiment an initial success and warrant further investigation.

The reason given for not accounting for ramp was something along the lines of “The reps know how to do outreach, they shouldn’t need to ramp”.

As absurd as the reasoning may seem on the surface, there’s usually significant validity to a seasoned executive’s judgment on the capacity of his team. Had there been only a tendency to rule experiments a failure prematurely been the only issue in our operations, it likely would have gotten fixed after a few rounds of trial and error.

But as mentioned earlier, premature labeling something a failure tends not to be contained in one aspect of the business. At our office in San Jose, in addition to new go-to-market ideas, our sales reps were given very little time to onboard before being deemed a ‘failure’. We had that revolving door for the sales team spinning at a speed that gradually caused structural damage to the org, up to a point we were doing payroll in a damn parking garage.

If we’d just gone slower, we likely would have picked up more sales velocity in the long run. But instead, with an impatient, albeit passionate and nice CEO at the helm, the entire company tried over and over to hit the jackpot on a slot machine with sales method in the first column, sales rep in the second, and macroeconomic forces at play in the third.

In part 3, I cover practical methods to mitigate risks behind a ‘fail fast’ culture.

NOTE: Reporting conversion rates by week doesn’t work well unless there’s massive scale due to variance week over week being large for startups/smaller sales teams. While there are common methods to attenuate variance such as moving average, they tend to hinge on some lengthening of the ‘experimental period’.