Increasing Profit with
Newsletter # 115: 3/2011
Drilling Down - Turning Customer
Data into Profits with a Spreadsheet
Customer Valuation, Retention, Loyalty, Defection
Get the Drilling Down Book!
Hi again folks, Jim Novo here.
Is anyone in Online Marketing measuring success at the customer
level? As I pointed out in a blog post titled All Talk, No #Measure,
it seems really strange to me there is all this talk about
"customer centricity" and "social marketing" but
all most people use to measure online marketing success is
visitor-level metrics. What happened to
Other than for a quick snapshot in time,
this visitor-centric measurement approach is not really about
customers. What would happen - how might results change - if the success
measurement of Online Marketing became customer-centric at the same
pace as all this talk about centricity?
Fortunately, we have a great example of what might happen courtesy
of a fellow Driller. And it should make you wonder how much
money you are leaving on the table by talking customer - but measuring
Questions from Fellow Drillers
Increase Profit with Customer State
Q: We've been playing around with Recency / Frequency scoring in our customer email campaigns as described in your
book. To start, we're targeting best customers who have stopped
interacting with us. I have just completed a piece of analysis that shows after one of these targeted emails:
1. Purchasers increased 22.9%
2. Transactions increased 69%
3. Revenue increased 71%
A: There you go!
Q: My concern is that what I am seeing is merely a seasonal effect - our revenue peaks in July and August.
So what I should have done is use a control group as you described in the book - which is what I am doing for the October Email.
A: Yep, that's exactly what control groups are for - to strain out the noise of seasonality, other promotions, etc.
But don't beat yourself up over it, nothing wrong with poking around and trying to figure out where the levers are first.
Q: Two questions:
1. What statistical test do I use to demonstrate that the observed changes are not down to chance
2. How big should my control group be - typically our cohort is 500-800
A: Good questions.
On a group that small, you are probably not going to get anything "statistically significant" without ruining your total profit, e.g. might have to use 50% in control. If you have the leeway to do it, that's what I would do.
On the other hand, in some cultures people will go bonkers over giving up sales to learn something really important. OK, so take 10% as control and repeat it 3 times; if the results are stable then you have your proof.
Do another control every once and a while (every 6 months?) just to make sure it
tracks. Either way, you don't really need statistics.
Practically, confidence is the likelihood a sample represents the population.
This can be a really useful idea when you are forced into small sample
sizes or the event is risky to repeat.
But here, if you are testing a really large slug of the population, confidence is less useful.
Or if you can repeat the event (because essentially, you are in control of it and it's low risk), do you really need to force yourself through the screw of complying with the
statistical math? It's like using a 727 for crop dusting, overkill
for the situation, methinks. If you were running a drug manufacturing line, statistical concepts like confidence and significance are absolutely valuable. But for a marketing program?
That's why I love the idea of "beefy controls" in start-up projects
because I *do not* have to rely on statistics that the audience likely does not understand and
provide room to question the results, e.g. "Yea, but what if the result is an outlier?"
Very appropriate in high risk situations, with giant populations and a lot of money on the line.
For this situation, perhaps not. But, if you'd like to go that way, there's lots of calculators on the web that let you play with some of the numbers anyway.
Here's one, make sure to read the descriptions of the variables underneath the calculator:
Nice work on the core campaign idea, by the way! Now we just
have to tighten it up a bit...
(3 months later)
Q: We decided to tighten the targets and do a "best customer defection" email program. Basically, we look at customers who
has an RFM score of 555 in the previous scoring period who have dropped out of that score.
A: Interesting! So instead of targeting by
guessing the current score of a defecting best customer (say 355), you are looking for all customers who were formally best customers, regardless of current score.
This is a subtle difference, but much more of a LifeCycle approach and frankly why I prefer
these kind of ideas over "straight" RFM.
An example might be helpful. Let's say the acquisition folks run a huge new customer campaign in between the prior RFM scoring and the scoring done before your campaign drop.
A big inflow of new customers can artificially "force" certain groups of customers down in score - even though their own behavior has *not changed*.
In this case, the new score is not reflective of actual behavior, so
increases noise in the system.
That's the problem with the "Snapshot" or date-specific view of Customer State - it's a single point without reference.
By using prior score, you are acknowledging behavior over time and the primary importance of the former State, as opposed to the
current State - a Movie
of the behavior as opposed to a Snapshot.
Like I said, subtle difference and sometimes these approaches could be roughly the same, but a better approach overall, less risky.
And in general, more knowledge is generated in the post analysis from the
Movie approach than the Snapshot approach, for example: what were the scores of the responders versus
non-responders - what does the customer LifeCycle look like?
Can't ask this question with the Snapshot approach, since the score for all customers is the same.
Q: Good news on your advice. We ran a 50% control (500 purchasers in each group) and the results really nailed the issue for us. The actual number of purchasers remained unchanged at 20% but Total Revenue and Average Spend increased by 40% compared to
(Jim's Note: for those not following, a very precise target group
of 1000 was split into 2 groups of 500. One group received
this campaign, the other did not. People who did not
receive the campaign purchased at the same rate as people who did
receive the campaign, but the people who received the campaign
averaged 40% higher spend).
A: Awesome. So what you are seeing is Customer State makes a huge difference in terms of what offers
/ timing can be most effective for this "Recently defecting best customers" cohort.
If I'm reading your numbers correctly, no lift in response versus
control but a huge lift in revenue.
To me, that means these customers are early in the process of
defection - still buying, but without a special treatment, slowing
down the monthly spend. After all, they are very Recent (former
5XX), so highly likely to purchase again, which is why lift in
response was flat - they likely would have purchased anyway.
Not a bad time to hit them. Offers to a very Recent State
should focus on increasing order value, not generating response - you
don't want to spit into the wind, but go with the natural flow of the
In other words, these customers likely would have purchased anyway, but at lower price
points if they had not received the campaign.
The common way this is addressed is with "threshold"
discounts - if average order is $50, then something like "$10 off
any purchase over $50" - test different thresholds to maximize
Looks like you gave them the right offer ;)
On the other hand, a straight discount to this specific best
customer group - $10 off anything, and especially when their normal
category of purchase is promoted to them - almost ensures that you
will lose money. Why? Most of these
customers would have bought at full price anyway, as demonstrated by
equal buying activity whether the customer received the campaign or
not. So the discount turns into a loss versus no campaign at
Unfortunately, I see a lot of this exact type of campaign delivered
to best customers because all customers get some version of the same
offer. "Hey Jim, we're not sending the same message to
every customer, we send different messages by segment".
Sure, the copy and art are customized for different segments, but the
segmentation is not by Customer State, so the offers are mismatched
This is the value of using control groups; they drive understanding
of Marketing concepts like opportunity costs and subsidy costs.
These two concepts are the reasons why ignoring Customer State is
suboptimal: by not segmenting using State, you will get lower than
possible profit or sales at most customers, depending on Customer
Had you not delivered a campaign tailored for prior Customer State,
money would have been left on the table by way of lower order size. And 40% Revenue lift sounds like it might have covered
the cost of the campaign ;)
Q: We tried to run a Student T test on the results but our new statistician informed me that the distributions were not normal - so on her advice we ran a Wilcoxan Test which gave us a highly significantly p = 0.016
A: Oh, so you still went the stats route? Well, the fact you HAVE a statistician tells me the culture there is more familiar with interpreting these ideas, so more power to you.
Glad it worked out and keep me informed on how things go downstream.
Have a question on Customer Valuation, Retention, Loyalty, or Defection?
Go ahead and send it to me here.
Need help with creating action-oriented customer analysis or High ROI
Customer Marketing program designs? Click
That's it for this month's edition of the Drilling Down newsletter.
If you like the newsletter, please forward it to a friend! Subscription instructions are top and bottom of this page.
Any comments on the newsletter (it's too long, too short, topic
suggestions, etc.) please send them right along to me, along with any
other questions on customer Valuation, Retention, Loyalty, and
'Til next time, keep Drilling Down!
- Jim Novo
Copyright 2011, The Drilling Down Project by Jim Novo. All
rights reserved. You are free to use material from this
newsletter in whole or in part as long as you include complete
credits, including live web site link and e-mail link. Please
tell me where the material will appear.
What would you like to
the book with Free customer scoring software at:
Out Specifically What is in the Book
Marketing Models and Metrics (site article