Thanks for visiting the original Drilling Down web site!  

The advice and discussion continue on the Marketing Productivity Blog and Twitter: @jimnovo

Read the book-
First 9 Chapters: pdf

Get the book at

Relationship Marketing, Customer Loyalty, and Retention book

Customers Speak Up on Book & Site

Workshops, Project Work: Retail Metrics & Reporting, High ROI
Customer Marketing

Fresh Customer
Marketing Articles

8 Customer
Promotion Tips


Customer Retention

Customer Loyalty

High ROI Customer
Marketing: 3 Key
Success Components

LifeTime Value and
True ROI of Ad Spend

Customer Profiling

Intro to Customer
Behavior Modeling

Customer Model:

Customer Model:

Customer Model:
Recent Repeaters

Customer Model:

Customer LifeCycles

LifeTime Value

Calculating ROI

Mapping Visitor

Measuring Retention
in Online Retailing

Measuring CRM ROI

CRM Analytics:
Micro vs. Macro

Pre-CRM Testing for
Marketing ROI

Behavior Profiling

See Customer
Behavior Maps

Favorite Drilling
Down Web Sites

Book Contents

Contact Jim Novo



Drilling Down

Turning Customer Data into Profits

 with a Spreadsheet

Site and Book topic:

Maximizing marketing ROI with customer behavior analysis


Learn Methods, Metrics
(site map)

[ Home ]    [ FAQ ]    [ Download   [ Contact / About / Privacy ]

This is the faster loading site for slow connections.  The "pretty" version of this page is here.

Random Sampling, Control Groups, Halo Effects

High ROI Customer marketing depends a lot on a tool known as control groups.  A control group is a random sample of the customers targeted for some kind of test program.

Let's say that you wanted to test a discount mailing to your best customers.  You would select all these customers for your list, and then take a random sample of them to exclude from the mailing - usually anywhere from 3% to 10% of the total.  This group is known as the control group; the others who will receive the mailing are the test group.

Why do this?  Since the control and test customer groups are exactly the same, you can compare the buying behavior of the test group versus the control group over time to determine precisely what the effect of your mailing is.  Taking this approach screens out a lot of external noise (like other promotions these groups may be exposed to) and gives you a true read on your profitability.  

Using control groups also allows for inclusion of typically high ROI halo effects, which are rarely measured by most people doing promotions.  Halo effects occur when people respond to a promotion outside of the business tracking process but are "not counted" as having responded.  For example, you send a discount and the customer loses it but makes a purchase anyway because you "reminded" them of a need they had.  Typically, all anybody measures is response, which does not give a true read on ROI.  

You cannot measure halo effects without a control group.  If you aren't using controls, you are short-changing yourself, because the promotion could be many times more profitable if you include the halo effects.

Random Sampling

Most controlled testing in database marketing requires the creation of a random sample of your customer base, either for the test group - targets receiving the mailing, or the control group - those not receiving the mailing.

When you are testing new concepts, you usually don't want to blow a whole bunch of money, so a random sample of the target group is created for the mailing (test group), and the rest of the target group acts as control.  When you are going with proven high ROI concepts, you want to mail as many pieces as possible (test group), so the random sample is created to act as control.

For the first case, when testing new concepts, the larger the random sample is on a percentage basis, the more accurate its predictive power will be.  You want the results of a test to be repeatable - if it works, you want to do it again.  The larger the sample is, the more likely the results of the test can be repeated on the next mailing.

Three percent will give you a pretty good shot.  Larger samples will cost more to mail but will add extra stability to the predictive power of the sample; smaller samples could result in unstable predictive power, for example, the promotion makes money the first time but when repeated it loses money.

If you can afford it, go to 10%; 5% is good, but 3% is OK.  The smaller your database, the higher percentage you should take for a test, in general, to even out the instability that comes from testing small databases (under 5,000 customers).  If you have only 1,000 customers, consider a 20% test, or if you can afford it, run the test to every customer not in the control group.

In the second case, tracking proven high ROI concepts, the larger the control group sample is, the more reliable and repeatable the results of the promotion will be.  Early on in the life of a promotion, it is a good idea to use a "fat" control group, just to make sure the ROI is tracking.  Over time, you can reduce the size of the control group when you are confident the results are stable.

These tests are extremely important events, as the information gained is used extensively down the line.  Donít skimp on a test if you can help it.  Also make sure the sample is truly random, and doesnít introduce any bias, meaning the sample is not truly random because the selection methods used have distorted the selection process.

Here's an example of introducing bias during random sample selection:

Letís say you have 1,000 customers, and they were consecutively assigned customer IDís, meaning you oldest customers have the lowest ID numbers.  You want a 10% sample, or 100 customers.  Your customers happen to be sorted by customer ID, and you start choosing customers with customer ID 1 and select every 5th customer.  You would have the 100 customers you need by customer ID 500.  But your sample would be biased, because the customer group you have selected has a higher percentage of old customers than the entire customer base.

The customer base was sorted by ID, meaning your oldest customers have the lowest ID and newest customers the highest ID.  You stopped choosing at 500, instead of choosing through the entire customer base; this creates the bias towards older customers.  

If you had selected every 10th customer instead, you would have ended with your most recent customer and have an even sample with no bias against representation by a particular customer group.  Bias can occur geographically, by product type, and so on.  Be careful with the way a database is sorted if you are using a "choose every Nth customer" random selection technique.

A convenient way to generate a random sample, if you use consecutively numbered customer IDís, is to pick a digit location from the customer ID, and specify a value for it. Then choose every customer with this value at the specified location in the ID.  Youíll get a 10% sample.  For example, ďgive me everybody whose customer number ends in ď2Ē or ďgive me everybody having a 4 in the second to last digit locationĒ.  For this to work, you have to have at least one customer in the next highest (to the left) digit location.  For example, if you have 5,349 total customers, you could use any of the last 3 digit locations (left of the comma in 5,349) but not the lead (left-most) digit location.  Using the left-most digit would introduce bias, since the selection would complete halfway through the run, before a full 10% sample is taken.

What would you like to do now?

Get the book with Free customer scoring software at:     Barnes &

Find Out Specifically What is in the Book

Learn Customer Marketing Models and Metrics (site article list)


Note: This is the "slow connection" version of the home site.  It's intentionally sparse design allows faster page downloads.  To access this content in what some say is a more visually appealing format, click here.

Questions about any of the concepts on this site?   e-mail me.

What Will I Learn
in the Book?

About the Author
Newsletter Sign-Up

Example of the
Drilling Down Method

See Drilling Down
Results in Action

Relationship Marketing
Customer Retention

Customer Loyalty

Get the Book
with Free Software!

Fresh Drilling Down
Related Articles

Advanced Customer
Modeling Articles

       [ Home ]    [ FAQ ]    [ Download ]    [ About / Contact / Privacy ]  

Welcome to the original Drilling Down web site;
recent advice and discussion are on the Marketing Productivity Blog and Twitter.

Contact me (Jim Novo) for questions or problems regarding this web site.   
Copyright The Drilling Down Project. All rights reserved.  Privacy Policy.