Mastering the Testing Spectrum

Paper, postage, pixels—the bulk of the cost of a direct marketing campaign is in the media buy. Creative, testing, and audience modeling are inexpensive compared to these costs, yet make a massive difference in the outcome of a campaign. Because of this, the only reasons not to test a communication is if you don’t have a large enough audience to make it worthwhile (and we’ll see some remedies to that below) or if something is so well tested you can’t possibly improve it (unlikely). 

Decoding the Testing Spectrum 

There are two schools of thought on what to test. One is the standard direct marketing approach of one variable at a time to get highly controlled results—effective, methodical, and slow. The other counsels “don’t test whispers”—changes must be dramatic enough to cause a significant impact, for good or ill. 

It’s time to acknowledge that both are right. You want a portfolio of risk in tests, from Google’s famously testing 41 different shades of blue for their links on one side of the spectrum to “let’s blow this up and see what happens” on the other. 

The trick is knowing when to deploy which type of test. Generally, the better you know the audience and the more tried and true the communication, the more you can nibble around the edges with small changes. 

But to find new audiences, messages, or approaches, you do need to step out completely of what you’ve done before. 

Strategic Testing in Action 

Let’s say you began with a premium-heavy direct mail program, where items like labels, calendars, notepads, and bumper stickers were frequently used to acquire new donors and featured prominently in your mail packages throughout the year. You realize you don’t have many non-premium donors, and you want to broaden your audience. Let’s try three scenarios: 

Scenario 1: You try a non-premium mail piece but don’t change your audience. This likely will not work. If you try it with your current donors, most are premium-based donors or donors you have trained to expect this. If you try it on an acquisition audience, your models and list strategies are all calibrated for your premium audience and will meet the same difficulty. 

Scenario 2: You try a non-premium audience but don’t change the appeal. Same thing in reverse—the message doesn’t fit the audience. 

Scenario 3: You try a non-premium mail piece and intentionally pick a non-premium audience. It works! You’ve acquired a new donor segment… who will run and hide when they see the premium-heavy rest of your mail program. 

Now, replace “premium” and “non-premium” with your organization’s equivalent. You want to move to acquire more younger donors, or donors of a different ethnic group, or religious or non-religious donors, or those of a different political persuasion. Whenever you look to broaden your appeal, which you must always do lest you get better and better about talking to fewer and fewer, you might get lucky and find your new message works for your old audience or that your old message works for your new audience.  

But you must always test audience and message simultaneously and be prepared to customize forever. 

In short, it’s hard. 

The New Horizon of Testing  

Thankfully, it’s getting easier to do these hard things. Generative AI makes it easier to generate message quantities and to rapidly prototype new approaches. Survey research makes it cheaper to test before you must buy pixels, paper, or postage. Machine learning makes it faster to look back at old communications to see advantages you may have missed, suss out the impact of multivariant testing offline or online, and to see the value of adding new channels. 

It’s why we just launched Moore Audience-Centric Testing suite (or Moore ACTS)—to make it easy to get the most out of when you do buy media. When you test before campaigns (or alongside them), you don’t have to worry about your audience size constraints—you can design tests that get you significant results. 

Regardless of how you do it, you should be experimenting with everything from font size to messaging tone to the right audience segments to target across digital and direct mail platforms. The result is efficiency for your budget and effort by placing only the best appeals in front of your audiences. It’s also about knowing that you can confidently expand your appeals to new audiences.  

Imagine walking into your next board meeting and confidently showcasing your latest campaign results—not because of a lucky guess, but because you tested, optimized, and refined every step of the way. That kind of strategic thinking separates great campaigns from merely good ones and growing programs from static ones.