Introducing test and measure | RetailNext

Comprehensive In-Store Analytics


Introducing test and measure

Tim Callan
Tim Callan
Chief Marketing Officer

As powerful analytics capabilities become available to bricks and mortar, it’s time to start using test-and-measure methods

The past year has seen strong growth in the awareness, implementation, and sophistication of analytics usage among brick-and-mortar retailers.  With this increasing awareness and sophistication, physical retail has the opportunity to take its learning to the next level by applying rigorous test-and-measure methodology.
The basic concept is pretty simple.  Marketing and sales professionals have the opportunity to sell more or to sell less based on how they approach their customers.  They can affect sales and profitability through their choices around products, messaging, outbound marketing programs, customer experience, and more.  In the case of brick-and-mortar retail, a huge component of success depends on what retailers do inside their four walls.  We’ll come back to that.


Once you appreciate that these different choices can lead to radically different results, you’re ready to start testing.  The most common configuration is a simple A/B split test.  You divide your customers into two homogenous and consistently performing groups.  Left to their own devices, both groups will exhibit the same buying behaviors and yield the same profit to your sales activities.  Because of that behavioral similarity, marketers can use the two groups to test the effects of various changes in their approach to the market.


A/B split testing has been a mainstay of direct marketers for fifty years and e-commerce practitioners for fifteen.  Let’s illustrate the basic idea with an example from one of these worlds.


If you’re a direct mailer, you can conduct these tests very easily.  Take a direct mail campaign (which classic direct marketers think of as a combination of mailing list, offer, and creative package) that already has an established track record.  Offer M mailed to list N with creative package O has been shown to reliably yield, let’s say, a 1% response rate.  Now the direct marketer can perform an A/B split test very easily.  Simply carve out a statistically significant portion of your list and mail them your new candidate for a creative package.  See how that portion of the list responds compared to the baseline creative package, and you’ve learned whether or not the new creative package is more effective in selling than the original was.  If the new creative package does a 1.1% response rate against the original 1% and if the results are across a large enough set of responses, you’ve now established that you have a more effective creative package for your direct mail offer campaigns.  Now the effective direct mailer has established a new baseline (creative package P) and continues to test and optimize from there.


The online world adopted this basic concept with a vengeance.  Web site operators began testing and measuring as soon as it was feasible and to the full extent that available applications allowed.  Huge businesses were established to serve this need, leading to such high water marks as the acquisition of Omniture by Adobe in 2009 for $1.8 billion and the widespread adoption of Google Analytics, which is used by roughly fifty percent of Alexa’s million most popular web sites.



Tragically, the brick-and-mortar retail channel has been badly disadvantaged in the analytics race.  From a computer science perspective it’s considerably harder to abstract, quantify, and report on what individuals do in physical environments than they do in online or computer interfaces.  Technology needed to significantly advance in its capability regarding input sources, video analytics, and processing huge volumes of Big Data before the same flavor of insight became feasible in physical stores.  One practical consequence of this capabilities gap was that bricks and mortar could not take advantage of A/B split testing methodology at any kind of scale.


The good news is that this problem is now in the past.  It is presently possible for platforms such as RetailNext to provide brick-and-mortar retailers with detailed, reliable, and easily understood measurement of how shoppers behave in their stores and to provide actionable insights that retailers can use to make the changes that will increase sales, reduce theft, or improve the customer experience.  Retailers have begun to use these advances to look more deeply at customer behavior inside the store and to use that evidence to refine their viewpoints on what makes the store succeed.


As this capability makes its way into more retailers, the industry has the opportunity to begin true split testing.  As defined above, a split test divides prospects (in this case shoppers) into homogenous groups and presents them with different stimuli to see how their responses change.  A retailer could easily come up with a sample set of stores that reflect the circumstances present throughout the chain (matching shopper demographics, geographic distribution, store size, form factor, and other important qualities) and then use this set as the “test cell.”  The remainder of the chain becomes your “base cell.”


I don’t want to imply that no testing like this goes on in the retail channel.  However, the frequency and comprehensiveness of this practice is still far, far back from what you see across the online channel.  I believe that owes itself to the fact that measurement tools have been so much blunter and more error-prone.  Now that the ability to measure behavior in stores is catching up with and may ultimately surpass that ability in online stores, I’m optimistic that we’re soon to see a new era of in-store testing, one in which nearly all retailers of any significant scale are continually testing shopper reactions to their stores and making the changes that contribute directly to the bottom line.


So what will retailers want to test?  Nearly every aspect of the store.  They can test product selection, layout, marketing programs, fixtures, signage, pricing, and staffing decisions.  They can categorize measured stores by banner, geography, form factor, and shopper demographic.  And of course each completed test, while showing the retailer specific ways to improve, also reveals new potential tests that can drive that improvement even further.


I personally am a direct marketer from all the way back to the beginning of my career and spent a lot of years optimizing web sites as well.  I can tell you that in all these efforts over many years, the marketing teams I was on never ran out of important questions to ask and worthwhile tests to run.  By the time we had answers to our first set of questions, an entirely new set had emerged in its place.  This practice went on continuously throughout the lifetime of any given product line or company, and every step of the way we became incrementally more accomplished.


The same opportunity sits in front of the community of brick-and-mortar retailers today, and I can’t wait to see all the things we learn.