Co Authors: Mike Jennings and Nick Vanderheyden - BestMark, Inc.
PART I: Mystery Shopping Dashboards, Analytics & Intelligence Systems

Analytics refers to the gathering and interpreting of data in order to make better business decisions and optimize business processes. In mystery shopping, the most common use of analytics revolves around trying to understand how various interpersonal experience attributes influence customer loyalty measures. Correlations can be determined between loyalty and simple attributes from extending a thank you all the way to product knowledge. How important is it that the customer is thanked? How important is it that the associate smiles when greeting the customer? Answers to these questions allow organizations to more effectively allocate resources and focus on the factors that will bring about the greatest return.

In order to draw confident conclusions on various relationships with these target outcomes using mystery shopping data, it’s critical that the mystery shopping program has three elements:

1. Survey Design: A properly designed mystery shopping survey (analysis friendly).
2. Sample Size: A large enough sample size.
3. Skill: A skilled analyst who understands the client’s business and market research analysis.

Survey Design - Three Critical Design Objectives for Analytics
Survey design plays a critical role in enabling effective analysis of mystery shopping data. Each survey should be developed to include three critical design objectives:

1. Measure the shopper’s overall experience rating against an organizations loyalty measures such as likeliness to return or recommend the store to friends and family.
2. Measure key driver attributes known to influence customer loyalty.
3. Measure variations in performance on key driver attributes over time.

Overall Experience Rating - Net Promoter Score, Loyalty Three, Customer Satisfaction Index
To understand the influence of various experience attributes (e.g. greeting, helpfulness, attitude, etc.) on loyalty, the survey form must include questions relating to overall satisfaction or probable future behavior of the shopper based on the experience. An organization’s customer experience metric (Loyalty Three, Net Promoter Score, Customer Satisfaction Index) might encompass questions such as:

» Based on this experience, how likely would you be to return to this location again?
» If you were in the market for this product, how likely would you be to return to this location?
» Based on this experience, how likely would you be to recommend this location to family and / or friends?

Once this information is gathered, shoppers can be divided into segments and analytics can begin. Identifying the differences and similarities between the experiences of shoppers against the anchors of the Net Promoter Score, Loyalty Three, or Customer Satisfaction Index will reveal attributes that drive motivation to return / recommend and those that do not.

Measuring Key Driver Attributes and Capturing Variations in Performance
A high score doesn’t always equate to a shopper’s intent to return. For example, the associate scored a 95% on the mystery shopping form, but the shopper indicated that she was somewhat unlikely to return again based on the experience with the associate. When this occurs, there are typically two possible causes.

The first is that the survey form simply doesn’t include questions that measure key drivers of loyalty / advocacy (e.g. helpfulness, friendliness, knowledge). Instead, it may be very “compliance” heavy, measuring attributes that don’t influence the customer one way or another.

The second potential cause is that the questions designed to measure these attributes fail to capture varying levels of performance.

» Did the associate greet you? Yes / No
» Did the associate ask questions about your needs? Yes / No
» Did the associate make a recommendation? Yes / No
» Did the associate thank you? Yes / No

Using questions and response options like those above means a great greet, an okay greet and a poor greet are all grouped together as a simple “Yes.” A thank you that makes someone feel valued as a customer and one that seems insincere and scripted are both grouped together. As a result, it’s impossible to know how often great greets are occurring. There is also no way to conduct analytics to determine the influence of a great greet on the overall experience.

Sample Size Considerations
It is important to understand the influence of sample size on the accuracy and validity of analytics. For more information on this, see T&A Consultores CEO Marcelo Tarica and Service Evaluation Concepts CEO Arcadio Roselli’s article titled “Sample Size Calculation in Mystery Shopping Programs.”

Analytic Skill / Capabilities
The final component is finding the right resource to perform the analytics. With an adequate sample size and the right data in hand, a skilled analyst should be poised to engage in high value analytics that can drive decision-making. A skilled analyst will uncover relationships between the key driver attributes and loyalty measures. To tell the story of how performance impacts customer experience, the analyst must consider which statistics to use and how to convey the information in the most effective manner.

Statistical Caution and Top-Box
Using the right statistic can make or break the impact of survey data. If a mean of five is reported, the distribution of responses could have been all fives or half tens and half zeros-either way the mean is five. To assume these two scenarios are equivalent is an obvious misuse of the statistical mean as a productive metric. This is where choosing your statistical measure is vital. One common measure is the top-box, the percentage of surveyed customers who assign an attribute the highest rating; this is in contrast to the bottom-box, the percentage who assign the lowest rating. Depending how the attribute is correlated with the overall loyalty measure or outcome (or dependent variable) we can understand where to focus managerial attention.

Keep it Simple
Most managers don’t have hours to spend interpreting data. To proactively counter this time constraint, even the most complex information must be presented in a simple and visually stimulating fashion. Using cutting-edge statistical analysis programs can aid an analyst in creating thriving dashboards that are both relevant and simple.