Bringing real-life insights to retail stores

Data Science / Machine Learning / shopping behaviour / fuzzy string / hierarchical clustering algorithm / kernel density estimator

by Faye Cheung, ASI Fellow 2016

Understanding shopping behaviour has played a major role in the success and growth of online shopping. Online retailers are able to watch their users’ web activity - search terms, number of clicks and so on - which allows them to target advertising and product recommendations to each customer.

Now, imagine you could translate this into real life: what kind of insights could we gain observing people as they shop?

I took part in the ASI Data Science Fellowship in October 2016 and worked with Aimia, a marketing and loyalty analytics company, to find out.

Aimia recently conducted an experiment where regular shoppers at a supermarket volunteered to have their in-store locations recorded at two-second intervals as they shopped.

Armed with 875 shopping paths recorded during the experiment, the goal of my project was to determine if distinct groups of shopping behaviour could be observed.

I labelled the areas of the store with letters of the alphabet and convert each path into a sequence of letters representing the areas a shopper passing through, followed by a fuzzy string matching to determine the pair-wise difference between each sequence. This was fed into a hierarchical clustering algorithm to produce four distinct groups of shoppers.

This ‘cob-web’ plot shows every path in one of the clusters overlaid on top of each other. Visually, it looks like a nice sketch - although slightly haunting. In more quantitative terms, it's not particularly useful to anyone. What we need is a visualisation that more readily identifies the areas of the store that these shoppers visit the most.

I used a two-dimensional kernel density estimator (scipy.stats.gaussian_kde) to produce a heat map from the (x,y) coordinate of every path in the cluster. The red regions indicate the parts of the store with the highest footfall. Here are the heat maps for the four clusters:

I obtained a more complete picture of each shopping group by studying what the shoppers in each cluster were likely to buy as well as the overall features of their visit such as the mean time in the store and the rate of visiting a store zone more than once.

The heat maps are ordered from left to right in order of mean path length. Unsurprisingly, the farther the distance walked, the deeper into the store shoppers went (the entrance is at the top left hand corner of the heat maps).

The data from this short experiment demonstrates that recording the in-store location data of shoppers could serve as the real-world equivalent of activity tracking that online stores perform.

This opens new door on how brand use customer insight to build incentive strategies and loyalty programmes to keep customer engaged.

In conclusion, the long-term vision for this project is to be able to determine these shopping behaviours in real time, and deliver promotional and advertising messages to customers as if they shopped through their mobile phone app.

Brick and mortar stores, open your doors to the new shopping revolution!

Faye Cheung took part in the ASI Fellowship September 2016. Prior to the Fellowship she completed a PhD in Experimental Particle Physics at the University of Oxford.

Share on: Twitter, Facebook, LinkedIn or Google+