Endeavour Groups Limited, is Australia’s largest drinks retailer with over 2000 venues across a wide range of brands. The EGL team has built a personalisation marketing engine using Databricks that performs segmentation of EGLs 4.5 million loyalty members. Personalisation is one of most important strategic investments in EGL and is expected to significantly increase revenue and improve customer experience.
This session will dive into how EGL has been able to develop effective brand campaigns that target the right people, at the right time with the right messages to millions of their loyalty members. EGL utilizes cutting edge reinforcement learning as a paradigm for Personalization Engine based on data. The Engine elegantly links marketing logics, AI and business processes together, and strives to intelligently determine the balance between the discovery and relevancy of email campaigns.
Xi Liang: Hello everyone. I’m Xi Liang from Endeavour Drinks. I’m very excited to bring to you our recent work on AI driven personalized email marketing. We have studied our work on personalized marketing last November and aware we’re using data breaks to empower all the design implementation of our personalized marketing. Let me start away on introduction on Endeavour Group Limited. We call ourselves Endeavour Drinks, within our company. It is Australia’s largest drinking retailer with over 2000 venue across a wider range of brands. For example, Dan Murphy’s, if you’re in Australia, you would know at Dan Murphy’s and BWS. Our data science team together with another cross-function team is building a personalized marketing engine that performs segmentation for different group of people. With targeted a message we have around a 4.5 million customers. That gave us a large number of people to support. It is one of the most important strategic investment in our company.
And we are responsible to bring to a significant increase of revenue and improve our customer experience. So in this session, we will talk about how we leverage our customize a personalization engine to build email marketing that target the right customer at the right time with the right product, with the right email. So in our work we use cutting edge reinforcement learning to build our engine. I will show you a more detail next. Today, I will talk about four things. What problem we are solving and how do we use AI to drive a personalized and marketing. What’s our solution look like and what challenges and how we solved them. So you could learn from our mistakes that made along the way, and then you have a more smooth way.
Now let’s start from the first one. What are the problem with solving. Effectively personalized marketing, we are solving four major problems. One is what is the theme of the email, whether it is about recommending a new product or product that you have not purchased before or have purchased it before or a new catalog, a new promotion, and this is a first a question. And the second is what product should be in the email for different email. This is about relevancy email, for example, maybe this is a product that you want to buy it again. Then we’ll have a different mechanism to find out what are the products that should be there. So that’s the second question. Another one is whom should send to their email to. So those email obviously are customized. So we have to determine who should receive what email and other one is when to send it the email.
So customer have a different preference in terms of when they open the email, click the product. So I have to figure out when to send to the email. Let’s start the problem one by one. The first one is what product does should the way show to whom. And this one is relatively easy. The way we solve it is to actually boost a model to rank the likelihood of people purchase a product. We’ll look at their past history, we’ll look at demographics. Those are the main source of information we have. We do look at other information for example, the marketing engagement, whether the product is on sale or any other promotion that is about to happen next a few weeks. So this is the first we do, this is also a product that would build at the very beginning as one of the business as usual campaign.
Before we get into personalization, we have this model to help us to target to the customer on the product that they have purchased before. Then the next one is, if we want to introduce a product that a customer haven’t purchased before, these become much harder, because if you introduce a new product, especially liquor, it can be a little bit hard to make a decision, oh, do I like the flavor or not? Especially for beer, people tend to stick with the same beer wherefore spirits or wine, they are more open to new options. So when we solve this problem, we look at marketing responses. For example, whether customer have a click of this product but have not purchased, and whether this product in their wishlist and whether this product ever in there, but they’re in the cart, or when they go online shopping, they have put them into the shopping cart, but they never pay for this.
So those are very good signal for us to capture in making the recommendations. Similarly, we also use XGBoost to generate the results. We also ranked the likelihood of purchase for the product which customer are more likely to buy. We recommended that product to customer. Next one is the hardest one is, what are the theme of the email? We have about 2000 different types of emails for customer to choose from, while for the engine to push to customer. For example, there’s, some of the email is about mystery gifts for somewhere. If you want a gift for yourself, we send a package to you while you do not know what exactly inside. Some of them is about a product that you used to buy is on promotion. Some of them is to recommend a new product for you to explore. This is the hardest problem we need to solve. We’re using a multi-armed bandit algorithm to solve that. There are two layer of information we’ll look at.
One is how likely you’re going to click the product in the email. The more likely you click the product, the more likely we’re going to send this email to you. Another layer of information is how likely you are going to buy the product in the email. Obviously those two are correlated, but not necessarily you’ll click and you will buy. So we have additional logic to enhance whether this person will buy the product after click it. So I will talk a little bit more how multi-armed bandit works and how we’re using Assess It in our email campaign. The last one is how to determine when is the best time to send to them. At the moment we haven’t set up any AI components in yet because most of the customer are relatively flexible. At the moment we send between three to five email to them at random time slots.
However, we did realize in the last a couple of days, it is quite important to find the right time. So as next step, we will work out what is the optimal frequency and the right time spot to sent to them so it does not annoy our customer to unsubscribe. We would love to give them a pleasant experience of shopping at the same time do not overwhelm with more information than necessary. So in our personalization engine, there are primarily three major component that I’m going to introduce today. Actually, in our engine, we have probably around 20 for modules to make business decisions, but those three of the most important one for our engine to make decisions. The first one is action incubator. What it does is a generator or different email content. What are the message, copy and image. All those are done through Airtable.
For example, we have a marketing team, they use Airtable to produce all those email content. What are the theme? Emotion of the email. For example, whether this email is a funny, it is about, when it is some kind of gift occasion, the message might sound very funny, very pleasant. If it is a mother day, there is full of love. So those are different variations of email content, and a copy and image we usually show. That’s probably one of the most important elements of the whole engine. So the engine can be smart, can try find it the right content, but if we do not have the right content to start with, the engine simply does not work. Then let’s look at the personalization engine, the key component of the brain. Obviously it will determine all the time or products and which emails will send. I would go to a little bit detail here. It’s a very sophisticated business logic. After that, we have dedicated a section for one-to-one channel. Basically it is connected to our email channel to send to the right output.
There are several examples about Christmas contents. For different people, they say different contents, and that they have very different message as well. For example, if it tastes great, why change it? So this is about to talk about a product that people have purchased before. Then this message is about have a different variation, instead of always saying, why don’t you buy it again? It just say, if it tastes great why change it? So four different people will have a variety of message to tailor their way of taking informations. And if you look at the right one, drinks discovery simplified for this customer, he has not drank this liquor before, but we think it maybe interesting. So this will be a different variation of the message that they’re receiving.
So this page, it just shows what a customer previously can not see, but can see now this example, we have a premium or whiskey lover on the left. This is what this customer has seen before. Previously only receive member statements, this customer named Lisa, and then the product she received is all generic list at a segment level. Segment level means what a product type for example is beer, or it is wine. So it does not really have any personal touch there. It will also adjust to look under their past purchase history as well, because this is only a recommend or something we think they are relevant with, very rough with this logic. After we have enabled a personalization, this is what the customer is saying. Basically, the message is around spirits, because we understand her preference is the whiskey, at the same time, the product we recommend feed to her, usual pattern as well. She is now receiving a whiskey product at a price tag of $350. That is what she used to have in the past. And we look at a different product.
Skew means the product, we consider the price. We look at the region where she usually shopping and then the feature of the product as well. And this is some email or action. We call it action as email. So this as email that suggested, does not exist if before we have our personalization. So after we have personalization, the customer received much more relevant product and information for them. So this is a big comparison between the business as usual on left and the personalization of the right. Now, that’s on the high level to have a good understanding of our products. Now let’s get into some technical details.
This is multi-armed bandit problem that we’re solving. Basically, we try to see, out of all those email templates, which are the email we should send. So if we think about this problem, it will take some time to find out what email works, basically we have a 2000 different email. How do we know which one we should send? And how long do we have to try different email to learn enough about the customer’s preference? So this is a multi-arm bandit problem. If I put it where I come from, basically, if you think this problem have a different slot to machine, and you’ll want to understand which machine will give you most amount of money. And then you’ve spent probably one hour just to play different machines, then you’ll figure out, okay, the first machine have a current success rate of 70% and the second one have 30% or 50%.
Well, now your call is whether which one you want to use. Some people may say, I want to use this one because I know it gave me the best return. However, it can come with the issue that one, you might not have enough understanding of the machine, to understand that the probability of success. So you might also want to send it to this machine as well. So you have more opportunity to learn because the longer you’ll learn, the more likely to have a better understanding the success rate. So the dilemma we’re facing here is we want to get our return as fast as possible and at the same time, we want to leave some room to explore. So we do not want to quickly narrow down the focus to limited number of email for people. So they do not end up receive the same email over and over again, without realizing there are some niche, email, or product that they can explore.
So this is the multi-armed bandit and the problem we try to solve. Effectively, we use Thompson algorithm to try to balance it between the exploration, to find out more opportunities and exploitation to get the optimal reward. So we have to get our decision making and optimization so much. The Thompson algorithm is our choice to solve this problem. And from the result we see it is very effective. Another decision we have to make is whether we want to have a cold start or warm start. Cold start means we just randomly send everyone the email template. We have 2000 different ones, and we just decide to randomly to send to everybody. This is called cold start. Warm start means we want to learn something about our customers and then decide at which template to send to them. Warm start, we have to use the business as usual email and send to them to understand.
However, the cold start, we call it as airplane, it is slow, it is big, and as a warm start is, we have already got something to start with. We are ready to go. This is accelerate. It is fast at first, but experience tell us the cold start will win in the end, because if we sand randomly to everybody, we can collect more information. With warm start, you do not have that opportunity. You can get early wins, but longterm it is a cold start win, because we spend enough time to randomize, to process. So customer can get all the information and we can collect the information as well.
Now, I want to further down to talk a little bit more about exploration and experimentation for the new users. How do I learn what is relevant, and for new offers, how many different new aspects to learn about, for example, styling a product, incentive, type, message, creativity, or sequencing. So for all of those things we have to consider. And as the exploring is, we do much about them. So we need a lot of experimental learn in order to increase the longterm return. And exploitation here is, we have to be confident with what works over time, and it be confidential for different email at a different time. So we can pick the best email and we can maximize the short-term gain. So now the dilemma we have is really try to balance the long-term and the short-term return.
So for us, it is very hard to balance between the exploration and exploitation. This is a very typical problem for solving multi-armed bandit problem. I believe in the current academia, there is no easy way that can successfully measures of optimal balance between the two because of the different business of problem settings. So there is no rule of them, but we spend quite a lot of time to come up with the right metrics, to monitor and assess and finalize what is the right balance between them. In the chart, what you will see the video is three things. One is relevant emails. Basically this is to send email message on product that they have purchased before, which are usually very effective email because people tend to click email that they have seen before, especially the product and the red bubble is the discovery email. The size shows how likely people are going to buy it and how many we send to customers. So the discovery obviously is feel what customer click it because that something new. And it will also have some generic email as well.
For example, there is no preference about new or old product. So this is about a generic. And in this chart, what we show is, we’re trying to measure what’s the center boost by having different balance and their owns, Y axis what’s the experiment cost. Basically we try to minimize the cost and then boost to the send. Let me click so you can observe the pattern. So have the first view. I will play it a couple of times. Basically, you can see the relevance action. We learn it really fast. So we center boost a lot of them basically were sent to many people and they move very fast as well. The more faster, means it learns really fast. Let me play it again. So you can just focus on the single color, and then you will see the comparison, how fast it learns compared to other actions.
As you can see it learns faster. Another one I want to, or hopefully to get your attention next around to play it again, is to see the variation at a different stage. When I generated those chart, we’re at the stage of introducing many different email templates. So we are still bringing a lot of new emails. So we always have to consider, we have to make sure while we have a new emails, it does not change the learning or knowledge from an old email. So you will see a little bit changes. So this help us to make sure we have a good understanding how the new email impacted the current pattern. So this round, I hope you can have a look at the turbulence, but at the same time, we have to make sure they are at the level of the balanced that are we comfortable with.
in the video replay, you probably be able to see there are different things you can look at in this chart. It has about six different dimension of information. I want to go through this time about a year, we found this probably one of the most advanced visualization assessment for multi-armed bandit problem. Now the last page, it shows we have a very good way to use automated method to make a decision. However, sometime the business logic are more important than the algorithm. For example, while we have to send a certain message, for example, Easter or some other announcement, we will have to make sure this go to a lot of business partners and customers as well. We will have to up weight certain email, making sure those email gets sent to customer and we’ll have to find a smart way to do that.
We operate a certain action or email so they can get a higher ranking. And this is the one way we do that. For example, if not one email is about a beer, then this chart just to show what are the up weight we should apply. And what are the corresponding additional emails and to receive. Then our email marketing team can find out if the marketing team decided we need to send additional 250K to customer, then he can find out, okay, the corresponding app wait, is about this level. So he can tell the engine and he needs to up weight this template or the email action to this weight. So this is one way to learn example only to bring additional business logic to our algorithm. There are many more, but for the time limitation, I show one that’s here.
So there are other business problem that we are facing, but we have to solve one by one. I just released several of them. That is very important for us to do. For example, are we sending too many emails that cost our subscription? All subscription obviously hurt business account, and also it does not bring good customer experience as well. So the way we do that is try to test out a different frequency. How many emails is out of the comfortable level. So we can optimize for that cohort of people. Then if certain email are very successful, the machine tend to send to the same email over and over again, or different variation of a similar email. In this case, the model learns it is very effective, but it can be very annoying if customer are getting similar email over and over again.
So we have to implement some anti-repetition rule so that it will limit the time people see the similar emails, obviously the engine, if you gave the engine to learn long enough, it will find it cannot do that, but just not fast enough. So we have to implement some anti-repetition business rule. And also how do we consider the customer journey? Now, what I showed there is only consider at a certain time point what as the email, however, it is the journey that customer with us. We have to make sure the whole journey is a pleasant. Not only at a certain time point. Also customer can be at a different stage, for example, certain customer might leave company to a different company because their provider certain services, then we have to make sure we have very different business target for those customers.
For example, the focus is not a sales anymore. It is about retention. Then we have to impose different models for different cohorts, which at the moment we haven’t got. Then another one is how do we make sure we have a consistent message across different channels? And now the focus is on email. It can be extremely challenging to have the same message across different channel because, the different channel, we found the message are very different in terms of their efficiency. So how can we balance across different channel? That’s a big problem for us to think about. And the last one is how do we choose email that a model has never seen before? This is very challenging. So we have to make sure for the email that people haven’t sent a scene before we have to have a upper hand how many email people can see. Sorry, how many people will receive this email.
So those are several things that we are trying to work on, which obviously is beyond what model can solve. We have to impulse the business logic there as well. So for next, we have a lot of plan in our to do list in terms of personalization. There are several example we are looking to, and we have started some of them, but have not finished. Personalized landing page, searching result, customer review, and offer, and basket building. Basket, building means, we can pre-populate some products into a bucket and when people open the email, they already see all the list of the product with the estimated promotion discount of those product as well. And also customer retention and a journey as I just said, which is one of the biggest element we want to tackle next. And also app push notification, basically at a right time when they get close to our store, we can send some push notifications for them. This is something we are already working on. Okay, that’s finish my presentation today. Thank you very much. Now is open to questions.
Xi Liang is Data Science Lead at Endeavour Drinks, who has more than 8 years' experience in data science and software engineering. She built a data science team from scratch at Endeavour Drinks. Prior...