Using Real-Time Propensity Estimation to Drive Online Sales

Accelerated adoption of online services creates an opportunity for retail organizations to drive growth. While the sudden spike in online sales seen in the early days of the pandemic has receded, consumers are now settling into a new normal where a larger portion of their spend occurs online. Shifts in terms of not just what consumers buy but where they buy it driven by high levels of connectivity, the emergence of the digital native shopper and new appreciation of the flexible fulfillment options provided by many retailers are expected to drive continued expansion of the online market for years to come. Retailers must now either embrace these changes or risk losing out in a rapidly evolving (and increasingly digital) marketplace.

Long gone are the days when simply setting up an online store was enough. In exchange for their time and money, consumers increasingly demand personalized experiences tailored not just to their needs and preferences but the context within which they are engaged. They want the ability to buy but they also want content and support enabling a post-purchase experience that maximizes the value received from their purchases. And they want flexibility in how and when their needs are fulfilled in order to integrate the new retail experience into their increasingly busy lifestyles.

Meeting these expectations is not easy, and just as retailers seem to settle into a new norm for online engagement, competitors seem to raise the bar as they vye for consumers' attention. Keeping up requires retail organizations to engage in a continuous process of reimagining the consumer experience and leaning into data and AI to provide feedback, insights and automation at scale.

Acting on Real-Time Insights Requires Solving Numerous Challenges

While data and AI are the key to delivering a modern online customer experience, grand insights and precise predictions mean nothing if the business cannot act upon them. As we see more and more organizations, especially those engaged in online commerce, fold data-driven capabilities into their operational workflows, moving from insight to action often entails the deployment of real-time capabilities, capable of sensing and responding to opportunities in the moment.

To illustrate this, consider a scenario where customer interactions in the context of an online shopping session signal their propensity to purchase a given item. Using knowledge about the customer, the product, the customer's prior interactions with the product in combination with information about the customer's current behavior, we might predict a customer is unlikely to purchase the item they are currently viewing and suggest alternative items we believe they may more strongly prefer. Or we may predict the customer is highly likely to purchase this item and suggest complementary items that expand the overall size of the cart associated with their eventual purchase. Or maybe the customer is somewhere in the middle and needs the nudge of a small discount, free shipping or simply a reminder of the flexible returns policy associated with the product to get them into the buying mood.

With each click on the website, the customer clarifies their intent. But for us to keep up with this, we need to rapidly process the event information they are generating and update our estimate of their propensity to purchase. The computational challenges of this work are complex, and those complexities are compounded by the fact that customers may engage in long-lived shopping sessions that drag on over days or even weeks, requiring us to keep track of very large volumes of data over long periods of time.

If this sounds like a lot, it is! But thankfully the capabilities needed to meet each of these challenges are found in the Databricks platform.

Databricks Enables the Operationalization of Real-Time Insights

Databricks was built with real-time enablement in mind. Recognizing the difficulties many organizations had in building real-time data processing workflows, Databricks brought the mechanics of traditional, periodic batch processing to real-time workflows with Structured Streaming. With the introduction of the Databricks Photon engine, record-setting high-performance data processing is enabled, allowing organizations not only to keep up with operational data needs but do so at Big Data scales.

Model training and deployment have been simplified with the integration of MLFlow, an open source capability for model monitoring and management. The Databricks Feature Store further simplifies this process by connecting models trained by Data Scientists with the information assets consumed as part of the prediction process. Support for high-performance online feature store capabilities and scalable, hosted Model Serving capabilities ensure that models are not just more easily deployable but can actually support the demands of the operational systems into which they are integrated.

No one is saying meeting consumer demands for personalized, data & AI-enabled experiences is easy, but with Databricks, the technical barriers to delivering these experiences are being addressed.

Talk is Cheap. Let's See It In Action.

Saying these things is easy. Demonstrating them is a whole other thing. Through our Solution Accelerator program, we work to demonstrate end-to-end solutions to real-world problems faced by our customers. We leverage publicly available datasets and make available detailed notebooks and coded assets that walk Data Scientists and Data Engineers through all the ends and outs of getting a solution off the ground.

We are pleased to announce the delivery of our Real-Time Clickstream Propensity solution accelerator, demonstrating how live event data generated by an ecommerce website can be used to predict whether users will complete the purchase of a product with which they are interacting. The assets associated with this accelerator are quite voluminous, but organizations can use them to develop a complete understanding of what it takes to deliver such a scenario and then rapidly deploy a solution tailored to their own organizations needs using the patterns and features demonstrated in the notebooks. If you're ready to dive in, download the notebooks today.