Real-Time Retail

Try Databricks for free

What is real-time data for Retail?

Real-time retail is real-time access to data. Moving from batch-oriented access, analysis and compute will allow data to be “always on,” therefore driving accurate, timely decisions and business intelligence. Real-time use cases, such as demand forecasting, personalization, on-shelf availability, arrival time prediction and order picking and consolidation, provide value to the organization through improved supply chain agility, reduced cost to serve, optimized product availability, and stocking replenishment.

We witnessed the move to e-commerce and omni-channel commerce happen over the past 20 years, only to see consumer behaviors fundamentally shift once the COVID-19 pandemic occurred. In just 10 weeks, we witnessed a rate of change that took the previous 10 years to accomplish. As physical stores faced lockdown orders, consumers shifted purchasing to digital channels. Restaurants saw internal dining evaporate, while drive-thru and delivery skyrocketed. With the shift in dollars came other changes: increased fraud, shifting customer expectations, a higher volume of returns, and increased costs to serve customers curb-side service and delivery.
Compounding consumer-led changes has been the recent volatility in supply chains. The greatest risk to retail and consumer goods for the next several years is volatility.
Legacy business strategies were instantly outdated - demand forecasting predicted the wrong demand, customer preferences changed, which precipitated stock outages, hence retailer’s margins suffered. As consumers purchased in real-time, businesses had to shift their outdated data warehouse architectures to those that could run and respond in real-time - hence Lakehouse for Retail.

Rapid data ingestion at scale makes advanced insights available across the value chain in real-time, reducing costs and minimizing errors. Retailers make mistakes when they make decisions without information. These mistakes can manifest in many ways, including some of the following:
- Underestimating demand leads to expedited shipping costs for rush delivery
- Incorrectly predicting how much of an item to produce leads to excess carrying costs, missed sales, and higher waste
- Reacting to breakdowns leads to unplanned outages that disrupt full production cycles
- Order fulfillment with incomplete or inaccurate data leads to additional shipping costs or higher rates of return
- Missing an opportunity to engage a consumer based on current data leads to missed sales opportunities
Processing data in real-time enables all parts of the value chain to see the status of operations without delay and make better-informed decisions that help avoid these problems.

Databricks’ Lakehouse uses technologies that include Delta, Delta Live Tables, Autoloader and Photon to enable customers to make data available for real-time decisions.
Lakehouse for Retail supports the largest data jobs at near real-time intervals. For example, customers are bringing nearly 400 million events per day from transactional log systems at 15-second intervals. Because of the disruption to reporting and analysis that occurs during data processing, most retail customers load data to their data warehouse during a nightly batch. Some companies are even loading data weekly or monthly.
A Lakehouse event-driven architecture provides a simpler method of ingesting and processing batch and streaming data than legacy approaches, such as lambda architectures. This architecture handles the change data capture and provides ACID compliance to transactions.
Delta Live Tables simplifies the creation of data pipelines and automatically builds in lineage to assist with ongoing management.
The Lakehouse allows for real-time stream ingestion of data and analytics on streaming data. Data warehouses require the extraction, transformation, loading, and additional extraction from the data warehouse to perform any analytics.
Photon provides record-setting query performance, enabling users to query even the largest of data sets to power real-time decisions in BI tools.