Introducing Lakehouse for Media & Entertainment

There are few industries that have been disrupted more by the digital age than media & entertainment. For decades, media organizations acted as wholesalers for content, which was a vehicle monetized mostly through advertising – with very little focus on the consumer side. Beyond the advent of cable in the 1980s, broadcasting, outdoor, publishing and entertainment saw very little change over a long period of time. Then came digital.

The rise of FANG companies has heightened consumer expectations around smarter, personalized experiences, making data and AI table stakes for success. Brands have shifted their ad budgets to digital channels such as connected TV, mobile and search advertising to more definitively target their ad spend, while also driving compliance with increasing privacy regulations.

Driving better AI outcomes for consumers, advertisers and employees is now a board-level initiative for most media and entertainment companies. The problem? Traditional data architectures weren’t built to support AI/ML use cases, especially across broad teams of data engineers, data scientists and analysts, while supporting the scale and agility media companies need to support evolving customer demands. This has led to heavy investments in more modern data technologies and industry partnerships that help organizations use data more thoughtfully to shape the entire consumer, advertising and content lifecycle. This is achieved by:

• Having a single view of all data in a single architecture, including unstructured data like video, images and voice content.
• Ensuring data is in a ready state for all analytics and AI/ML use cases.
• Having a cloud infrastructure environment based on open source and open standards so IT and data teams can move with agility.

In a nutshell, ensuring all of your data is AI and business intelligence (BI) ready and being able to move fast to stay ahead of consumer and employee expectations is a critical strategy for every media organization.

Introducing the Lakehouse for Media & Entertainment

Today, we are thrilled to announce the Lakehouse for Media & Entertainment (M&E), which enables organizations across the media ecosystem to deliver better outcomes for consumers, advertisers, partners and employees with the power of data and AI. By eliminating the technical limitations of legacy systems, the Lakehouse for M&E empowers organizations to leverage all of their data to build a holistic view of consumers and advertisers, make real-time decisions and drive innovation in engagement and advertising outcomes with advanced analytics.

So, why is Lakehouse for M&E critical for success? Through purpose-built capabilities, such as solution accelerators, libraries for common use cases and a certified ecosystem of partners, the platform brings together learnings from industry innovators to foster collaboration and accelerate analytics and AI use cases that provide the ability to personalize, monetize and innovate the consumer and content lifecycle. Here are the biggest challenges around transforming into a data-driven M&E organization (and how Lakehouse addresses them):

Creating a unified audience profile

Audience data has traditionally been captured, stored and managed directly in disparate systems (e.g., DMP, ESP, data lake, data warehouse), depending on size/granularity, intended use case(s), and data types. This siloed approach is incredibly complex, especially when it comes to managing customer data as an asset that can be used to support a variety of use cases (e.g., content recommendations, next best offer).

How Lakehouse Helps: Lakehouse supports the use of all data types (structured, unstructured and semi-structured) with Delta Lake and Apache Spark™ at the foundation and data stored in an open-source format that prevents vendor lock-in. Additionally, Databricks provides technical assets in the form of notebooks, deployment guides and reference architectures to help customers stand up new use cases in days to weeks – not months – specifically aligned to helping organizations build and maintain their audience profiles. And as data sharing becomes critical to every media organization, Delta Sharing provides an open-source sharing capability that promotes data collaboration.

Delivering a 1:1 user experience

A byproduct of media consumers having more choice than ever before is that delivering a flawless customer experience is now merely table stakes. At the same time, doing so requires being able to identify the quality of service issues in near real-time, a capability that is not directly supported by the existing tech stack at many companies. Legacy data warehouses cannot support data processing at B2C scale, nor are they the right place to handle streaming ML workloads for real-time consumer lifecycle use cases.

How Lakehouse Helps: The Lakehouse for Media & Entertainment overcomes these challenges with a scalable platform built in the cloud with:

• Lightning-fast performance at B2C scale. With Spark and Delta Lake – the defacto enterprise standards for driving more performance and reliability for data at massive scale – under the hood, the Lakehouse delivers massive scale and speed. And because it’s optimized with performance features like indexing and caching, Databricks customers have seen ETL workloads execute up to 50% faster.
• Elastic cloud scale. Built in the cloud, the Databricks Lakehouse provides scalable resources at the click of a button to meet the demands of any sized job. Autoscaling compute clusters scale up or down based on the size of your workload so you only use as much processing power as needed to meet the demands of your workloads.

Moving beyond aggregation to advanced analytics

Prior to using analytical techniques, such as media mix modeling for spend optimization or survival analysis for churn mitigation, a big lift is often needed to acquire and harmonize at scale. In some cases, this work requires a capital investment and cross-team coordination.

The Lakehouse for M&E combines your consumer, content, advertiser and operational data with a full suite of capabilities to deliver on all of your analytics and AI use cases.

• Ability to Handle All Data: Lakehouse has an end-to-end environment for unstructured data workflows – a query engine built around Delta Lake, fast annotation tools, and a powerful ML compute environment. This allows users to unlock the value of unstructured data, an impossibility for most data warehousing solutions.
• Collaborative data science: The Lakehouse provides an interactive notebook environment that enables cross-functional teams to collaborate on data products with a wide range of analytics and ML capabilities, including support for multiple languages (R, Python, SQL and Scala) and popular ML libraries.
• Easily manage the ML lifecycle: Manage the complete ML lifecycle from model development through deployment with managed MLflow. Centralize models and features in the registry so teams can easily collaborate on highly iterative data science projects and reuse existing work.

Driving value with the Lakehouse for Media & Entertainment

The Lakehouse for M&E builds off learnings from industry innovators to foster collaboration and provide the ability to personalize, monetize and innovate around the consumer and content lifecycle.

Pre-built solution accelerators for media & entertainment

Built on top of Lakehouse for M&E, Databricks and our ecosystem of partners offer packaged solution accelerators to help organizations tackle the most common and high-value use cases in the industry. Popular accelerators include:

• Multi-Touch Attribution: Measure ad effectiveness and optimize marketing spend with better channel attribution
• Gamer/User Toxicity: Foster healthier user communities with real-time detection of toxic language and behavior
• Behavioral Segmentation: Create advanced segments to drive better purchasing predictions based on behaviors
• Recommendation Engines: Increase conversions and engagement with personalized omnichannel recommendations
• Video Quality of Experience: Analyze batch and streaming data to ensure a performant content experience for streaming services

A Growing partner ecosystem

Databricks and AWS: Databricks is working with industry-leading cloud, consulting and technology partners to enable best-in-class solutions. We have a long-standing relationship with AWS helping customers across the media industry ecosystem deliver real-time audience experiences, better advertiser outcomes and derive more value from their digital media assets. Databricks and AWS have hundred of joint Lakehouse customers, including Sega, which is delivering the next generation of 1:1 gamer experiences at scale; Discovery which is focused on frictionless, smarter experiences for viewers around the glove; and Acxiom which is helping its customers collect and activate personalization anywhere, anytime and on any channel.

Databricks M&E Implementation Partners: Databricks has also partnered with system integrators to deliver scalable industry solutions that help customers more rapidly address common use cases:

• Cognizant has jointly built a streaming quality of experience solution that enables customers to mitigate video quality issues that drive viewers to churn. Cognizant’s solution pairs fine-grained telemetry data with AI/ML to quickly identify and remedy video quality issues in near real-time.
• We have partnered with Lovelytics on a sports and entertainment analytics solution that brings streaming data to life. With AI and predictive analytics to predict and forecast performance, the Lovelytics solution enables sports and entertainment organizations to optimize strategy in-game, as well as the fan and live event experience.

Databricks M&E Technology Partners: Our technology partners are critical to success and augment Databricks with industry-specific capability.

• Labelbox – a leading data training platform for machine learning and the first company Databricks invested in as part of Databricks Ventures – helps media organizations label and derive actionable insights from their unstructured video and images files, which has historically been a massive challenge for media organizations.
• As a data integration platform, Fivetran helps our media customers connect to the dozens of ad and mar tech data sources in their organization so they can better understand and activate the data coming from various sources in their media ecosystem.

Want learn more about Lakehouse for Media & Entertainment? Click here for our solutions page. We could not be more excited to launch the Lakehouse for Media & Entertainment as we seek to help media leaders put data, AI and analytics at the very center of their organization.