Announcing the General Availability of Lakeflow Connect

We’re excited to announce the General Availability of Lakeflow Connect for Salesforce and Workday. Lakeflow Connect introduces no-code ingestion connectors for popular SaaS applications, databases, and file sources—enabling you to build scalable ingestion pipelines and unlock the full power of your data with the Databricks Data Intelligence Platform. The Salesforce and Workday connectors are the first managed connectors to be released to GA since the Lakeflow Connect announcement last July, with many more on the way.

Lakeflow Connect’s managed data connectors are powered by serverless compute and are available across AWS, Azure and GCP. They are fully integrated with the Data Intelligence Platform for governance, observability and enterprise security.

Lakeflow Connect is the first of three components launched as part of Lakeflow, the unified engineering solution for ingestion, transformation and orchestration. It works seamlessly with DLT for transformation and Databricks Workflows for orchestration.

Early adopters like Porsche Holding, Insulet, Equiniti, Stroër SE & Co. and Flick are already using Lakeflow Connect to solve some of their toughest data challenges.

Data silos cause ingestion complexity, inefficiency

Today, it’s difficult to build and maintain ingestion pipelines for SaaS applications yourself because each API is unique and changes often. Meanwhile, database ingestion can be even more complex, often requiring extra infrastructure such as virtual machines for extraction tools, Kafka for streaming, and a strong understanding of change data capture (CDC) technologies for efficient ingestion.

As a result, organizations spend a lot of time building, optimizing, and maintaining a patchwork of pipelines. This slows down projects and can create a fragmented ETL architecture that’s difficult to govern. It also forces highly skilled data engineers to spend time moving data and monitoring pipelines instead of developing strategic innovations.

Built-in data ingestion support for Salesforce and Workday

Lakeflow Connect eliminates these challenges by making it easy to create production-ready pipelines with a few clicks in the UI or a few lines of code. Under the hood, ingestion is efficient, with incremental updates and optimized API usage. As your managed pipelines run, we take care of schema evolution, seamless third-party API upgrades, and comprehensive observability with built-in alerts.

"With the Salesforce connector, we successfully decommissioned four services, which significantly streamlined our tech stack and lowered our costs. This integration not only simplified our architecture but also improved our operational efficiency, allowing us to focus on delivering more value with fewer resources." 

— Simon Salmeri, Head of Data Platform, Ströer

Lakeflow Connect for Salesforce allows you to pair Databricks' advanced analytics and AI tools with Salesforce's CRM data to build a more effective seller experience: predicting sales trends, deepening insights into customer behaviors, and personalizing engagement strategies. The connector supports Salesforce-specific requirements, like custom objects and formula fields.

“We have been using the Lakeflow Connect Salesforce connector for a marketing segmentation use case. Our Salesforce data holds crucial information about our customers, and the connector has been the missing piece to bring this dataset into our lakehouse.… We are now at just the beginning of understanding our customers better and improving their experience.” 

— Ana-Maria Mocanu, Data and Insights Lead, Flick

Databricks and Salesforce announced a strategic partnership in 2023 to build robust integrations for our joint customers. Since then, we launched the zero-copy Salesforce Data Cloud Connector so customers can discover, query, and govern Salesforce Data Cloud directly from Unity Catalog. Paired with Lakeflow Connect for Salesforce, customers can use both their CRM data and their CDP data within Databricks.

Lakeflow Connect for Workday also provides simple, low-maintenance ingestion. You can access custom Workday reports for centralized analytics on your employee workforce.

Both connectors seamlessly integrate with the Data Intelligence Platform, so you can use:

• DLT to transform data declaratively and update transformations incrementally as data continues to stream in
• Databricks Workflows to orchestrate these pipelines
• Databricks Asset Bundles (DABs) to manage continuous integration and delivery
• Advanced analytics and machine learning tools to drive business value
• Unity Catalog to govern your data and pipelines
• Lakehouse Monitoring to maintain the quality of your data

Driving high-value business use cases

Lakeflow Connect helps unlock high-value use cases across various industries, such as understanding customer behavior, analyzing HR data, or identifying upsell opportunities.

A common use case is predicting customer churn. In the video below, we demonstrate how a retailer can do this by ingesting Salesforce customer order data and combining it with customer interactions across other channels for enriched insights.

Demo: How to use Lakeflow Connect for Salesforce to predict customer churn.

Another use case is creating a customer support dashboard for executives and support teams. You can even use LLMs to diagnose the trends that appear in the dashboard, as shown in this step-by-step blog post.

For human resources departments, the Workday connector can help track and predict HR trends, such as employee turnover. For example, you can ingest relevant Workday reports, transform them with DLT, and incorporate the results into an AI/BI dashboard.

Hundreds of customers using Lakeflow Connect

Customers are already ingesting data from a variety of sources with Lakeflow Connect.

As one of the world’s most successful automotive companies, Porsche Holding Salzburg (Porsche) is Europe’s largest automotive retail company with operations in 23 countries. Porsche wanted to address their data integration challenges and streamline access to their customer data in Salesforce Sales Cloud. Using Lakeflow Connect, Porsche was able to ingest data from Salesforce, transform it in batch and streaming modes, and deploy and operate the pipelines in production. Today, the Databricks Data Intelligence Platform forms the core of the Porsche group’s Data Platform. Learn more about Porsche’s success using Lakeflow Connect.

“Using the Salesforce connector from Lakeflow Connect helps us close a critical gap for Porsche from the business side on ease of use and price. On the customer side, we're able to create a completely new customer experience that strengthens the bond between Porsche and the customer with a unified and not fragmented customer journey.” 

— Lucas Salzburger, Project Manager, Porsche Holding

Insulet is a medical device company that developed the Omnipod product platform, and they were an early adopter of Lakeflow Connect. With a mission to simplify life for people with diabetes and other conditions, they sought to tap into their CRM insights to better understand their customer needs. They used the Salesforce connector to reduce their ETL latency from days to minutes—improving the resolution process for their customers with near real-time insights.

“The LakeFlow Connect’s Salesforce connector was especially important since it allowed us to directly ingest customer feedback, sales data and service interactions from the Salesforce CRM into Databricks. LakeFlow Connect completely changed how we handle Salesforce data. We got rid of unreliable middleware and can now analyze everything directly in Databricks - faster and with less manual effort.”

— Bill Whiteley, Senior Director of AI, Analytics and Advanced Algorithms, Insulet

Equiniti (EQ) is a global leader in shareholder, pension, and remediation services, and they wanted to centralize data and insights into their operations. Using Lakeflow Connect for Salesforce, Workday, and SQL Server, EQ seamlessly solved for ingestion—thus freeing up critical engineering talent to focus on using Mosaic AI to enhance the customer experience for their 6,000 companies across 136 countries. For EQ, the ability to seamlessly create no-code integrations to their core platforms helped them save time and additional costs, letting them focus on value multipliers like Gen AI applications. Learn how EQ built a secure and unified data and AI platform in three steps.

Getting started with Lakeflow Connect

Data teams can now use Lakeflow Connect to build efficient, incremental pipelines at scale, with just a few clicks or lines of code. Get started today with the Salesforce or Workday connector to help unlock high-value use cases, such as predicting customer churn, running personalized customer analytics, and identifying upsell opportunities.

Lakeflow Connect’s roadmap includes additional connectors like SQL Server, Google Analytics, ServiceNow, SFTP, SharePoint, and PostgreSQL. Stay tuned for more announcements this year, or reach out to your Databricks account team if you’re interested in joining a preview for early access.

Lakeflow Connect has a compute-based pricing model. Salesforce and Workday, which run exclusively on serverless infrastructure will incur serverless DLT DBT charges. See more details on rates.

To get started with Lakeflow Connect for Salesforce and Workday, you can review the documentation or try this solution accelerator.