Attack of the Delta Clones (Against Disaster Recovery Availability Complexity)
Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. Notebook: Using Deep Clone...
This is a collaborative post from Databricks and Airbyte. We thank Simon Späti, Data Engineer & Technical Author at Airbyte, for their contributions.
Today, we are thrilled to announce a native integration with Airbyte Cloud, which allows data replication from any source into Databricks for all data, analytics, and ML workloads. Airbyte Cloud, a hosted service made by Airbyte, provides an integration platform that can scale with your custom or high-volume needs, from large databases to a long-tail of API sources. This integration with Databricks helps break down data silos by letting users replicate data into the Databricks Lakehouse Destination to process, store, and expose data throughout your organization.
As an open source standard for ELT, Airbyte provides more than 150 editable pre-built connectors – or easily create new ones in a matter of hours.
With a dedicated Databricks connector, joint users can sync any data source that Airbyte supports into Databricks Delta Lake. The best part? The connector supports incremental and full refresh and allows use cases with CDC from your OLTP systems directly into Databricks, without the hassle of implementing it yourself. Check out a tutorial on loading data into Delta Lake to follow along.
As we continue to deepen the overall integration between Airbyte and Databricks Lakehouse Platform, we are excited about the upcoming addition of Airbyte Cloud to Databricks Partner Connect, a one-stop portal for customers to quickly discover a broad set of validated data, analytics, and AI tools and easily integrate them with their Databricks lakehouse across multiple cloud providers.
Airbyte Cloud helps unify your data integration pipelines under one fully managed platform powered by an active open-source community. Via Partner Connect, Databricks and Airbyte will bring a seamless experience for you to replicate data from any source into Databricks. Coming soon, any Databricks customer will be able to start a free trial of Airbyte Cloud from Partner Connect and automatically integrate the two products. That said, the two products already work great together, and we encourage you to connect Airbyte Cloud to Databricks today.
Speaking of working and learning together, I hope you stop by Airbyte CEO Michel Tricot's presentation on open source powers the modern data stack and learn more at their conference move(data).
Stay tuned for more exciting updates on how Databricks works with Airbyte, and watch their GitHub repository for new releases.
Try Databricks for free
