Deploy Fully Managed Change Data Capture Pipelines With Arcion and Databricks Partner Connect

We are thrilled to announce that Arcion, the cloud-native, distributed change data capture replication platform for simpler real-time data pipelines, is now available in Databricks Partner Connect. Arcion enables real-time data ingestion from transactional databases like Oracle and MySQL into the Databricks Lakehouse Platform with their fully-managed cloud service.

Arcion and Databricks have been working towards simplifying data replication and real-time data ingestion for over two years now. This integration is the latest in our continued effort to make real-time data sync with the lakehouse even easier for our joint customers and will result in faster and highly-automated analytics and AI and ML workflows.

Real-time data ingestion to Databricks starts with just a click

Transactional databases like Oracle have become a critical part of modern data infrastructure. They are extremely secure and often store mission-critical business data. Unfortunately, the design of transactional databases limits collaboration teams, especially analytics, resulting in stale data and limited business visibility. Arcion solves this issue – while combating the slow, expensive batch processes and brittle pipelines of traditional solutions – with its fully-managed, distributed change data capture (CDC) technology that ensures lower cost of ownership, reduced DevOps, and peace of mind with end-to-end data consistency. Arcion's pipelines can be stopped and resumed at will without causing data loss and has minimal impact on the production source.

Arcion brings high-volume, concurrent data ingestion into Databricks through data pipelines that can achieve 10k ops/sec/table and support tables with billions of rows. But connecting the platforms still required users to configure, transfer credentials, and validate the connection manually. Or it did until today.

With Partner Connect, users can simply choose Arcion as the data ingestion partner of choice, and Databricks will automatically configure resources, provision an SQL endpoint, and transfer credentials. Once a secure connection has been established, users will be taken to Arcion directly where they can log in (or start a free trial).

Easy access to fully-managed data pipelines

Deploying pipelines and starting real-time data ingestion in Arcion only takes a few steps:

• Select the Replication Mode
• Choose a Source (launching with Oracle, Oracle Exadata, Oracle RAC, MySQL and Snowflake, and more sources coming in the coming months). For the destination, Databricks is automatically pre-selected and pre-configured as the target.
• Filter the data (schemas, tables, and columns)
• Start replication

And that's it. Once the replication completes, you can go into Databricks and view the ingested Delta tables in the Databricks Data Explorer, query them, or go straight to analytics in the Lakehouse.

Databricks and Arcion support some of the most demanding data requirements across a myriad of industries, AI-based or otherwise. From real-time fraud detection in finance to more accurate demand forecasting in retail, and hundreds of other use cases in between - Arcion + Databricks can boost your data strategy and results.

Arcion + Databricks for data-driven enterprises

Our partnership with Arcion transcends just connectors and integrations, Databricks and Arcion share a common philosophy of greater data accessibility and improved data analytics. For instance, Arcion handles schema changes out of the box, requiring no user intervention. This helps mitigate data loss and eliminate downtime caused by pipeline-breaking schema changes by intercepting changes in the source database and propagating them while ensuring compatibility with the target's schema evolution. Pairing this technology with Partner Connect's automatic configuration helps enterprises unify data silos much faster and more reliably.

Try out Arcion for yourself (for free)

Not an existing Arcion user? No worries, Arcion offers a 14-day free trial so you can try out Partner Connect and start ingesting data into Databricks in real-time right away. For a more detailed walkthrough of real-time data ingestion into Databricks Partner Connect using Arcion, read this Arcion blog with step by step breakdown.