Accelerate Business Value from Data Sharing with Databricks Unity Catalog and Tredence UnityGO!

Enterprise leaders are turning to the Databricks Data Intelligence Platform to create a centralized source of high-quality data that business teams can leverage to develop insights, make better decisions, and accelerate innovation. In a recent survey, Chief Data Officers (CDOs) said they wanted to establish clear and effective data governance (51%); improve data quality (48%); build and maintain advanced analytics capabilities (42%) and business intelligence capabilities (36%); develop data monetization capabilities (21%); and improve data, analytics, and artificial intelligence (AI) ethics (21%). Clearly, data transformation is top of mind.

Historically, data, IT, and security teams have struggled with the following challenges related to unifying and democratizing data:

• Securing and governing data: Enterprise teams are rushing to deploy large language and domain-specific models, which require extensive clean data thus creating new issues with data governance and security.
• Mitigating data risks: As data volumes and business requirements grow, teams often duplicate data sets and recycle them across use cases and platforms. In addition, users can corrupt data pipelines feeding multiple use cases. These substandard processes increase data complexity, costs, and business inefficiencies.
• Scaling and speeding up data access processes: Using different data platforms and sources for experimentation isn't sustainable. However, many business teams want to maintain specialty platforms for other purposes. In addition, business teams have historically waited for IT support to set up data-sharing tools, provision data, or self-provision select data sets for exploration. These lengthy and ill-defined processes meant teams make decisions with lagging and limited data, potentially missing out on time-sensitive business opportunities.

Delivering Governed and Shared Data Capabilities

The Databricks Data Intelligence Platform includes Unity Catalog, the industry's first unified governance solution for data and AI on the lakehouse architecture. Natively built into Databricks, Unity Catalog provides:

• Unified visibility into data and AI: Discover and classify structured and unstructured data, ML models, notebooks, dashboards, and arbitrary files on any cloud. You can also consolidate, map, and query data from various platforms, including MySQL, PostgreSQL, Amazon Redshift, Snowflake, Azure SQL, Azure Synapse, and Google's BigQuery in one place. With this, boost your productivity by securely searching, understanding, and extracting insights from your data and AI using natural language.
• Single permission model for data and AI: Simplify access management with a unified interface to define access policies on data and AI assets and consistently apply and audit these policies on any cloud or data platform. Now, you are able to securely access data from other computing platforms using open interfaces, with consistent permissions managed in one place. You can also enhance security with fine-grained control on rows and columns, while efficiently managing access through low-code attribute-based access policies that scale seamlessly.
• AI-powered monitoring and observability: Harness the power of AI to automate monitoring, diagnose errors, and uphold data and ML model quality. Benefit from proactive alerts that automatically detect personally identifiable information data, track model drift, and effectively resolve issues within your data and AI pipelines to maintain accuracy and integrity. Unity Catalog also allows you to streamline debugging, root cause analysis, and impact assessment with automated column-level data lineage. Plus gain comprehensive lakehouse observability into your data and AI with operational intelligence utilizing built-in system tables for billing, auditing, lineage, and more.
• Open data sharing: Easily share data and AI assets across clouds, regions and platforms with open source Delta Sharing, natively integrated within Unity Catalog. Securely collaborate with anyone, anywhere to unlock new revenue streams and drive business value, without relying on proprietary formats, complex ETL processes or costly data replication.

Speeding Time to Advantage with UnityGO!

Tredence is excited to announce the launch of its new Unity Catalog Brickbuilder Accelerator, UnityGO!, which simplifies and speeds up enterprise data implementations to Unity Catalog on the Databricks Data Intelligence Platform by 60%. Thus supporting C-suite leaders' objectives to use data and AI to improve workforce productivity, unlock faster innovation, streamline operations, and enhance customer satisfaction.

UnityGO! automates the conversion of metadata, such as tables and views; code, in the form of notebooks; and ACLs to Unity Catalog, simplifying and speeding migration significantly compared to manual processes. UnityGO!:

1. Reduces the risk of unoptimized model/notebook deployment by providing a test workspace environment before data is transformed and migrated to a Unity Catalog-enabled Data Intelligence Platform.
2. Optimizes migration resources and project planning by assessing workspace migration complexity early in the project, allowing for more granular planning and resourcing.
3. Improves Databricks Unity Catalog by augmenting automation to convert mount points and notebooks, moving objects to catalogs, and streamlining workflow.
4. Lessens technical resource requirements by automating the conversion of analytical and AI/machine learning applications to run on a Unity Catalog-enabled Data Intelligence Platform.

Accelerating Time to Business Value

With Unity Catalog and UnityGO!, teams speed up time to value and scale. The open-source architecture of the Databricks Data Intelligence Platform and the straightforward processes of UnityGO! enable teams to integrate structured and unstructured data to super-charge data-driven decision-making.

UnityGO! leverages code designed for the Unity Catalog to automate the creation of critical workflows. It tests retrofit scripts in a sandbox before deploying them, eliminating the risk of migration failures. In addition, teams can configure UnityGO! for company-specific patterns and applications to meet their unique requirements.

With new capabilities enabled by Unity Catalog and UnityGO!, enterprise teams can securely share data to improve first-party data insights, segmentation, and personalization strategies. They can also sell anonymized data to partners, capturing their share of the data monetization market, slated to grow to $7.3 billion by the end of 2027.

UnityGO! uses a four-step process to migrate data from any platform or source.

1. Setting up the catalog: Teams create a catalog and scheme in the Unity Catalog metastore and create objects that point to where data is stored. Next, they create a catalog for development, quality assurance, and production teams in the catalog name.
2. Enabling UnityGO! Tredence's migration accelerator requires a solution workspace that stores the migration kit that collects metadata from application workspaces to enable migration. The infrastructure must be provisioned for a Web application (a cloud-based service to run web apps and Postgres database) that powers a user interface (UI) that simplifies the migration to Unity Catalog.
3. Collecting and configuring metadata: Teams select the workspaces that need to be migrated to Unity Catalog, automating the collection of metadata to assess the complexity of the workspaces and enable the configuration of mountpoints, schemas, managed tables, notebook folders, and user groups in UnityGO! UI Interface.
4. Retrofitting and migrating the code base: Next, UnityGO! solutions retrofit scripts to a sandbox Unity Catalog-enabled workspace, automates the creation of DDL scripts, notebooks, ACL scripts, and Databricks workflows. Teams can run scripts in a sandbox environment to migrate their objects to Unity Catalog and test them without impacting existing applications. Then, they enable an existing application workspace for Unity Catalog and migrate the code from the sandbox environment to the existing development workspace. Teams can then test and migrate the code to quality and production workspaces through existing continuous integration and delivery (CI/CD) processes.

Teams that use UnityGO!:

• Automate processes: With highly automated processes, teams save time, reduce business disruption, and accelerate impacts.
• Leverage code designed for Unity Catalog: UnityGO! identifies patterns and converts code, automating the creation of workflows.
• Reduce risks: Unity Catalog retrofit scripts are tested in a sandbox before being automated. As a result, migration failures are mitigated.
• Can customize patterns: Teams can configure UnityGO! for company-specific patterns and applications, retrofitting existing patterns to run on Unity Catalog.

About Databricks Brickbuilder Accelerators

Databricks Brickbuilders Accelerators pair the expertise of consulting partners with the Databricks Data Intelligence Platform to quickly implement a specific methodology or Databricks capability. With Brickbuilder Accelerators, teams gain:

• Access to a trusted partner: Databricks collaborates with Tredence because of their extensive industry domain-specific knowledge and experience in building and executing enterprise generative AI and Data Intelligence Platform strategies. Brickbuilder Accelerators help teams solve critical analytics challenges, reduce costs, and enhance productivity with minimal friction. Tredence already has a large portfolio of Brickbuilder Solutions for the retail, manufacturing, telecommunications, and healthcare segments.
• The ability to use credible frameworks: Tredence has over 350 data engineers and scientists trained and certified on the Databricks Data Intelligence Platform and Unity Catalog. This team of experts delivers UnityGO! and provides the experience required to address enterprise teams' biggest data, analytics, and AI needs.
• The tools to speed time to value: Tredence's services and automated frameworks, combined with the Databricks Data Intelligence Platform and Unity Catalog, help teams quickly gain a single source of trustworthy data for secure sharing and collaboration and get to innovation faster.

Want to learn more?

To learn more about UnityGo!, sign up for the upcoming webinar, Accelerate Data Sharing with Unified Data and AI Governance on December 12.

Featuring Maulik Dixit, Director and Senior Azure, Databricks and Cloud Architect at Tredence, and Zeashan Pappa, Global Lead for Data Governance at Databricks, this webinar will outline how to overcome data sharing hurdles and accelerate business value while also maintaining secure governance and lineage.