A Journey Towards Unified Data Governance at bp

"The benefits and impact that Unity Catalog has already delivered and those that it unlocks by enabling the next generation of features in the Databricks Platform cannot be understated.”

—  Liam Donohoe, Principal Architect, bp

In today's data-driven world, effective data governance is crucial for organizations to harness the full potential of their data assets. bp, a global leader in the energy sector, has embarked on a transformative journey by implementing Databricks Unity Catalog to streamline and enhance their data governance framework. This program of work was successfully completed in close partnership between bp and Databricks. This blog post delves into bp’s experience, highlighting the challenges, strategies, and benefits of this significant migration.

The Need for Unified Data Governance

Bp faced several challenges typical of large enterprises dealing with vast amounts of data spread across various systems and platforms. These challenges included:

• Lack of centralized oversight: Managing data across disparate systems made it difficult to enforce consistent data governance policies.
• Data set insights:  Discoverability and visibility of dataset ownership took time leading to concerns on accuracy of data provenance,
• Compliance and security: Ensuring compliance with stringent data protection laws such as GDPR and industry-specific regulations was a complex task.

To address these challenges, bp turned to Databricks Unity Catalog, a unified governance solution that promised to provide a comprehensive framework for managing their data assets.

Why Unity Catalog?

Unity Catalog, Databricks' unified governance solution, provided bp with a comprehensive framework to address these challenges. Key features of Unity Catalog that benefited bp include:

• Centralized data governance: Unity Catalog offers a common namespace that permits access to any data within the organization, ensuring consistent governance policies across all data assets.
• Granular access control: With Unity Catalog, BP can implement fine-grained access controls at the row and column levels, ensuring that users only have access to data they are entitled to see and query. This is crucial for maintaining data security and compliance.
• Enhanced data lineage and auditability: Unity Catalog provides automated data lineage at the column level across data and ML pipelines and audit logs, enabling bp to trace the entire journey of their data. This transparency is essential for compliance and simplifies the audit process.

Migration Scope and Scale

The scale of bp’s Unity Catalog migration was impressive. The company successfully migrated over 270 Databricks workspaces from two separate data hub platforms and 15 standalone instances. These workspaces served over 10,000 people in key business entities, including Finance ERP, Customer and Products, Production and Operations, and Enterprise.

The migration was executed in a phased approach; each quarter, bp was able to migrate an order of magnitude more workspaces than the quarter before

• Quarter 1: Pilot migrations
• Quarter 2: Early adopters 
• Quarter 3: First majority 
• Quarter 4: Remaining workspaces

This structured approach allowed bp to learn and adapt throughout the process, resulting in an accelerated pace of migration in the later stages.

Implementation Strategy and Lessons Learned

bp's implementation strategy for Unity Catalog was methodical and strategic, tight engagement with Databricks, partners, and internal teams was key to addressing challenges quickly and effectively.

1. Prioritizing Business-critical Tables: bp conducted a comprehensive review of all data assets to classify them according to their importance to business operations, sensitivity, and compliance requirements. Constant progress measurement helped keep the project on track.
2. Flexible Data Integration: To expedite the integration of all data into Unity Catalog, bp adopted a flexible approach, meeting the data where it was and aligning it with the bronze, silver, and gold layers over time. Weekly calls were crucial in maintaining project momentum.
3. Automation-first Approach: bp leveraged automation to manage the long tail of data consumers and ensure that all data was governed effectively without overwhelming their data teams. This approach was crucial in adapting to the unique challenges of each business group, accommodating different coding patterns, data access patterns, and feature usage.

Benefits Realized

"Our Unity Catalog Migration project, involving over 200+ Databricks workspaces, has achieved outstanding results in cloud cost savings, governance, and operational efficiency. By consolidating workspaces under Unity Catalog, we have significantly reduced operational costs, centralized governance, enhanced security, and streamlined data sharing and compliance. The close collaboration with Databricks engineers ensured this migration exceeded our expectations in terms of speed and efficiency. This milestone not only drives cost efficiencies but also advances secure and efficient data management at bp, enabling greater value for our business stakeholders.”

—  Srinivas Chandolu, Staff Platform Engineer, bp

The implementation of Unity Catalog at bp has significantly improved data security and compliance, enhanced data visibility, and increased operational efficiency across the organization.

Technical Benefits

The implementation of Unity Catalog at bp yielded several significant technical benefits:

• Access to the latest Databricks Features: Unity Catalog unlocked access to cutting-edge features like Databricks Genie, Delta Sharing, and Databricks Assistant, providing substantial benefits to bp's data engineers.

• Addressing Technical Debt: The migration allowed bp to upgrade from runtime version 9.1, which had reached the end of its serviceable life, resolving issues with their custom entitlement framework.

• Improved Data Visibility: Unity Catalog significantly enhanced data visibility across the organization. As a result, bp can now avoid duplication and silos.

• Improved Data Security and Compliance: Unity Catalog’s centralized governance and robust security features have streamlined compliance processes and ensured that only authorized personnel have access to sensitive data.

Financial Benefits

The Unity Catalog migration is expected to deliver substantial financial benefits to bp:

• Cost Savings: The Databricks consolidation effort, included as part of the Unity Catalog migration, has the potential to save nearly $1 million annually.

• Operational Efficiency: By enabling access to cost and runtime job data directly from system tables, bp eliminated the need for explicit management of Overwatch jobs, saving approximately $4,000 weekly.

• Optimized Runtime Costs: Access to the latest optimized Unity Catalog-compatible Databricks runtime has the potential to significantly reduce runtime costs.

Operational Improvements

The migration to Unity Catalog brought about several operational improvements:

• Dedicated Service Line: bp can now operate Unity Catalog as a dedicated Service Line, enhancing their ability to manage operations and support.

• Enhanced Data Access Framework: bp deployed a Platform Experience Portal with an improved Data Access framework, which improved security, compliance, and efficiency in data management.

• Streamlined Access Management: The new framework facilitates updates and reduces administrative overhead for over 10,000 users, promoting collaboration among teams.

Unlocking New Capabilities

Unity Catalog has positioned bp to leverage several advanced features:

• Delta Sharing: a full Delta-Parquet conversion was part of the migration, unlocking Delta sharing for non-Databricks consumers.

• SQL Serverless Warehouses: These have been extensively implemented, enhancing bp's data processing capabilities.

• AI Features: bp is now well-placed to use advanced AI features like model serving, fine-tuning, and vector databases.

• GenAI Augmentation: bp can now use auto-generated comments, AI assistants and Genie to increase engineering productivity and business user accessibility.

• Enhanced Telemetry: Access to system tables and granular telemetry provides bp with deeper insights into their data operations.

The Road Ahead

bp's journey with Unity Catalog exemplifies how a unified data governance solution can transform data management practices in large enterprises. bp plans to continue building on this by further consolidating workspaces and decommissioning the legacy hive metastore. By addressing key challenges and leveraging the robust features of Unity Catalog, bp has set a new standard for data governance in the energy sector. This migration not only solved immediate data governance challenges but also positioned bp to leverage advanced data and AI capabilities in the future, cementing its position as a data-driven leader in the energy industry.