Building a sustainable energy grid for generations to come
E.ON uses Databricks to optimize energy grid asset management across Europe
Business applications
integrated into a
centralized data lake
Of data ingested daily
E.ON, a large energy grid provider in Europe, delivers green energy to households with sustainability at the forefront of their mission. Committed to this goal, E.ON continually seeks ways to digitize, remain compliant and meet increasing demands for efficiency, scalability and flexibility in energy systems. These three initiatives essentially boiled down to improving the company’s energy grid asset management — whether capturing data from various assets like solar panels and substations or providing 20+ distribution system operators (DSOs)/business units (BUs) with unified data formats for analysis, demand forecasting and energy consumption insights. Unfortunately, data silos, accessibility issues, a wide variety of data sources, scalability limitations and data governance concerns hindered E.ON from achieving their ambitious objectives. The company invested in the Databricks Data Intelligence Platform to consolidate disjointed data into the E.ON Data House (EDH) — laying the foundation to easily build business intelligence data products to support their enterprise data needs.
Overcoming fragmentation to achieve a unified data framework
As energy demands increase and sustainability goals become more pressing, energy providers must deal with mounting pressure to efficiently manage grid assets. As a leader in this charge, E.ON aimed to stop letting their valuable data from solar panels, substations and other important assets slip through the cracks of their sustainability and operational strategies. Without proper access to this data, DSOs — responsible for operating, maintaining and developing electricity distribution networks to deliver power from the transmission system to end users — may find it challenging in the future to ensure grid stability, properly support energy distribution or respond to fluctuations in demand.
E.ON’s disjointed data also made it challenging to deliver similar intelligence focused on energy distribution, customer service and innovation to DSO/business units. Without dependable data for analysis, planning or decision-making, the company couldn’t perform these essential functions like optimizing operations and achieving sustainability objectives. Naveen Kanneganti, Global Lead Data Architect at E.ON, explained, “Without a consolidated data framework, accessing and aggregating our data from various energy assets and different source systems was time-consuming and highly challenging. This fragmentation stunted our operational efficiency and made it difficult to provide consistent and reliable data to our distribution system operators and business units.”
Moreover, the high total cost of ownership (TCO) associated with developing, maintaining and scaling the data pipelines from hundreds of disparate systems further underscored the need for a unified solution. E.ON’s ability to handle the growing volume of their data — composed of billions of records — was crucial to nurturing forward-looking intelligence and addressing data governance concerns. Yet, implementing security and access controls across numerous platforms, with regulations like the General Data Protection Regulation (GDPR) in mind, posed legal risks to the organization and their brand reputation. All these obstacles only highlighted the necessity for a centralized data lake, the E.ON Data House. With it, the company could decouple applications from data sources to aggregate their data, streamlining access, reducing costs, enhancing scalability and strengthening data governance — and the Databricks Platform would allow them to execute this initiative successfully.
Implementing a scalable data strategy to support innovation
E.ON faced a pivotal challenge in transforming their data infrastructure, requiring a solution that could seamlessly scale with their growing needs and unify their fragmented data. The company turned to Databricks, leveraging Delta Lake and Unity Catalog to build a resilient foundation for analytical/business intelligence (BI) data products to support their enterprise data needs. Since Delta Lake — an open source storage layer that brings reliability to data lakes — provided unmatched scalability, E.ON processed and ingested over 1 terabyte of data daily without performance degradation. This drove efficiency in managing the massive volumes of structured and semi-structured data generated from their energy assets and a wide variety of data sources that have data including solar panels, substations and other important assets. By decoupling applications from data sources, Delta Lake ensured that all of this data from various points in the energy grid is extracted once from the wide variety of source systems, unified and distributed to many analytical applications for further processing.
At the same time, Unity Catalog was implemented to address E.ON’s complex data governance requirements. This solution provided a centralized governance layer on Delta Lake, enabling E.ON to seamlessly manage data security and access controls across their 20+ DSOs and BUs. With Unity Catalog’s fine-grained access controls, E.ON ensured compliance with GDPR and other energy-specific regulations, while simplifying the management of personally identifiable information (PII) and non-PII data, segregated by user roles. Plus, the platform supported hundreds of data users consuming the data from over 120 business applications and data products, offering transparency and building trust by leveraging Unity Catalog audit events to understand who has access to the specific dataset at any time. With the seamless data governance capability of Unity Catalog, E.ON enabled secure, simplified data consumption and improved collaboration, with plans for future self-service capabilities. This vision also laid the groundwork for creating a digital twin, or virtual representation of the company’s entire energy network so that they can easily analyze and optimize grid performance, through EDH.
By automating data pipelines via the metadata-driven approach, Databricks allowed E.ON to further process ready-to-use data into their various BUs. This reduced manual processes and freed up resources for more strategic tasks. This automation was crucial in streamlining data flow, enhancing productivity and ensuring that data was quickly accessible where needed. Naveen highlighted, “The automation of data pipelines and the data governance allowed us to break down silos between BUs and DSOs to make data more accessible, discoverable and reliable. It was key in improving decision-making and maintaining compliance across our operations.”
Reducing data access time through pipeline automation and more
The integration of Databricks into E.ON’s operations have led to substantial improvements in data processing, scalability and operations. With Delta Lake, E.ON now processes billions of records daily without any downtime or performance issues, dramatically improving the speed and reliability of their data handling. By combining streaming data with batch data under a single architecture within the EDH, the energy company can now provide near real-time data ingestion, allowing for faster time to insights and a more unified view of both data sources across BUs and beyond.
One of the most significant advantages of the investment has been the reusability of modular code, which has increased productivity across teams. By modularizing code into scalable pieces, E.ON has enabled the team to leverage existing pipelines for reuse, improving collaboration while also lowering TCO. This new approach to code reusability has streamlined the development of new applications and features, allowed teams to quickly add different use cases and reduced duplication of efforts to easily adapt to new challenges and opportunities.
In terms of compliance and data governance, Unity Catalog has played a pivotal role in E.ON’s Databricks journey. It enables data historization and pseudonymization, ensuring adherence to the GDPR and other regulations. This move has strengthened E.ON’s ability to secure and govern sensitive data while reducing legal risks. Even better, it paved the way for the company to discuss the potential of GenAI from automating document retrieval for substations to business intelligence and machine learning use cases. Although AI applications are not yet in production, the infrastructure is now in place with EDH to support exciting future developments and further enhance the organization’s operational capabilities and innovation potential.
