Next Generation Business Intelligence
Faster data availability (30 seconds instead of 15 minutes)
Instead of months to deploy ML use cases
Vattenfall is one of the largest utilities in Germany and a leading power producer in Europe. Its German hydropower business plays an integral part in reaching the goal of fossil-free living within one generation. With infrastructure built on Databricks and expertise from initions the business unit was able to outgrow the multiple separate on-prem solutions that existed before.
The path to digitalization
The European energy market is constantly changing. This leads to more complexity in the markets and also to entirely new markets. For Vattenfall to contribute in the best possible way to a fossil-free future, tackling the existing data structure was key. And creating actionable insights from a unified data platform was necessary to ensure reliable data accessibility and quality, near real-time data availability and also to enable analytics and AI workloads.
Next generation BI with Databricks and the lakehouse
In order to meet these multilayered challenges, a platform technology had to be found that was capable of guaranteeing high-performance parallel processing and at the same time offered many options for connecting a wide variety of source systems out of the box. Vattenfall decided to implement a data platform based on the Databricks Data Intelligence Platform. The lakehouse serves two worlds in this context: the functional scope and data management of a typical data warehouse and the flexible object storage of a data lake. The Databricks Data Intelligence Platform provides a lean and simplified data architecture.
A classic lakehouse approach was used to build the data platform. Establishing a three-level data repository consisting of Bronze, Silver and Gold tables — which progressively improve the structure and quality of data — ensures both consistency and stability in the data warehouse. The quality-checked data and data products are provided to the connected business units via Databricks SQL and Microsoft Power BI. This gives analysts a high degree of flexibility and makes use of already familiar front-end tools.
High speed and high quality at the same time — no conflict
For a data and AI architecture to meet the demands of business, it must be able to perform the balancing act of attaining high data quality and high speed. This is ensured by building the lakehouse with Databricks. By implementing a Git workflow using Databricks repos and Azure DevOps, Vattenfall was able to meet this need in the HySBAP (Hydro Strategic Business Analysis Platform) project for development as well.
Through a standardized process model and deployment framework, the various tasks in the continuous integration and continuous delivery (CI/CD) as well as DevOps contexts are optimized. The initions toolbox here comprises automation concepts that include the provision of notebooks, jobs and queries.
The whole is the sum of the parts
With the architecture built in the HySBAP project, Vattenfall Wasserkraft GmbH is now able to meet its goals and make an important contribution to a fossil-free future. The new hydro data and AI architecture integrates all the information that is important for commercial and technical plant operation, leading to better decision-making. Data integration and scalable ETL pipelines are equally important as advanced analytics, drone integration or digital twins. Advanced decision-making is possible for the first time using a holistic view of data. Now, data is available up to 30x faster and is more reliable than before. Data availability and accessibility are key to bringing data-driven decision-making closer to everyday operations and into the hands of more than just the company’s data science teams.
