Using data to power millions of Australian households
Cost savings on the data platform
Faster data processing
Reduction in data platform operational overhead
With over 11 billion electricity, gas and network connection assets, having a consolidated view of customer energy and equipment usage are essential to help AusNet Services make data-driven decisions and better serve their 1.5 million customers. The organization embarked upon its data transformation journey to address the expected growth of over 65TB of data over the next four years, generated from diverse sources and assets housed in separate data warehouses. This resulted in challenges to derive any meaningful data insights, in addition to managing maintenance schedules to assess the condition of assets ahead of incurring damages. Since implementing the Databricks Data Intelligence Platform on Azure, AusNet Services now has a complete view of the performance of its assets, and can deliver data-backed recommendations to predict risks and save maintenance costs.
Legacy system made data processing challenging
AusNet Services is a diversified Australian energy and infrastructure business, operating electricity and gas distribution networks on top of providing commercial energy services to households, communities, businesses and governments in Victoria, Australia.
“As we embarked on our data transformation journey, it was inevitable that we needed a robust modern data platform that could serve our needs. Data engineers needed to be able to easily process the large volumes of full metering data. Data scientists needed to be able to use that data to run experimentation models, and the business analysts needed to run queries and their own ad hoc analysis,” said Souvik Das, Senior Data Platform Lead at AusNet Services.
AusNet Services was facing stability and scalability challenges with their existing compute platform. They turned to the Databricks Data Intelligence Platform on Azure to centralize all their data from various sources onto a unified, scalable infrastructure. This migration enhanced the performance of their data team and enabled downstream analytics and machine learning use cases that provide a better customer experience by predicting maintenance schedules and reducing the impact of asset failure.
Improved data reliability and cross-team collaboration
Within a couple of months, AusNet Services migrated all their production workloads from the legacy system to Databricks, allowing the data team to manage the full data lifecycle more reliably. From data preparation and processing to running experiments and ad hoc analysis, data team performance was largely enhanced.
Souvik shared, “By leveraging Delta Lake’s capabilities and fast performance, we were able to efficiently process large volumes of data at greater speed for our data model based on Data Vault 2.0. Delta Lake’s Time Travel feature allows us to access any historical version of the data, which has been useful.”
In addition, Databricks’ optimized cluster management capabilities have helped to improve the overall maintenance of various projects. Data engineers are able to scale infrastructure up and down as needed, giving them more compute power to process data or queries. Business analysts are empowered to use Databricks interactive notebooks to perform their own ad hoc analysis. The notebooks also enable data scientists to easily conduct and deploy experiments, further fostering collaboration between the different business units.
Predicting asset failure before the customer is impacted
AusNet also wanted to more effectively and safely manage their assets relying on the assessment of the condition, criticality and risk of every asset, ahead of any damages. The team shifted to using data insights and predictive maintenance for three of their assets classes that resulted in significant risk reduction and cost optimization. As a result of this early success, AusNet has expanded this initiative and is currently implementing predictive maintenance across its entire fleet of assets.
“Databricks has been an integral part of this journey, making the migration and onboarding experience seamless. The team is very supportive, and their assistance is almost immediate,” said Souvik.
Moving forward, the data team at AusNet Services is planning to onboard more features to enable the future energy needs of customers across Victoria, Australia. David Betts, Lead Data Engineer, AusNet Services said, “Some of the features include Databricks streaming to improve data processing and enable real-time use cases, Databricks SQL for easy data analysis, as well as the integration of data visualization tools, such as Power BI, for data-driven insights leveraging Databricks.”
The energy sector is undergoing rapid transformation and AusNet Services’ investment in Databricks today will unlock the potential of data and insights to achieve a sustainable energy future.
