Alabama Power Company leverages Databricks for Outage and Storm Modeling

As we continue to navigate the complexities of the modern world, it's becoming increasingly clear that data-driven decision making is the key to unlocking success. APC Company (APC, an operating company of Southern Company) has been working tirelessly to develop a cutting-edge storm management system and outage modeling system that leverages the power of data to drive more informed decision making.

In this blog, we'll take a deeper dive into two cutting-edge applications, SPEAR and RAMP, that APC designed to improve storm management and reliability analytics. We’ll explore the development, architecture, and benefits.

“Databricks is highly beneficial to our APC data analytics team working in the cloud on large data sets because it provides a unified platform that enables seamless collaboration, scalable data processing, and real-time analytics. This ensures efficient handling of big data workflows while integrating with our cloud services for enhanced performance and flexibility. It has enabled us to create cutting-edge technology for Grid Reliability, as well as helping us understand and prepare to respond to large-scale events like Hurricane Francine and Hurricane Helene.”

— Shane Powell, Data Analytics and Innovation Manager, APC

Before modernization, the storm management process was primarily manual, which allowed for improvement in efficiency and accuracy. For forecasting, APC relied on spreadsheets and various data sources,but there was potential to enhance clarity and situational awareness in the field. The development of SPEAR and RAMP is a testament to the power of innovation and collaboration, and has enhanced our ability to respond swiftly and effectively to cases of trouble.

APC has embraced emerging technologies, such as cloud computing, data lakes, and advanced analytics; , By building RAMP and SPEAR on Databricks, they knew they had a trusted partner that could empower them to deliver an end-to-end solution - from BI to AI - that automates storm management processes, offers insights in near real-time, and does so in an accurate and secure way.

Let’s review the two applications:

• RAMP, which stands for Reliability Analytics Metrics and Performance, is a cloud-based reliability application that provides a comprehensive view of the power grid's performance, including reported values, customer experience values, and device failures. The application helps identify areas of improvement and provides insights into the root causes of reliability issues.
  
• SPEAR, which stands for Storm Planning, ETR and Reporting, is a forecasting application on cloud that uses data from weather vendors and internal systems to predict the impact of severe weather events on the power grid. The application provides a detailed forecast of the number of incidents, resources needed, and estimated time of restoration, allowing the company's storm center to make more informed decisions and allocate resources more effectively.
  

The APC data team worked closely with E Source, a utilities focused consulting, research, and data science company, to design, develop, and deploy RAMP and SPEAR on Databricks. Databricks has been playing a crucial role in helping APC harness the full potential of their AMI data and other data sources to drive grid improvements and operational efficiencies.

The Databricks Data Intelligence Platform provides a unified environment where APC can integrate, process, and analyze vast amounts of AMI data alongside other critical datasets like GIS, outage management, and weather information. This integration allows for more comprehensive insights and predictive analytics. Databricks' scalable architecture enables APC to efficiently handle the high-volume, high-velocity data streams from millions of smart meters, while its advanced analytics and machine learning capabilities facilitate the development of sophisticated models for load forecasting, outage prediction, and grid optimization. The platform's collaborative workspace and support for multiple programming languages empower both data scientists and engineers to work seamlessly on complex data projects.

Additionally, Databricks' data governance features ensure that sensitive customer data is handled securely and in compliance with regulations. By leveraging Databricks, APC can more effectively clean, curate, and aggregate AMI data, build user-friendly interfaces for data exploration, and even incorporate cutting-edge technologies like large language models to enhance data interpretation and accessibility. This comprehensive approach is helping APC transform raw AMI data into actionable insights that drive grid modernization, improve reliability, and enhance customer service.

APC explored all three hyperscaler’s native services and a proprietary AI platform, but landed on Azure Databricks because of Databricks' ability to handle large volumes of data and provide a unified platform for data engineering, data science, data analytics, and AI.

“Databricks Genie is accelerating AI development at APC by enabling quick access to large datasets through natural language queries. This allows our team to quickly gather the data needed to train, test, and refine AI algorithms. Furthermore, Genie’s ability to learn from our interactions and continuously improve its querying accuracy makes it an efficient tool for feeding high-quality data into AI development processes. We are excited about integrating it into our existing tools to create an even higher level of cutting-edge data technology for our company.”

— Shane Powell, Data Analytics and Innovation Manager, APC

Business impact:

APC has improved its grid management and storm response with two innovative applications built on Databricks: RAMP and SPEAR. These solutions have transformed the company's approach to data-driven decision-making and operational efficiency, enabling monitoring of 1.5 million customers, 2,400 substations, and 250,000 devices

RAMP (Reliability Analytics Metrics and Performance)

RAMP enables real-time monitoring of assets, allowing proactive maintenance and replacement of underperforming equipment. This shift from monthly to near real-time reporting has led to significant improvements:

• With 70,000 annual outages, a targeted 5% reduction (3,500 outages) could save $17.5M in crew costs alone.
• Customer outage history retrieval has improved from four hours to just four seconds, a 99.97% (3600X) efficiency gain.

SPEAR (Storm Planning, ETR and Reporting)

SPEAR proactively predicts storm impacts on the grid, including outages and Estimated Time of Restoration (ETR). It optimizes resource allocation to avoid over or under-provisioning, resulting in substantial benefits:

• The system can predict storm impact within a 10% margin of error.
• For a 10-day storm with 500 customer outages, the cost at a 20% margin of error would be $34.2M. With Databricks enabling a 10% margin of error, the cost reduces to $31.3M, potentially saving $2.8M per storm event (an 8% reduction).

These Databricks-powered solutions significantly enhance APC's operational efficiency and storm readiness, leading to substantial cost savings and improved customer service. By utilizing Databricks, APC is enhancing its ability to respond to and mitigate the effects of severe weather events, which are among the most unpredictable challenges facing utility companies.

This data-driven approach allows the utility to make more informed decisions, optimize resource allocation, and ultimately improve service reliability for its customers in the face of increasingly frequent and severe weather events. The implementation of these solutions demonstrates APC's commitment to leveraging cutting-edge technology to enhance its services and operational capabilities.

Architecture

At the foundation of APC’s architecture lies a robust data ingestion layer. It's designed to handle a diverse array of data sources:

• Outage Management System (ADMS): Real-time grid status and outage information
• Weather Data Vendors: Multiple sources for more accurate weather predictions
• Advanced Metering Infrastructure (AMI): Smart meter data from customer premises
• Grid Telemetry: Sensor data from various devices across the distribution network

These data streams are continuously ingested and initially land in a cloud storage solution Azure Blob Storage.

Databricks: The Central Nervous System

Databricks serves as the core processing and analytics engine in the above architecture. Here's how it's structured:

1. Data Processing and Transformation
   APC uses Delta Lake as the storage layer for their data lakehouse. This provides them with :
   • ACID transactions for data reliability
   • Schema evolution to adapt to changing data structures
   • Time travel capabilities for auditing and rollbacks
   Raw data from various sources (e.g. smart meters, customer systems, grid sensors) is ingested into Delta tables using a combination of Azure Data Factory, Delta Live Tables (DLT) and the underlying power of Spark for distributed computing. DLT pipeline helps with automatically handling incremental processing, data quality checks, and dependency management.
2. Data Science and Machine Learning
   APC has implemented a comprehensive data science and machine learning environment using Databricks, integrating key components to streamline their workflow for grid optimization, customer analytics, and energy forecasting. APC uses Databricks Notebooks as their primary interface for data analysis and model development, MLflow to manage their machine learning lifecycle, from experimentation to deployment and AutoML to quickly generate baseline models and accelerate their machine learning projects.
   
   This approach allows their teams to collaborate more effectively, manage the entire ML lifecycle, and rapidly prototype and deploy models for various aspects of their operations, from grid management to customer service optimization. Additionally, Databricks' Lakehouse Monitoring enhances this process by enabling data-driven decision-making through continuous tracking of data quality and model performance. The monitoring system automatically detects statistical changes in input features and model outputs, alerting teams to potential data drift or performance degradation. This proactive approach empowers organizations to make informed decisions on when to retrain models, ensuring they remain accurate and relevant in dynamic environments.
3. Data Governance and Security
   APC has implemented Databricks Unity Catalog to centralize metadata management across multiple workspaces, enhancing data governance and collaboration. This unified approach allows for consistent access controls and security policies across all data assets, ensuring that sensitive information is protected while enabling efficient data discovery and collaboration among data scientists, analysts, and engineers. The implementation also facilitates comprehensive data lineage tracking and maintains detailed audit logs, supporting regulatory compliance efforts. By leveraging Unity Catalog, APC has significantly improved its ability to manage, secure, and utilize its data assets effectively, fostering a more collaborative and compliant data ecosystem across the organization.
4. Advanced Analytics
   APC has implemented a sophisticated data analytics infrastructure to optimize grid operations and planning. They also use GraphFrames to analyze grid topology, GeoSpark for geospatial processing of assets, and custom time series models for demand and outage prediction. While Databricks handles core data processing, specialized tools like NetworkX and Mapbox are integrated for specific functions. The outputs are visualized in RAMP and SPEAR, containerized applications built by E Source, ensuring high availability and scalability.

With this architecture APC is now able to process large amounts of data quickly, efficiently, and securely, as well as share their applications across the organization.

In summary, APC has partnered with Databricks and E Source to develop innovative data analytics solutions for storm management. This collaboration has enabled APC to:

1. Gain better insights into storm data using the SPEAR application
2. Predict storm impact more accurately using predictive models created in databricks and by applying historical data to current weather patterns to determine how and when AL Power customers will be negatively impacted.
3. Improve preparation strategies for their 1.5 million customers and proactive deploying the resources in the field and informing their customers in advance via notifications.

By leveraging advanced data science techniques, APC is enhancing its ability to respond to and mitigate the effects of severe weather events, which are among the most unpredictable challenges facing utility companies. This data-driven approach allows the utility to make more informed decisions, optimize resource allocation, and ultimately improve service reliability for its customers in the face of increasingly frequent and severe weather events.