Kraken sparks energy industry innovation with Databricks

Legacy tools can’t keep pace with growth

Kraken’s platform integrates all parts of utility systems, offering energy customers a unified view of their business that enables a springboard for data-driven transformation. Previously, Kraken relied on a collection of various tools for data analysis. But as the company began to grow — expanding their retail business in the UK and creating more Octopus Energy businesses worldwide — their disparate technology couldn’t handle the rising volume of data. The number of people working on the data team grew as well, but relying on pure SQL extracts and ETL approaches meant that work had the potential to be duplicated by different members of the team.

One of Kraken’s major use cases is helping clients transition from their previous IT systems onto Kraken’s system. Energy businesses rely on data to make daily decisions, so access to their data is critical. In the past, when Kraken shared data with clients, the client needed a sophisticated data team to replicate the data and build a data lake, and the process could take a lot of time.

In a highly regulated industry, it was also important for Kraken and their clients to know their data would be well controlled. “The data needs to remain with us at all times. That’s one of our main philosophies,” said Javi Asensio, Head of Data Platform Engineering at Kraken.

Kraken needed a data solution that could scale along with the business, support efficient operations, align with regulatory best practices and offer value to their growing roster of clients.

Hitting the mark for scalability, streamlining

After evaluating other options, Kraken chose Databricks for its scalability and superior performance. Data control was also key. With Delta Sharing, Kraken can keep data within the company’s on-premises and cloud storage and process it with their own machines — an important consideration for interactions with regulators.

Kraken also appreciates the flexibility and transparency offered by Databricks’ open source origins. “Knowing that there is a community behind it, and we can look at the code and understand how things work, is a huge benefit for us,” said Asensio.

Databricks also offered a superior experience for developers. Tasks that were easy to do in Databricks were “incredibly cumbersome” in other tools, Asensio said. The team also appreciates the ability to manage data processes through APIs and Terraforms rather than a user interface.

Meanwhile, Delta Sharing streamlines data sharing with Kraken’s clients. Kraken can aggregate all the data sources that clients need, so clients no longer need advanced teams to ingest data. Databricks centralizes data in one place, empowering Kraken clients to easily access it. With the Databricks unified platform, Kraken clients can see a full, fine-grained picture of their data and move quickly to leverage it for innovation. Delta Sharing also enables Kraken to be proactive when it comes to solving data issues. It manages dozens of different data workspaces for clients that are completely isolated from each other while sharing aggregated, anonymized data to a central repository where it can be analyzed. This process is automated, making it fast and easy for analysts. “Delta Sharing helps us easily distribute centralized datasets across the organization and manage around 40 Databricks accounts. That is something that we really appreciate,” Asensio said.

Supporting growth while reducing cost and effort

Databricks has provided the scalability the Octopus Energy Group needed. When Kraken decided to adopt Databricks in 2021, they had 15 people working on the data team. Today, they have more than 700 employees interacting with data. “We never had any kind of real struggle in terms of scalability,” said Asensio.

Using Databricks has also supported Kraken’s growth by making it easy to onboard new operations. In the Octopus Energy Group’s Valencia office, for example, Asensio visited during the first several months of operation to offer training in Databricks. He hasn’t needed to return in the last 18 months, because the office runs data operations on its own — despite the office’s growth from 5,000 customers to 180,000 over the last two years.

Kraken has also been able to streamline operations, saving time and money. For example, near real-time weather data is crucial for the various Octopus Energy Group businesses for forecasting. Before the adoption of Databricks, each energy business was processing this key data separately, duplicating efforts and wasting resources. Now, that data is processed in a central location, and each business has access to the data it needs. With the consolidation made possible by Databricks, processing data is eight times less expensive than before.

In another example, when a bug prevented data replication, Kraken previously took three days to reload data for a client. Using Databricks, the same type of job was completed in eight hours.

Kraken is only just starting their exploration of Delta Sharing capabilities. “We are just scratching the surface of using Delta Sharing to help clients access data,” said Asensio. “We’re going to start creating more reports for them, making their lives easier by bundling data and sharing that with them. We’re excited about the possibilities.”

Hear more from Kraken about how they use Delta Sharing to scale their global business operations. Watch their 2024 Data + AI Summit session, “Enterprise Collaboration With Delta Sharing.”