Lakehouse for Financial Services: Paving the Way for Data-Driven Innovation in FSIs

When it comes to “data-driven innovation,” financial service institutions (FSI) aren’t what typically come to mind. But with massive amounts of data at their potential disposal, this isn’t for lack of imagination. FSIs want to innovate but are continually slowed down by complex legacy architectures and vendor lock-in that prevent data and AI from becoming material business drivers.

Largely as a result of these challenges, the financial services industry has arguably seen little innovation in recent decades – even as other regulated sectors such as healthcare and education continue to break barriers. Even for the most established incumbents, a lack of innovation can quickly lead to being taken over by a new, digital-native company – a move some of us at Databricks call Tesla-fication. This is where one disruptive, data and AI-driven innovator becomes disproportionately more successful than the incumbents who previously dominated the space. One indication of this success can be found in the stock market. Today, Tesla boasts a $900+ billion market capitalization, making it worth more than the next 10 leading automotive competitors combined. Incumbency is no longer a moat.

In fact, we’re already starting to see Tesla-fication happening in financial services. Nubank, a Brazilian fintech launched in 2014, has quickly changed the competitive dynamics in its home country and beyond. Early on, Nubank disrupted the credit card market, by enabling online applications, as well as by extending credit to those with no credit history. Today, it uses bleeding-edge technology, data and AI to develop new products and services. Data science plays an essential role in every aspect of their business – from customer support to credit lines. Seven years after their launch, in December of 2021, Nubank became one of the largest IPOs in Latin America and briefly eclipsed the market capitalization of Brazil’s largest bank. Signs of Tesla-fication are emerging across all segments of financial services, from banking to insurance to capital markets. For FSIs, this means that the traditional sources of competitive advantage – capital and scale – no longer cut it. Today, transformation requires leaders to focus their investments on two modern sources of competitive advantage: data and people.

Introducing Lakehouse for Financial Services

Today, we’re thrilled to introduce Lakehouse for Financial Services to help bring data and people together for every FSI. Lakehouse for Financial Services addresses the unique requirements of FSIs via industry-focused capabilities, such as pre-built solutions accelerators, data sharing capabilities, open standards and certified implementation partners. With this platform, organizations across the banking, insurance and capital market sectors can increase the impact and time-to-value of their data assets, ultimately enabling data and AI to become central to every part of their business – from lending to insuring.

So, why is Lakehouse for Financial Services critical for success? When speaking with our customers, we identified the biggest challenges around transforming into a data-driven organization (and how Lakehouse addresses them):

• Risk of vendor lock-in: FSIs are particularly vulnerable to being stuck with proprietary data formats and technologies that stifle collaboration and innovation. Lakehouse is powered by open source and open standards, meaning that data teams can leverage the tools of their choice.
• No multi-cloud: Increasingly, regulators are asking FSIs to consider systemic risk arising from overreliance on a single vendor. Lakehouse solves this by offering full support for all major cloud vendors.
• Real-time data access for BI: The most recent data is typically the most valuable, but traditional architectures often make it a hurdle for data analysts to access it. With Lakehouse, data teams across functions always can access the most up-to-date, reliable data.
• Lack of support for all data sets: The fastest-growing data in FSIs is unstructured data sets (text, images, etc), which makes data warehouses less than ideal for critical use cases. Lakehouse handles all types of data - structured, semi-structured and unstructured - and even offers data sharing capabilities with leading providers such as Factset.
• Driving AI use cases. Although the regulated aspect of financial services makes it difficult to embrace and scale AI, the main hurdles are internal policies around risk adversity coupled with siloed infrastructures and legacy processes. Lakehouse makes AI accessible and transparent via MLflow; coupled with Delta Lake time travel capability, AI has been adopted as a next generation of model risk management for independent validation.

What makes Lakehouse for Financial Services Equipped to Tackle These Challenges?

We built Lakehouse for Financial Services specifically to tackle these challenges and empower organizations to find new ways to gain a competitive edge, innovate risk management and more, even within highly-regulated environments. Here’s how we’re doing just that:

Pre-built Solution Accelerators for Financial Services Use Cases

Lakehouse for Financial Services aligns with our 14 financial services solution accelerators, fully functional and freely available notebooks that tackle the most common and high-impact use cases that our customers are facing. These use cases include:

• Post-Trade Analysis and Market Surveillance: Using an efficient time series processing engine for market data, this library combines core market data and disparate alternative data sources, enabling asset managers to backtest investing strategies at scale and efficiently report on transaction cost analysis.
• Transaction Enrichment: This scalable geospatial data library enables hyper-personalization in retail banking to better understand customer transaction behavior required for next-gen customer segmentation and modern fraud prevention strategies.
• Regulatory Reporting: This accelerator streamlines the acquisition, processing and transmission of regulatory data following open data standards and open data sharing protocols.
• GDPR Compliance: Simplify the technical challenges around compliance to the “right to be forgotten” requirement while ensuring strict audit capabilities.
• Common Data Models: A set of frameworks and accelerators for common data models to address the challenges FSIs have in standardizing data across the organization.

Industry Open Source Projects

As part of this launch, we’re thrilled to announce that we have joined FINOS (FinTech Open Source Foundation) to foster innovation and collaboration in financial services. FINOS includes the world’s leading FSIs such as Goldman Sachs, Morgan Stanley, UBS and JP Morgan as members. Open Source has become a core strategic initiative for data strategies in financial services as organizations look to avoid complex, costly vendor lock-in and proprietary data formats. As part of FINOS, Databricks is helping to facilitate the processing and exchange of financial data throughout the entire banking ecosystem. This is executed via our Delta Lake and Delta Sharing integrations with recent open source initiatives led by major FSIs.

Databricks is working to help empower the standardization of data by significantly democratizing data accessibility and insights. Ultimately, we want to bring data to the masses. That’s why we recently integrated the LEGEND ecosystem with Delta Lake functionalities such as Delta Live Tables. Developed by leading financial services institutions and subsequently open-sourced through the LINUX Foundation, the LEGEND ecosystem allows domain experts and financial analysts to map business logic, taxonomy and financial calculations to data. Now integrated into the Lakehouse for Financial Services, those same business processes can be directly translated into core data pipelines to enforce high-quality standards with minimum operation overhead. Coupled with the Lakehouse query layer, this integration provides financial analysts with massive amounts of real-time data directly through the comfort of their business applications and core enterprise services.

Simple deployment of the Lakehouse environment

With Lakehouse for Financial Services, customers can easily automate security standards. More specifically, the utility libraries and scripts we’ve created for financial services deliver automated setup for notebooks and are tailored to help solve security and governance issues important to the financial services industry based on best practices and patterns from our 600+ customers.

A data model framework for standardizing data

In addition to solution accelerators, Lakehouse provides a framework for common data models to address the challenges FSIs have in standardizing data across the organization. For example, one solution accelerator is designed to easily integrate the Financial Regulation (FIRE) Data model to drive the standardization of data, serve data to downstream tools, enable AI quality checks and govern the data using Unity Catalog.

Open data sharing

Last year, we launched Delta Sharing, the world’s first open protocol for securely sharing data across organizations in real-time, independent of the platform on which the data resides. This is largely powered by our incredible ecosystem of partners, which we’re continuing to scale and grow. We are thrilled to announce that we have recently invested in Ticksmith, a leading SaaS platform that simplifies the online data shopping experience and was one of the first platforms to implement Delta Sharing. With the TickSmith and Databricks integration, FSIs can now easily create, package and deliver data products in a unified environment.

Implementation Partners

Databricks is working with consulting and SI partner Avanade to deliver risk management solutions to financial institutions. Built on Azure Databricks, our joint solution makes it easier for customers to rapidly deploy data into value-at-risk models to keep up with emerging risks and threats. By migrating to the cloud and modernizing data-driven risk models, financial institutions are able to reduce regulatory, operational compliance risks related and scale to meet increased throughput.

Databricks is also partnering with the Deloitte FinServ Governed Data Platform, a cloud-based, curated data platform meeting regulatory requirements that builds a single source of truth for financial institutions to intelligently organize data domains and approved provisioning points, enabling activation of business intelligence, visualization, predictive analytics, AI/ML, NLP and RPA.

Conclusion

Tesla-fication is starting to happen all around us. Lakehouse for Financial Services is designed to help our customers make a leapfrog advancement in their data and AI journey with pre-built solution accelerators, data sharing capabilities, open standards and certified implementation partners. We are on the mission to help every FSI become the Tesla of their industry.

Want to learn more? Check out this overview and see how you can easily get started or schedule a demo.