How Willis Towers Watson Transformed Work & Rewards with Data Intelligence Powered by Unity Catalog

Willis Towers Watson (WTW) is a multinational company that provides a wide range of services in commercial insurance brokerage, risk management, employee benefits, and actuarial analysis—serving 91% of Fortune Global 500 companies. WTW’s Work and Rewards division delivers data-driven insights, technology solutions, and services to support clients’ hiring and retention decisions through valuable market-based data.

Our core business depends on robust data transformation and governance capabilities. At its heart is our proprietary calculation engine, which powers our salary survey reports. Recently, we began a strategic migration from SQL Server on Azure VMs to Azure Databricks, unlocking significant business advantages. This shift has accelerated report generation by 10x, reduced our data environments by 50%, and cut storage costs to one-third of our previous SQL Server expenses.

Databricks Unity Catalog has been a key enabler of this transformation, redefining our approach to data governance and management. In this blog, we’ll share the challenges we encountered, how Databricks and Unity Catalog helped us overcome them, and the impact this transformation has had on our business.

Challenges: Scaling, governance, and performance Bottlenecks 

Before implementing Databricks and Unity Catalog, we faced several technical and organizational challenges stemming from limitations in our existing tech stack.

Limited scalability driving costs, slow ETL, and delayed time to market: Our report calculation business is seasonal, with distinct peak periods for data acquisition and report generation. However, our legacy database server lacked scalability, forcing us to provision large database instances year-round—resulting in significant idle time and unnecessary costs. Scaling web and app servers during peak periods required custom logic in our application code. WTW’s largest reports often took between 10 to 36 hours to generate, especially for our most critical surveys. These long runtimes created friction in customer relationships and added operational costs by slowing down other processes.

Inflexible schema design in legacy SQL Server implementation limiting agility: Our use of rigid relational data models made it difficult to adapt to evolving data requirements. This inflexibility led to high development and maintenance costs.

Data and environment duplication complicating governance: We maintained data across more than five environments to meet processing and regional compliance needs, incurring over $300K annually. To ensure fast data retrieval, we relied on approximately 70 TB of P40 storage disks in SQL Server on Azure VMs—adding to our infrastructure expenses.

Compliance challenges in managing regional data: As a global organization, WTW must adhere to regional data privacy and compliance regulations. Due to limitations in our prior system, we chose not to collect personally identifiable information (PII) to avoid increased complexity. Additionally, all data was stored in a single region, limiting our ability to meet localized compliance requirements.

Insufficient data lineage and auditing: We lacked automated tools to track data lineage and audit changes—both critical for troubleshooting and understanding the downstream impact of data modifications.

Improvements with Databricks Data Intelligence Platform and Unity Catalog

In our next-generation system built on Databricks and Unity Catalog, we are already realizing several benefits that enable our business to scale more effectively.

10x faster report generation unlocks business growth: Reports that previously took 10 hours to generate now complete in under 50 minutes—a 10x improvement in calculation time. Most reports are seeing 5x to 20x performance gains. This acceleration is especially valuable for our largest reports, which used to take over a day to process. Faster report generation decreases time-to-market, unlocks upsell opportunities by delivering insights to clients more quickly and frequently, and frees up our team to focus on new client engagements.

30% cost savings through Unity Catalog’s efficient data management: By centralizing data access with Unity Catalog, we've reduced storage and infrastructure costs by 30%. Previously, our system relied on six separate environments to meet compliance and operational needs. Today, a single Databricks workspace can seamlessly access multiple catalogs across regions, eliminating duplication and simplifying governance. In addition, storing data in Unity Catalog managed tables—with built-in compression and performance optimizations—has cut storage costs to one-third of our previous SQL Server expenses. These managed tables also streamline operations by automatically optimizing table layouts based on query patterns.

Improved compliance and data residency capabilities: We can now meet privacy requirements from the European Union, Germany, China, California, and other jurisdictions using Unity Catalog’s flexible architecture, paired with Databricks’ global availability across Azure regions. Storing data closer to where it’s accessed has the potential to reduce cloud egress costs and improve report performance. Unity Catalog’s fine-grained access controls and integration with Entra ID groups also give us better control over PII—benefiting both our compliance posture and our customers.

Better auditability and lineage increased developer productivity by 33%: Unity Catalog’s lineage features, along with Databricks’ orchestration tools and AI capabilities, enable better exploratory data analysis and deeper understanding of our datasets—especially as new reports are developed. Based on our estimates, reports that previously took up to three weeks to develop now take two weeks or less.

Databricks training accelerated team ramp-up: The availability of both free and paid Databricks training programs enabled teams with no prior experience in Databricks or Python to become productive within 2–3 months. By the end of eight months, most developers reached approximately 75% proficiency.

Looking forward

The migration to Databricks and Unity Catalog has been a game-changer for the Work & Rewards division at Willis Towers Watson. We’ve streamlined data governance, improved compliance, reduced costs, and dramatically accelerated report generation.

Looking ahead, we plan to harness Databricks’ AI-powered capabilities—such as AI/BI Genie and automated ML solutions—to build new data-driven products. With a modernized data infrastructure in place, we are well-positioned to drive innovation, enhance customer experiences, and unlock new revenue opportunities.