Equiniti: From Zero to AI

Introduction

Equiniti (EQ), a global leader in shareholder, pension, and remediation services, leveraged Databricks to revolutionize its data-driven approach and enhance customer experience across 136 countries. Serving over 6,000 companies, EQ recognized the need to adapt to increasingly complex and regulated environments by harnessing the power of advanced analytics and generative AI. 

Given that timely access to information is critical to Equiniti’s customers’ success, they wanted to make data (and data-driven insights) the foundation of their operational and strategic approach. Equiniti aimed to implement more informed, efficient and effective business practices and take advantage of new developments in advanced analytics and GenAI that would enhance customer experience and drive internal innovation. 

To meet these goals, Equiniti needed to build a future-proof, secure and performant data platform that could support any existing or new data and AI applications. This blog describes how and why they selected Databricks Data Intelligence Platform to support their infrastructure and elaborates on the advanced use cases they have already explored by leveraging the Databricks Platform and Databricks Mosaic AI tools, such as the development of PensionGuru, their GenAI-powered chatbot. 

Step 1: Identify value and build solid data foundations

Rather than starting with the question, ‘What can we use AI for?’, Equiniti asked, ‘How can we provide new value to our clients, using high-quality, trusted data and modern tools and techniques?’

A few common themes emerged: gaining access to trusted data at scale, having the agility to experiment and move quickly and cost-effectively, expediting the enablement of domain subject matter experts (SMEs) and existing resources, and being able to quickly modernize their offerings to meet client needs.

From that initial work, Equiniti identified key requirements for a future cloud data and AI platform that would let them best unlock the value of their data:

●  Integrated data and AI governance: Without governance and control, there can be no value. Equiniti needed robust security features, access controls, automatic lineage and auditing that would help maintain compliance with regulatory requirements by tracking the flow and transformation of data across the platform and build trust with internal and external stakeholders and clients.

●  A unified and open platform: Another requirement was a simple architecture that could support data engineering, data science, advanced analytics, and GenAI. Equiniti wanted to eliminate silos and unnecessary data duplications and avoid being locked into a proprietary solution. They wanted a platform that was built on open standards and protocols. In addition, they needed support for both batch and streaming data sources in any format for GenAI workloads. With the distributed nature of their data and systems, a single platform that could become an analytical source of truth would be a huge step forward.

●  Cost optimization: Finally, Equinity needed scalable and optimized compute that enhanced data processing and lowered TCO with a true consumption-based model. The ability to begin with a low initial investment and then scale as needed was essential.

With these requirements in mind, Equiniti chose the Databricks Data Intelligence Platform as the backbone of their modern cloud data and AI platform. 

Step 2: Move fast and leverage integrated toolsets

Traditionally, it takes input from many different teams to evaluate separate components and distinct services that form a data platform, requiring the navigation of competing priorities and resources to implement it. However, Equiniti was able to quickly and easily deploy the Databricks Platform and explore all of its integrated capabilities. The option to experiment and scale quickly but cost-effectively meant that Equiniti could confidently make decisions in prototyping connectivity, data processing and analytical capabilities without significant up-front investment in time or cost. Once Equiniti established the primary use cases for their initial AI implementation, they collaborated with the Databricks team to create an initial architecture, as shown in Figure 1 below. Through a set of workshops, Databricks solution architects showcased how to best utilize the built-in capabilities of the platform; Equiniti also used comprehensive self-paced learning resources to upskill themselves.

One of the most valuable features of the Databricks Platform is Unity Catalog, a unified and open governance solution for data and AI. The ability to track the automatically captured lineage of the ingested data and how it was transformed and used in the model was key to building trust, understanding, and approval from Equiniti's InfoSec and Risk teams. Equiniti was able to demonstrate what and where data was being used, without additional cost, implementation overhead and time in managing a separate data catalog. In addition, Delta Sharing and Databricks Marketplace have been transformational, as they allowed Equiniti to externally share data with partners for the very first time. Gaining the ability to visualize data from multiple sources that were previously inaccessible or siloed and utilizing data from external providers without having to store and maintain petabyte-scale datasets has allowed Equiniti to quickly and easily develop insights that were previously out of reach. The ability for business teams to easily discover and use the same tools and data assets from a central, trusted source will continue to drive quality and value in their data platform.

For Equiniti's small engineering team, one of the biggest time-saving features of the Databricks Platform was LakeFlow Connect. Databricks LakeFlow provides built-in connectors for ingesting data from enterprise applications and databases. The ability to seamlessly create no-code integrations to our core platforms such as Workday, Salesforce and SQL Server massively reduced the time it took to make data available in Databricks for models to consume. It significantly reduced storage and compute costs and saved Equiniti months of development work compared to the traditional method of building API integrations and ETL processes to store and manage data. Equiniti's team could then focus on value multiplier areas such as creating Gen AI applications that would deliver value to the business.

Finally, developing these new GenAI applications requires a new type of "knowledge developer." These are domain SMEs who deeply understand the business (in Equiniti's case, the pensions market). These experts must have seamless access to tools and platforms to provide crucial feedback and make sure GenAI applications are delivering accurate and high-quality results. The ease of use and accessibility of the Databricks Platform platform made it easy for SMEs to effectively collaborate with the development and engineering teams in building GenAI applications. By leveraging their expertise and deep business insights, Equiniti was able to establish ground truth and receive valuable feedback, which helped fine-tune responses and generated content for use across the organization.

Step 3: Show value, deliver results, and keep innovating

One of Equiniti's first GenAI use cases was the development of their GenAI chatbot, PensionGuru. Given Equiniti's role in administering numerous pension plans, its employees often need to navigate and interpret an extensive volume of documents, including policies, trust deeds, and guidelines. PensionGuru addresses this challenge by offering instant, accurate responses, streamlining access to complex information and improving productivity.

The app significantly boosts business efficiency by automating document analysis and minimizing the time required to extract critical details, thereby reducing administrative overhead. Tasks that were taking many hours in the past are now completed in minutes.  PensionGuru empowers employees to quickly and accurately retrieve information, improving service delivery and decision-making processes. By utilizing advanced natural language processing, the app understands and processes user queries intelligently, delivering contextually relevant information from vast datasets. This innovation not only saves time but also enhances data-driven insights, allowing for a more strategic approach to pension scheme management.

To create PensionGuru, Equiniti began by taking thousands of pension documents, originally stored as PDF files, and loading them into a Databricks Volume, as shown in Figure 2 above. Then, Equiniti efficiently managed these unstructured files through Unity Catalog, right from the point of ingestion. The next step was to extract text from the documents, divided it into manageable chunks, and store the data in a Delta Table. Using Mosaic AI Vector Search with a serverless setup, Equiniti easily constructed a vector database to support search and retrieval capabilities.

To power the application, Equiniti leveraged Mosaic AI Model Serving to establish an LLM endpoint based on the powerful and cost-effective open source Meta Llama 3.1 70B model. Finally, Equiniti was able to seamlessly and securely deploy the chatbot to end users with Databricks Apps, a new simple and serverless solution for creating production-ready apps with built-in governance on top of the Databricks Data Intelligence Platform. The integrated Apps feature was a huge time saver and a massive game changer, as it removed the need for Equiniti's data team to deploy, manage and maintain the underlying infrastructure to support the application. The team could instead focus on delivering business value instead of spending time on mundane tasks like siloed services integration and IT infrastructure management. 

The initial PensionGuru results and feedback have been highly encouraging, and Equiniti continues to refine and enhance the application's performance through ongoing experimentation and model training. They are also exploring the incorporation of an AI agent framework that would allow them to further customize and extend the capabilities of PensionGuru, making it even more responsive and tailored to the specific needs of pension scheme administration. With this approach, Equiniti aims to deliver even greater accuracy and efficiency in processing and retrieving critical pension information. 

Conclusion

By selecting the Databricks Data Intelligence Platform, Equiniti has delivered a solution that is modular, extensible and capable of meeting all current and future data and AI needs. Databricks' ability to unify data engineering, data science, machine learning, and GenAI into a single solution allows Equiniti to achieve high levels of efficiency and scalability. This comprehensive approach is anchored around the foundational data governance with Unity Catalog, which promotes data accessibility across the organization. 

Additionally, the Databricks Platform's advanced tools and environments for AI model development and deployment have unlocked new opportunities, fueling both innovation and operational efficiency without sacrificing data integration, security and governance. 

"Although we are early on our Generative AI journey, we are confident in our ability to deliver meaningful business value with the Databricks Data Intelligence Platform."

— James West, Strategic Director of Data at Equiniti

Equiniti is now in the process of migrating, consolidating and bringing all their data sources into the Databricks environment and training and onboarding new users and have a large number of advanced analytics and AI use cases in the pipeline to deliver in the near future.

This blog post was jointly authored by James West (Equiniti) and Tomasz Kurzydym (Databricks).