Madhu Kotian is a Sr. Director Engineering at Northwestern Mutual. He is responsible for leading the mission, vision and implementation of different applications along with data and reporting capabilities to help provide meaningful insights to the Field Advisors.Madhu has over 25+ years of experience in the field of Information Technology with experience and expertise in data strategy, data engineering and data architecture. He has helped transform and migrate reporting platform to AWS using Spark on Databricks. He also manages the data organization of NM’s Investment Product System. Madhu is currently working on transforming the data landscape for Investment Products.
May 26, 2021 03:15 PM PT
The volume of available data is growing by the second (to an estimated 175 zetabytes by 2025), and it is becoming increasingly granular in its information. With that change every organization is moving towards building a data driven culture. We at Northwestern Mutual share similar story of driving towards making data driven decisions to improve both efficiency and effectiveness. Legacy system analysis revealed bottlenecks, excesses, duplications etc. Based on ever growing need to analyze more data our BI Team decided to make a move to more modern, scalable, cost effective data platform. As a financial company, data security is as important as ingestion of data. In addition to fast ingestion and compute we would need a solution that can support column level encryption, Role based access to different teams from our datalake.
In this talk we describe our journey to move 100’s of ELT jobs from current MSBI stack to Databricks and building a datalake (using Lakehouse). How we reduced our daily data load time from 7 hours to 2 hours with capability to ingest more data. Share our experience, challenges, learning, architecture and design patterns used while undertaking this huge migration effort. Different sets of tools/frameworks built by our engineers to help ease the learning curve that our non-Apache Spark engineers would have to go through during this migration. You will leave this session with more understand on what it would mean for you and your organization if you are thinking about migrating to Apache Spark/Databricks.