Semi-Structured Communication Data with Delta Lake and Microservices

Our talk presents a solution for efficiently storing and managing semi-structured communication data for eDiscovery. With the rise of short-form messaging platforms, the volume and complexity of data used in legal processes have significantly increased. Delta Lake is particularly suited to handle the complexities and requirements of eDiscovery data. Our approach involves developing microservices that provide abstracted layers of data access, enabling efficient querying, processing, and analysis of semi-structured communication data using technologies like delta-rs, Arrow Flight, and DataFusion. Our solution is a strategic response to the changing landscape of digital communication and its implications for legal processes. Our talk will provide details on the implementation and integration of Delta Lake and these microservices, of interest to software engineers, data architects, and anyone interested in technology and the legal industry.