Joseph de Castelnau currently serves as Senior Vice President of Software Engineering for Nielsen – MROI Solutions. He joined Nielsen following their acquisition of Marketing Analytics where he previously worked as VP – Engineering. He specializes in hands-on, product-oriented agile software and technology management and is actively involved in the Distributed Data and Technology Lead communities in Chicago. Last year, he was nominated as best technology manager in the Chicago area by his peers. He holds a patent in algorithmic trading workflow automation.
• 17+ years software engineering experience, 10 years analytics software development
• 6 years Nielsen & Marketing Analytics, Inc.
• 7 years at Townsend Analytics (RealTick), acquired by Lehman Brothers
• 4 years building E-commerce solutions
• Leads Nielsen MROI Solutions Software Engineering
• Master in Information Systems from Université Paul-Sabatier, Toulouse, France
Despite the many advantages of using Spark in the cloud, the inertia of organizational concerns can at times seem impossible to overcome. Simply put, moving to the cloud may be daunting and not considered a viable option for many organizations due to these concerns. This session examines one such situation at Nielsen - we will share the technical challenges encountered when developing on a local Spark cluster and highlight solutions we deployed to overcome this. We will conclude with technical strategies for mitigating organizational resistance to doing data analytics in the cloud, helping you avoid these issues from the start, that are all too common for on-prem deployments. Topics include: data quality management, data science development life-cycle, environment isolation with Docker, orchestration with Luigi, and performance tuning on a fixed cluster.