Originally a biologist, Dr Lucas Partridge switched to a career in software engineering. After 12 years with IBM he moved to GE Aviation where, since 2012, he has been supporting a team of Data Scientists based in Southampton, UK. His primary focus in the team is making sure the Data Scientists’ algorithms will scale with big data on Apache Spark.
GE Aviation has hundreds of data scientists and engineers developing algorithms. The majority of these people do not have the time to learn Apache Spark and continue to develop on local machines in Python or R. We also have lots of historical code that was not developed for Spark. However, the business wanted to deploy to a Spark environment for scalability, as quickly as possible. So how did we bridge the gap? A data scientist and software engineer will co-present to share how we approached the problem of building, unifying and scaling these algorithms.