Bridging the Gap Between Data Scientists and Software Engineers - Deploying Legacy Python Algorithms to Apache Spark with Minimum Pain - Databricks

Bridging the Gap Between Data Scientists and Software Engineers – Deploying Legacy Python Algorithms to Apache Spark with Minimum Pain

GE Aviation has hundreds of data scientists and engineers developing algorithms. The majority of these people do not have the time to learn Apache Spark and continue to develop on local machines in Python or R. We also have lots of historical code that was not developed for Spark. However, the business wanted to deploy to a Spark environment for scalability, as quickly as possible. So how did we bridge the gap? A data scientist and software engineer will co-present to share how we approached the problem of building, unifying and scaling these algorithms.



« back
About Lucas Partridge

Originally a biologist, Dr Lucas Partridge switched to a career in software engineering. After 12 years with IBM he moved to GE Aviation where, since 2012, he has been supporting a team of Data Scientists based in Southampton, UK. His primary focus in the team is making sure the Data Scientists' algorithms will scale with big data on Apache Spark.

About Peter Knight

Dr Peter Knight is a senior data scientist with the GE Aviation UK data science team. He has over fifteen years' experience of developing analytics on aviation data sets. He has an Aerospace PhD and 1st class master's degree from the University of Southampton. He has 4 papers and 5 patents to his name. He was awarded the Derek George Astridge Safety in Aerospace Award in 2005 and has also won a number of GE awards including 'Global Customer Champion Award', 'Most Inspirational Employee Award' and open innovation challenge winner.