Carlos is a staff software engineer and leads Data Platform at Abnormal Security, a cybersecurity company using ML to stop sophisticated email threats. Prior to joining Abnormal, Carlos worked on high throughput, low latency ads serving systems at Twitter via its acquisition of TellApart. Prior to TellApart, Carlos worked on the AWS Platform team at Amazon. He holds a Bachelor’s degree in Computer Science from Carnegie Mellon University and is originally from Valencia, Venezuela.
May 27, 2021 04:25 PM PT
At Abnormal Security, Spark has played a fundamental role in helping us create an ML system that detects thousands of sophisticated email threats every day. Initially, we set up our Spark infrastructure using YARN on EMR because we had previous experience with it. But after growing very quickly in a short amount of time, we found ourselves spending too much time solving problems with our Spark infrastructure and less time solving problems for our customers. Given we’re in a high growth environment where the only constant is change, we asked ourselves: aren’t these problems only going to get worse as we add more employees, more products, and more data?
Over the past few months, Abnormal Security executed a full migration of our Spark infrastructure to Databricks, not only improving cost, operational overhead, and developer productivity, but simultaneously laying the foundation for a modern Data Platform via the Lakehouse architecture.
In this talk, we’ll cover how we executed the migration in a few months' time, from pre-Databricks-POC, through the POC, through the migration itself. We’ll talk about how to really figure out exactly what it is that you care about when evaluating Databricks, splitting out the must-haves from the nice-to-haves, so that you can best utilize the Databricks trial and have concrete and measurable results. We’ll talk about the work we did to execute the migration and the tooling we created to minimize downtime and properly measure performance and cost. Finally, we’ll talk about how the move to Databricks not only solved problems with our legacy Spark infrastructure, but will save us huge amounts of time in the long-run from adopting a Lakehouse architecture.