I have been working at Facebook for around a year on a team focused on the efficiency of our compute engines. Most of my time has been spent on setting up and properly using profilers to guide our optimizations. In college, I focused on Systems related courses and went to the ACM-ICPC World Finals, so performance, efficiency, and optimization has always been a focus of mine.
Identifying the most important areas for optimization can be difficult. One large factor of this difficulty is the fact that different production workloads have different needs and different bottlenecks. Important efficiency concerns for one workload might be negligible for another. As a result, it is very important that engineers working on efficiency know how to profile their workload and use data to prioritize the right things. There are multiple aspects of this which are worth discussing. Firstly, it is important to get the profiling right.
There are different types of profilers that have different pros and cons, and as an engineer looking at profiles, it is important to have a high level understanding of what is happening under the hood to put everything in context. For instance, at Facebook we use both a perf-based profiler called Strobelight that has minimal Java specific context, and the third party async-profiler, which is a Java specific solution that misses native processes such as transforms.
Both of these solutions are important for different use cases. Secondly, being able to interpret the results and view them in meaningful ways is vital to properly identify the biggest potential efficiency wins. There are many different ways of looking at the data which can be useful in different situations. At Facebook, we rely heavily on data driven investigations to determine where our efficiency efforts are best spent. This involves heavy use of profiling.
In this talk, I will go over some of the different profilers and other tools we employ to gather the data. I will also provide some examples of how we've uncovered issues in the past, both deliberately and incidentally, and the process involved to come to the conclusions we did.