Engineering Manager/Director – Spark Runtime Performance

Mission

We are seeking a strategic technical leader who will be responsible for managing the engineering teams that ensure the performance of the Databricks Runtime (built on Apache Spark) is best in class. This group currently consists of two quickly growing teams:

• Customer Performance works on the next generation execution engine for the Databricks Runtime, and develops performance improvements ranging from Spark’s optimizer (Catalyst) to its code generation framework (Tungsten). They combine large amounts of data about performance bottlenecks with insights from the product management and field teams to improve performance and scalability where it matters most to our customers.
• Benchmarking Infrastructure builds the infrastructure to fully automate the process of running a wide range of performance benchmarks on every change to the Databricks Runtime, so that the engine becomes faster all the time.

In this role, you’ll report to the head of our European R&D centre in Amsterdam, The Netherlands. Constant collaboration with the broader company is crucial. In this role, you will work closely with engineering leaders across the company, as well as with product management, field engineering, recruiting and HR to ensure organizational alignment.

Outcomes

• Continuous improved performance and stability on the key metrics and benchmarks you’ll help identify, while not regressing on performance or correctness
• Grow a world class team of software engineers who work on Spark Runtime performance. Increase headcount by 10+ engineers in next 18 months, with continued growth beyond that according to product objectives
• Ensure consistent delivery of milestones and goals
• Manage technical debt, including long term technical architecture decisions and balance product roadmap

Competencies

• Previous experience building and leading teams in a complex technical domain, such as on distributed data systems or database internals
• Ability to attract, hire, and coach strong systems engineers who meet the Databricks hiring standards
• Management of growing team, including managing technical leads
• Ability to communicate across departments, with product management and customers

About Databricks

Databricks’ mission is to accelerate innovation for its customers by unifying Data Science, Engineering and Business. Founded by the original creators of Apache Spark™, Databricks provides a Unified Analytics Platform for data science teams to collaborate with data engineering and lines of business to build data products. Users achieve faster time-to-value with Databricks by creating analytic workflows that go from ETL and interactive exploration to production. The company also makes it easier for its users to focus on their data by providing a fully managed, scalable, and secure cloud infrastructure that reduces operational complexity and total cost of ownership. Databricks, venture-backed by Andreessen Horowitz, NEA and Battery Ventures, among others, has a global customer base that includes Viacom, Shell, and HP.  For more information, visit www.databricks.com.

Apache, Apache Spark and Spark are trademarks of the Apache Software Foundation.