We are looking for talented engineers who are passionate about large-scale, high availability systems, and who want to make a strong impact on the growth of the company. The workflow team operates at the core of the Databricks infrastructure: it orchestrates all the workloads scheduled by the customers of Databricks, from the one-off experiment to the massive multi-day query running on hundreds of machines. As part of this team, you will be responsible for maintaining mission-critical operations, and at the same time pushing the boundary in terms of integrating with innovative AI solutions built on top of the Databricks platform. The responsibility covers mainly the backend service itself and all its adjacent functions, from low-level systems in Scala to dashboards and health monitoring, and public APIs for remote management.
- Maintain the existing backend of Databricks’ core scheduling service
- Own (as a team) the alerting and deployment systems around the backend
- Scale the scheduling service by 10x
- Own the testing infrastructure of the backend.
- Architect the workflow management component of Databricks
- Experience with backend systems written in java, scala, go, or c++
- Understanding of high-concurrency, reliable services
- Production quality coding standards and patterns
- Strong foundation in algorithms and data structures and their real world use cases
- Passionate about your customers. You want to work with them and develop an amazing experience for them.
- BS in Computer Science, Math, related technical field or equivalent practical experience in software engineering
- Strong problem solving skills and ability to work well with teammates
- Experience with SAAS/PAAS services (experience with developing cloud-based services strongly desirable)
Databricks’ mission is to accelerate innovation for its customers by unifying Data Science, Engineering and Business. Founded by the original creators of Apache Spark™, Databricks provides a Unified Analytics Platform for data science teams to collaborate with data engineering and lines of business to build data products. Users achieve faster time-to-value with Databricks by creating analytic workflows that go from ETL and interactive exploration to production. The company also makes it easier for its users to focus on their data by providing a fully managed, scalable, and secure cloud infrastructure that reduces operational complexity and total cost of ownership. Databricks, venture-backed by Andreessen Horowitz, NEA and Battery Ventures, among others, has a global customer base that includes Viacom, Shell, and HP. For more information, visit www.databricks.com.
Apache, Apache Spark and Spark are trademarks of the Apache Software Foundation.