Today, I am excited to announce that I have officially joined Databricks as an Engineer on the Data Science team. This move comes after over a year of founding and running Staroid, a cloud-based platform that simplifies the delivery and deployment of open source projects at the enterprise level. I have been a heavy Apache SparkTM user since version 0.6, and prior to starting a company, I created Apache Zeppelin, an open-source data science notebook. My history with Spark and obvious passion for open source makes joining Databricks feel like a natural progression.
The journey to Databricks
So, why Databricks? At a technology level, Databricks has always been committed to open source and contributing to the broader community. In addition to Spark, the company has developed four other major open-source projects: Delta Lake, MLflow, Koalas and Delta Sharing. The Databricks Lakehouse Platform was even founded with “open” as one of its core principles.
At the “human” level, what drew me to Databricks was the leadership and very personable relationships we built from the get-go. The first time I interacted with Ali Ghodsi was when I first started Apache Zeppelin and working on my own startup, called Zepl (previously NFLabs)l, in South Korea. Ali contacted me about joining Databricks (I specifically remember shouting “Oh, yeah!” to myself – to give you an idea of my excitement). Since I had just founded a company, the timing wasn’t right, but it definitely piqued my interest.
After Zepl was acquired and I had started Staroid, Reynold Xin and Patrick Wendell once again asked if I would be interested in joining Databricks. My interest grew. This reminded me of Three visits to the cottage (三顾茅庐), a famous story from the period of the Three Kingdoms of China. Liu Bei, who founded one of the kingdoms, visited the cottage of Kongmin, a statesman, three times personally to meet him. It was only after the third visit that Kongmin accepted his service. He would later become one of the greatest talents of the Kingdoms. While I cannot compare myself to the talent of Kongmin, the story is an analogy to my own experience with the company and made me aware of how much Databricks prioritized its people.
What’s next
Databricks has been leading the industry ever since the creation of Spark, and Delta Lake and the Lakehouse platform enable data use cases -- even at the largest enterprises -- to unseen levels.
I’m very excited about the opportunity ahead. In my new role, one of the aspects I’m looking forward to the most is furthering development on Databricks Notebooks. While I’ve worked on standalone notebooks before, Databricks has really transformed the extent to what data science notebooks can do...and I think a lot more lies ahead. Nothing overrides my excitement for the people I’ll be working with. As I learned from starting a company: your product, business model and innovation are only as good as the talent and relationships behind it.