A Summer of Personal and Professional Growth at Databricks

This summer, I worked at Databricks as a software engineering intern on the Growth team. By introducing two new features, user groups and API tokens, I simplified the user management experience and improved security for API authentication.

In this blog, I briefly discuss their use and merits and share my personal experience as an intern in Databricks’ engineering group.

Groups API

The Groups API allows admins to create and manage groups. Databricks previously exposed two groups: admins and all users. For fine-tuned control over access control (ACLs), admins had to set ACLs for each user individually. This is a pain point for large organizations in which members are often divided into groups with shared permissions or you want to add users’ ACLs in bulk. Groups API addresses both those pain points.

We released the Groups API in July. Since then, over a dozen organizations have used the Groups API, and created over two dozen groups.

API Tokens

Personal access tokens allow users to perform token authentication for REST APIs. Personal access tokens provide an expirable and secure form of authentication as an alternative to username/password authentication. This is important for organizations using Single Sign-On, as only admin users have passwords, preventing non-admin users from easily using REST APIs. This also improves security since tokens have a limited lifetime and can be immediately revoked without affecting the user's access in general.

We released personal access tokens in August

With user groups and API tokens, you can now make calls like the following.

<br>curl -H "Content-Type: application/json" -H "Authorization: Bearer yourtokenvalue" \
  -X POST \
  -d '{
    "group_name": "Growth team",
    "user_name": "[email protected]"
  }' \
https://yourinstance.cloud.databricks.com/api/2.0/groups/add-member

Personal and Professional Growth

In my spare time, I helped work on team tasks separate from my intern project, allowing me to grow and go beyond the assigned projects. I also participated in an engineering-wide hackathon, held every three months, to prototype a user interface (UI) for the groups API - and won a prize from the product management team.

This internship allowed me to improve my skills as a software developer, as my development involved the full stack of Databricks’ architecture; interns are treated like full-time employees and provide real value to the company. The most significant part of my internship was working with Databricks engineers, who are consistently brilliant and excited to build both innovative and sustainable products. The company's values of being customer obsessed, data-driven, owning it, and collaboration permeate throughout the organization.

As someone who has witnessed the full trajectory - I was an AMPLab intern with Matei Zaharia, Patrick Wendell and Ion Stoica when Spark was still a Ph.D. project in Berkeley - I can say with complete confidence that this team is poised to go the distance.