How I Optimized Large-Scale Data Ingestion

Over the past three months, I had the opportunity to work as a Product Management Intern on the Ingestion team at Databricks. During this time, I worked on large-scale, deeply technical projects that enhanced my understanding of the data lakehouse architecture. I also gained a thorough understanding of how innovations like LakeFlow Connect, Auto Loader, and COPY INTO efficiently pull in data from an extensive array of data formats and sources. This experience has been transformative for my growth as a product manager, with Databricks' cultural principles elevating my ability to identify customer needs, craft impactful solutions, and deliver them successfully to market.

The Databricks Ingestion Team

Data ingestion is often the gateway to the Data Intelligence Platform. It focuses on bringing in data simply and efficiently, such that it is unified with other Databricks tools like Unity Catalog and Workflows. In this way, the data is made available for analysis, machine learning, and many other downstream applications.

Defining the problem

Given the potential impact of our work on nearly all customers using the Databricks platform, I was driven to deliver high-quality results. I began by focusing on Databricks’ core cultural principle of customer obsession. I had the chance to meet with and learn from nearly 30 customers—discussing their workloads, Jobs To Be Done (JTBD), and requests for the platform. Through these hypothesis-driven discussions, I gained insight into the various architectures our customers set up to ingest billions of files into the lakehouse. I observed that data ingestion into Databricks helps support critical use cases, such as generating a variety of dashboards or developing tailored AI chatbots for their organizations.

Defining the customer experience

A major aspect of my role involved clearly and concisely documenting insights through the data I gathered from customers. This included improving step-by-step user journeys, consolidating customer feedback, and analyzing competitors. Starting from first principles, I looked for opportunities to remove sharp edges, reduce the number of steps and context switches, and automate configurations wherever possible. Given the high visibility of these documents among leadership—occasionally receiving direct feedback from our CEO—having crisp and concise documentation was crucial.

Along the way, I collaborated closely with the world-class engineers on my team, working in a "two in a box" fashion. This allowed me to not only combine my customer insights with their deep technical expertise—but also to improve my own understanding of data engineering systems. And to validate the solutions that we designed, we gathered extensive feedback from distinguished engineers and product managers on complementary teams. Finally, I worked closely with UI/UX designers to translate these insights into intuitive interfaces.

Building Connections

Beyond this rewarding work, my internship was filled with unforgettable experiences that allowed me to explore San Francisco and bond with fellow interns. I attended my first major league baseball game watching the San Francisco Giants, visited the intriguing exhibits at the Exploratorium, and enjoyed the Bay Area R&D cruise (where we PM interns won second place in the cornhole tournament). Building relationships with such talented and wonderful people added a special dimension to my final college internship, creating lasting memories that made the summer even more enjoyable.

Conclusion

My internship at Databricks has been both challenging and rewarding. I gained deep technical insights, honed my communication skills, and thrived in cross-functional collaboration. These experiences have sharpened my skills and fueled my drive for product management. I’m excited to apply what I’ve learned to future opportunities and continue growing in this dynamic field.

If you want to work on cutting-edge projects alongside industry leaders, I highly encourage you to apply to work at Databricks! Visit the Databricks Careers page to learn more about job openings across the company. Or if you're ready to streamline your data ingestion process, explore how LakeFlow Connect can enable every practitioner to implement data pipelines at scale.