Rahul Potharaju

Principal Engineering Manager, Microsoft

Rahul Potharaju is a Principal Engineering Manager at Microsoft’s Azure Data group working on Azure Synapse Analytics. Previously, he worked as a Researcher in the Cloud and Information Services Lab (CISL) at Microsoft. He earned his Computer Science PhD degree from Purdue University in a joint industrial collaboration with Microsoft Research and Computer Science Master’s degree from Northwestern University. He is a recipient of the Motorola Engineering Excellence award and the Purdue Diamond Award. Rahul’s work has been adopted by several business groups inside Microsoft and has won the Microsoft Trustworthy Reliability Computing Award.

UPCOMING SESSIONS

PAST SESSIONS

Hyperspace: An Indexing Subsystem for Apache SparkSummit 2020

At Microsoft, we store datasets (both from internal teams and external customers) ranging from a few GBs to 100s of PBs in our data lake. The scope of analytics on these datasets ranges from traditional batch-style queries (e.g., OLAP) to explorative, 'finding needle in a haystack' type of queries (e.g., point-lookups, summarization etc.). Resorting to linear scans of these large datasets with huge clusters for every simple query is prohibitively expensive and not the top choice for many of our customers, who are constantly exploring (and demanding!) ways to reducing their operational costs - incurring unchecked expenses are their worst nightmare. Over the years, we have seen a huge demand for bringing 'indexing' capabilities that come de facto in the traditional database systems world into Apache Spark.

Among many ways to improve query performance and lowering resource consumption in database systems, indexes are particularly efficient in providing tremendous acceleration for certain workloads since they could reduce the amount of data scanned for a given query and thus also result in lowering resource costs. In this talk, we present our experiences in designing, implementing and operationalizing Hyperspace, an indexing subsystem for Apache Spark that introduces the ability for users to build, maintain (through a multi-user concurrency model) and leverage indexes (automatically, without any changes to their existing code) on their data (e.g., CSV, JSON, Parquet etc.) for query/workload acceleration. We will cover the necessary foundations behind our indexing infrastructure including the API design, how we leveraged Spark's Catalyst optimizer to provide a transparent user experience and also discuss our development roadmap as we work towards open sourcing our work for the benefit of the broader community. Through presentation, benchmarks, code examples and notebooks, this will be one fun session, so come join us as we get started on this journey.

.NET for Apache SparkSummit Europe 2019

We present a new, free, open-source framework aimed at making Spark accessible to millions of .NET developers. In this session we will provide a high level overview of the .NET bindings for Spark effort, demonstrate some key capabilities on how you can use and get involved with the effort, and also cover how you can use the .NET bindings for Spark with other .NET frameworks like Databricks' Delta for building E2E real-time analytics solutions. This will be one fun session with demos galore, so come join us as we get started on the .NET bindings for Spark journey!

Introducing .NET Bindings for Apache SparkSummit 2019

We present a new, free, open-source framework aimed at making Spark accessible to millions of .NET developers. In this session we will provide a high level overview of the .NET bindings for Spark effort, demonstrate some key capabilities on how you can use and get involved with the effort, and also cover how you can use the .NET bindings for Spark with other .NET frameworks like ML.NET for building E2E real-time analytics solutions. This will be one fun session with demos galore, so come join us as we get started on the .NET bindings for Spark journey!