Before joining Apple 4.5 years ago, Tim worked as software engineer at Fermi National Accelerator Laboratory as one of the key contributors to dCache( https://www.dcache.org/about/) Mass Storage System widely used in High Energy Physics including many Tier 1 and Tier 2 Large Hadron Collider Worldwide LHC Computing Grid (https://home.cern/science/computing/grid-system-tiers). Later he worked in HERE, a Nokia’s Digital Maps company, on Map based cloud services for Automative Customers. As a Big Data Engineer @ Apple , Tim develops a data analytics software which provides key business isights for some of Apple’s most exciting iCloud products and services.
May 26, 2021 12:05 PM PT
Big data presents new challenges for protection of privacy and integrity of sensitive information. Straightforward application of traditional file encryption and MAC techniques can't cope with staggering volumes of data, flowing in modern analytic pipelines.
Apple addresses these challenges by leveraging the new capabilities in the Apache Parquet format. We work with the Apache Parquet community on a modular data security mechanism, that provides privacy and integrity guarantees for sensitive information at scale; the encryption specification has been approved and released by the Apache Parquet Format project. Today, there are two open source implementations of this specification - in Apache Arrow (C++) and in Apache Parquet-MR (Java) repositories. The latter has just been released in the parquet-mr-1.12 version - which means the Apache Spark and other Java/Scala based analytic frameworks can start working with Apache Parquet encryption.
In this talk, Gidon Gershinsky and Tim Perelmutov will outline the challenges of protecting the privacy of data at scale and describe the Apache Parquet encryption technology security approach. We will give a quick intro to usage of Apache Parquet encryption API in pure Java and in Apache Spark applications. We will also discuss the roadmap of the community work on new encryption features and on deeper integration with Apache Spark and other analytic frameworks. Finally, we will show a demo of the Apache Parquet modular encryption in action, sharing our learnings using it at scale.