Spark Summit EU 2017 Recap and Reflections
“Dublin is now a truly cosmopolitan capital, with an influx of people, energy, and ideas infusing the ever-beguiling, multi-layered city with fresh flavors and kaleidoscopic colors,” writes the Lonely Planet. Add to this multi-layered city three days of Spark Summit, held at the Guinness barrel-shaped Dublin Convention Center, and you get an infusion of visionary...
Crossing The Ocean for Spark Summit EU
This is a guest post from Movile. Eiti Kimura and Flavio Clésio share their highlights and what they're looking forward to the most at Spark Summit EU in Dublin, Ireland. About the Authors: Eiti Kimura has over 15 years of experience working with software development. He holds a Master’s Degree in Electrical Engineering. He has...
The Biggest EU Summit Ever: Over One Hundred Presentations, Two Conference Days, and a Day of Training
SAVE 15% ON REGISTRATION* WITH CODE "DATABRICKS". This year’s Spark Summit Europe is poised to be the biggest yet and we here at Databricks could not be more excited to be hosting this event. The level of participation and continued growth of the summit is a clear indicator of how critical Apache Spark has become...
Bay Area Apache Spark Meetup at HPE/Aruba Networks Summary
On September 7th, we held our monthly Bay Area Apache Spark Meetup (BASM) at HPE/Aruba Networks in Santa Clara. We had two Apache Spark related talks: one from Aruba Networks’ Data Engineering team and other from Databricks’ Machine Learning team. For those who missed the meetup, below is the video and link to each presentation...
Learn about Apache Spark’s Memory Model and Spark’s State in the Cloud
Since Apache Spark 1.6, as part of the Project Tungsten, we started an ongoing effort to substantially improve the memory and CPU efficiency of Apache Spark’s backend execution and push performance closer to the limits of the modern hardware. This effort culminated in Apache Spark 2.0 with Catalyst optimizer and whole-stage code generation. Because Spark...
Databricks invites Colleen Lewis to Speak about Diversity in the Workplace
First I'll start with the sad truth. The technology industry at large has taken many hits over the years for discriminatory practices and underrepresentation of both women and minorities. Ageism, too, is a beast that lurks in the Valley. So as an employee, I'm happy to announce that Databricks has formed a Diversity Committee to address...
Learn about Apache Spark APIs and Best Practices
Since Apache Spark 1.3, Spark and its APIs have evolved to make them easier, faster, and smarter. The goal has been to unify concepts in Spark 2.0 and beyond so that big data developers are productive and have fewer concepts to grapple with. Built atop the Spark SQL engine, with Catalyst optimizer and whole-stage code...
Build, Scale, and Deploy Deep Learning Pipelines with Ease
At the Spark Summit in San Francisco in June, we announced an open-source project Deep Learning Pipelines. Deep Learning Pipelines provides high-level APIs for scalable deep learning in Python with Apache Spark, and the library leverages Spark for its two strongest facets: In the spirit of Spark and Spark MLlib, it provides easy-to-use APIs that...
Do your Streaming ETL at Scale with Apache Spark’s Structured Streaming
At the Spark Summit in San Francisco in June, we announced that Apache Spark’s Structured Streaming is marked as production-ready and shared benchmarks to demonstrate its performance compared to other streaming engines. Structured Streaming is a novel way to process streams. Not only does this new way make it easy to build end-to-end streaming applications,...
Bay Area Apache Spark Meetup at Pinterest Summary
On August 22, we held our monthly Bay Area Apache Spark Meetup (BASM) at Pinterest in San Francisco. In all, we had three Apache Spark related talks: two from Pinterest’s Data Engineering team, and one from Databricks’ Machine Learning team. For those who missed the meetup, below is the video and links to the presentation...
