Joint Blog Post: Bringing ORC Support into Apache SparkJuly 16, 2015 by Zhan Zhang, Cheng Liang and Patrick Wendell in Engineering Blog This is a joint blog post with our partner Hortonworks. Zhan Zhang is a member of technical staff at Hortonworks, where he collaborated...
Introducing Window Functions in Spark SQLJuly 15, 2015 by Yin Huai and Michael Armbrust in Engineering Blog Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake. In this blog post...
New Visualizations for Understanding Apache Spark Streaming ApplicationsJuly 8, 2015 by Tathagata Das, Shixiong Zhu and Andrew Or in Engineering Blog Earlier, we presented new visualizations introduced in Apache Spark 1.4.0 to understand the behavior of Spark applications. Continuing the theme, this blog highlights...
Guest blog: PMML Support in Apache Spark's MLlibJuly 2, 2015 by Vincenzo Selvaggio in Engineering Blog This is a guest blog from our friend Vincenzo Selvaggio who contributed this feature. He is a Senior Java Technical Architect and Project...
Understanding your Apache Spark Application Through VisualizationJune 22, 2015 by Andrew Or in Engineering Blog The greatest value of a picture is when it forces us to notice what we never expected to see. - John Tukey In...
Announcing Apache Spark 1.4June 11, 2015 by Patrick Wendell in Engineering Blog Today I’m excited to announce the general availability of Apache Spark 1.4! Spark 1.4 introduces SparkR, an R API targeted towards data scientists...
Announcing SparkR: R on Apache SparkJune 9, 2015 by Shivaram Venkataraman in Engineering Blog I am excited to announce that the upcoming Apache Spark 1.4 release will include SparkR, an R package that allows data scientists to...
Statistical and Mathematical Functions with DataFrames in Apache SparkJune 2, 2015 by Burak Yavuz and Reynold Xin in Engineering Blog We introduced DataFrames in Apache Spark 1.3 to make Apache Spark much easier to use. Inspired by data frames in R and Python...
Project Tungsten: Bringing Apache Spark Closer to Bare MetalApril 28, 2015 by Reynold Xin and Josh Rosen in Engineering Blog In a previous blog post , we looked back and surveyed performance improvements made to Apache Spark in the past year. In this...
Recent performance improvements in Apache Spark: SQL, Python, DataFrames, and MoreApril 24, 2015 by Reynold Xin in Engineering Blog Read Rise of the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data...