Engineering | Databricks Blog

Page 61

Real-Time End-to-End Integration with Apache Kafka in Apache Spark’s Structured Streaming

April 4, 2017 by Sunil Sitaula in Engineering

View the Notebook in Databricks Community Edition Structured Streaming APIs enable building end-to-end streaming applications called continuous applications in a consistent, fault-tolerant manner...

Next Generation Physical Planning in Apache Spark

April 1, 2017 by Aaron Davidson, Eric Liang and Thomas Desrosiers in Engineering

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. — Andrew Tanenbaum, 1981 magine a cold, windy...

On-Demand Webinar and FAQ: Apache Spark MLlib 2.x: How to Productionize your Machine Learning Models

March 28, 2017 by Richard Garris and Jules Damji in Engineering

On March 9th, we hosted a live webinar— Apache Spark MLlib 2.x: How to Productionize your Machine Learning Models —to address the following...

Analyse One Year of Radio Station Songs Aired with Apache Spark, Spark SQL, Spotify, and Databricks

March 27, 2017 by Paul Leclercq in Solutions

Read Rise of the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data...

Voice from Facebook: Using Apache Spark for Large-Scale Language Model Training

February 28, 2017 by Tejas Patil and Jing Zheng in Engineering

This is a guest post from Facebook. Tejas Patil and Jing Zheng, software engineers in the Facebook engineering team, show how to use...

Working with Complex Data Formats with Structured Streaming in Apache Spark 2.1

February 23, 2017 by Burak Yavuz, Michael Armbrust, Tathagata Das and Tyson Condie in Engineering

In part 1 of this series on Structured Streaming blog posts, we demonstrated how easy it is to write an end-to-end streaming ETL...

Processing a Trillion Rows Per Second on a Single Machine: How Can Nested Loop Joins be this Fast?

February 16, 2017 by Reynold Xin, Ala Luszczak and Bogdan Raducanu in Engineering

This blog post describes our experience debugging a failing test case caused by a cross join query running “too fast.” Because the root...

Intel’s BigDL on Databricks

February 8, 2017 by Sue Ann Hong and Joseph Bradley in Engineering

Try this notebook on Databricks Intel recently released its BigDL project for distributed deep learning on Apache Spark. BigDL has native Spark integration...

Real-time Streaming ETL with Structured Streaming in Apache Spark 2.1

January 19, 2017 by Tathagata Das, Michael Armbrust and Tyson Condie in Engineering

Explore why lakehouses are the data architecture of the future with the father of the data warehouse, Bill Inmon. Try this notebook in...