Integrating Apache Airflow with Databricks
This blog post is part of our series of internal engineering blogs on Databricks platform, infrastructure management, integration, tooling, monitoring, and provisioning. Today...
Serverless Continuous Delivery with Databricks and AWS CodePipeline
Two characteristics commonly mark many companies' success. First, they quickly adapt to new technology. Second, as a result, they gain technological leadership and...
Benchmarking Big Data SQL Platforms in the Cloud
For a deeper dive on these benchmarks, watch the webinar featuring Reynold Xin. Performance is often a key factor in choosing big data...
Introducing Apache Spark 2.2
Today we are happy to announce the availability of Apache Spark 2.2.0 on Databricks as part of the Databricks Runtime 3.0. This release...
4 SQL High-Order and Lambda Functions to Examine Complex and Structured Data in Databricks
Read Rise of the Data Lakehouse to explore why lakehouses are the data architecture of the future with the father of the data...
Declarative Infrastructure with the Jsonnet Templating Language
This blog post is part of our series of internal engineering blogs on Databricks platform, infrastructure management, integration, tooling, monitoring, and provisioning. At...
Shell Oil Use Case: Parallelizing Large Simulations with Apache SparkR on Databricks
This blog post is a joint engineering effort between Shell’s Data Science Team ( Wayne W. Jones and Dennis Vallinga ) and Databricks...
Managing and Securing Credentials in Databricks for Apache Spark Jobs
Since Apache Spark separates compute from storage, every Spark Job requires a set of credentials to connect to disparate data sources. Storing those...
Analysing Metro Operations Using Apache Spark on Databricks
This is a guest blog from EY Advisory Data & Analytics team, who have been working with Sporveien in Oslo building a platform...
Five Spark SQL Utility Functions to Extract and Explore Complex Data Types
Try this notebook on Databricks For developers, often the how is as important as the why . While our in-depth blog explains the...