Fast Parallel Testing at Databricks with Bazel
The Databricks Developer Tools team recently completed a project to greatly speed up the pull-request (PR) validation workflows for many of our engineers: by massively parallelizing our tests, validation runs that previously took ~3 hours now complete in ~40 minutes. This blog post will dive into how we leveraged the Bazel build tool to achieve...
Announcing the MLflow 1.1 Release
We’re excited to announce today the release of MLflow 1.1. In this release, we’ve focused on fleshing out the tracking component of MLflow and improving visualization components in the UI. Some of the major features include: Automatic logging from TensorFlow and Keras Parallel coordinate plots in the tracking UI Pandas DataFrame based search API Java...
Automated Hyperparameter Tuning, Scaling and Tracking: On-Demand Webinar and FAQs now available!
On June 20th, our team hosted a live webinar—Automated Hyperparameter Tuning, Scaling and Tracking on Databricks—with Joseph Bradley, Software Engineer, and Yifan Cao, Senior Product Manager at Databricks. Automated Machine Learning (AutoML) has received significant interest recently because of its ability to shorten time-to-value for data science teams and maximize the predictive performance of models....
How Databricks IAM Credential Passthrough Solves Common Data Authorization Problems
In our first blog post, we introduced Databricks IAM Credential Passthrough as a secure, convenient way for customers to manage access to their data. In this post, we'll take a closer look at how passthrough compares to other Identity and Access Management (IAM) systems. If you’re not familiar with passthrough, we suggest reading the first...
Announcing Databricks Runtime 5.5 and Runtime 5.5 for Machine Learning
Databricks is pleased to announce the release of Databricks Runtime 5.5. This release includes Apache Spark 2.4.3 along with several important improvements and bug fixes as noted in the latest release notes [Azure|AWS]. We recommend all users upgrade to take advantage of this new runtime release. This blog post gives a brief overview of some...
Brickster Spotlight: Meet Heather
At Databricks, we build platforms to enable data teams to solve the world’s toughest problems and we couldn’t do that without our wonderful Databricks Team. “Teamwork makes the dreamwork” is not only a central function of our product but it is also central to our culture. Learn more about Heather, our VP of Mid Market...
Migrating Transactional Data to a Delta Lake using AWS DMS
Note: We also recommend you read Efficient Upserts into Data Lakes with Databricks Delta which explains the use of MERGE command to do efficient upserts and deletes. Challenges with moving data from databases to data lakes Large enterprises are moving transactional data from scattered data marts in heterogeneous locations to a centralized data lake. Business data...
Making the Move to Amsterdam: Bilal Aslam
While we are proud of our Berkeley roots, Databricks now calls many cities around the world our home. In addition to offices in London, Singapore, New York and our headquarters in San Francisco, we have one of our major engineering hubs in the fast-growing European R&D center, Amsterdam. Databricks offers the exciting opportunity to relocate...
Brickster Spotlight: Meet Vida
At Databricks, we build platforms to enable data teams to solve the world’s toughest problems and we couldn’t do that without our wonderful Databricks Team. “Teamwork makes the dreamwork” is not only a central function of our product but it is also central to our culture. Learn more about Vida, our Sr. Director of Field...
Getting Data Ready for Data Science: On-Demand Webinar and Q&A Now Available
On June 25th, our team hosted a live webinar — Getting Data Ready for Data Science — with Prakash Chockalingam, Product Manager at Databricks. Successful data science relies on solid data engineering to furnish reliable data. Data lakes are a key element of modern data architectures. Although data lakes afford significant flexibility, they also face...
