Databricks, AWS, and SafeGraph Team Up For Easier Analysis of Consumer Behavior
This notebook was produced as a collaboration between SafeGraph and Databricks Ryan Fox Squire, Product Data Scientist @ SafeGraph Andrew Hutchinson, Solutions Architect @ Databricks Prasad Kona, Partner Solutions Architect @ Databricks We’ve created this Databricks notebook (.dbc download here), and published this blog, so that you can hit the ground running using SafeGraph Data...
Use Databricks Pools to Speed up your Data Pipelines and Scale Clusters Quickly
Data Engineering teams deploy short, automated jobs on Databricks. They expect their clusters to start quickly, execute the job, and terminate. Data Analytics teams run large auto-scaling, interactive clusters on Databricks. They expect these clusters to adapt to increased load and scale up quickly in order to minimize query latency. Databricks is pleased to announce...
Automate and Fast-track Data Lake and Cloud ETL with Databricks and StreamSets
Data lake ingestion is a critical component of a modern data infrastructure. But enterprises often run into challenges when they have to use this data for analytics and machine learning workloads. Consolidating high volumes of data from disparate sources into a data lake is difficult, even more so if it is from both batch and...
Using AutoML Toolkit’s FamilyRunner Pipeline APIs to Simplify and Automate Loan Default Predictions
Introduction In the post Using AutoML Toolkit to Automate Loan Default Predictions, we had shown how the Databricks Labs’ AutoML Toolkit simplified Machine Learning model feature engineering and model building optimization (MBO). It also had improved the area-under-the-curve (AUC) from 0.6732 (handmade XGBoost model) to 0.723 (AutoML XGBoost model). With AutoML Toolkit’s Release 0.6.1, we...
Celebrating Growth at Databricks and 1,000 Employees!
This November, Databricks hired our 1,000th full-time employee! Founded in Berkeley in 2013, our six co-founders created Databricks to help data teams solve the world’s toughest problems - and since then, we’ve grown tremendously! Not only have we had some major milestones like our Microsoft partnership, resulting in Azure Databricks and the creation of new...
New Microsoft Azure Data Warehouse Service and Azure Databricks Combine Analytics, BI, and Data Science
In the last two years since it first became available, thousands of companies have adopted Azure Databricks, making it one of the fastest growing data and AI services on Microsoft Azure. Customers now process over 2 exabytes per month with millions of server-hours spinning up every day. All of this is driven by organizations like...
Solving the Challenge of Big Data Cloud Migration with WANdisco, Databricks and Delta Lake
This is a guest blog from Paul Scott-Murphy, VP of Product Management, Big Data / Cloud at WANdisco. Migrating from Hadoop on-premises to the cloud has been a common theme in recent Databricks blog posts and conference sessions. They’ve identified key considerations, highlighted partnerships and described solutions for moving and streaming data to the cloud...
Scalable near real-time S3 access logging analytics with Apache Spark™ and Delta Lake
The original blog is from Viacheslav Inozemtsev, Senior Data Engineer at Zalando, reproduced with permission. Introduction Many organizations use AWS S3 as their main storage infrastructure for their data. Moreover, by using Apache Spark™ on Databricks they often perform transformations of that data and save the refined results back to S3 for further analysis. When...
Why we are investing 100 million euros in our European Development Center
A few days ago, we announced an investment of 100 million euros in our European Development Center in Amsterdam. I want to take a moment to describe why this is a pivotal moment for Databricks and why Amsterdam is a cornerstone of our growth strategy. Solving the Hardest Data Problems at Scale Our Unified Data...
Scaling Hyperopt to Tune Machine Learning Models in Python
Try the Hyperopt notebook to reproduce the steps outlined below and watch our on-demand webinar to learn more. Hyperopt is one of the most popular open-source libraries for tuning Machine Learning models in Python. We’re excited to announce that Hyperopt 0.2.1 supports distributed tuning via Apache Spark. The new SparkTrials class allows you to scale out...