10 Minutes from pandas to Koalas on Apache Spark
This is a guest community post from Haejoon Lee, a software engineer at Mobigen in South Korea and a Koalas contributor. pandas is a great tool to analyze small datasets on a single machine. When the need for bigger datasets arises, users often choose PySpark. However, the converting code from pandas to PySpark is not...
Developing Shiny Applications in Databricks
Join our live webinar hosted by Data Science Central on March 12 to learn more We are excited to announce that you can now develop and test Shiny applications in Databricks! Inside the RStudio Server hosted on Databricks clusters, you can now import the Shiny package and interactively develop Shiny applications. Once completed, you can...
Announcing Databricks Runtime 5.5 with Conda (Beta)
Databricks is pleased to announce the release of Databricks Runtime 5.5 with Conda (Beta). We introduced Databricks Runtime 5.4 with Conda (Beta), with the goal of making Python library and environment management very easy. This release includes several important improvements and bug fixes as noted in the latest release notes [Azure|AWS]. We recommend all users...
Automated Hyperparameter Tuning, Scaling and Tracking: On-Demand Webinar and FAQs now available!
On June 20th, our team hosted a live webinar—Automated Hyperparameter Tuning, Scaling and Tracking on Databricks—with Joseph Bradley, Software Engineer, and Yifan Cao, Senior Product Manager at Databricks. Automated Machine Learning (AutoML) has received significant interest recently because of its ability to shorten time-to-value for data science teams and maximize the predictive performance of models....
Announcing Databricks Runtime 5.5 and Runtime 5.5 for Machine Learning
Databricks is pleased to announce the release of Databricks Runtime 5.5. This release includes Apache Spark 2.4.3 along with several important improvements and bug fixes as noted in the latest release notes [Azure|AWS]. We recommend all users upgrade to take advantage of this new runtime release. This blog post gives a brief overview of some...
Enhanced Hyperparameter Tuning and Optimized AWS Storage with Databricks Runtime 5.4 ML
We are excited to announce the release of Databricks Runtime 5.4 ML (Azure | AWS). This release includes two Public Preview features to improve data science productivity, optimized storage in AWS for developing distributed applications, and a number of Python library upgrades. To get started, you simply select the Databricks Runtime 5.4 ML from the...
Introducing Databricks Runtime 5.4 with Conda (Beta)
We are excited to introduce a new runtime: Databricks Runtime 5.4 with Conda (Beta). This runtime uses Conda to manage Python libraries and environments. Many of our Python users prefer to manage their Python environments and libraries with Conda, which quickly is emerging as a standard. Conda takes a holistic approach to package management by...
Databricks Runtime 5.3 ML Now Generally Available
We are excited to announce the general availability (GA) of Databricks Runtime for Machine Learning, as part of the release of Databricks Runtime 5.3 ML. Built on top of Databricks Runtime, Databricks Runtime ML is the optimized runtime for developing ML/DL applications in Databricks. It offers native integration with popular ML/DL frameworks, such as scikit-learn,...
Simple Steps to Distributed Deep Learning: On-Demand Webinar and FAQ Now Available!
On February 12th, we hosted a live webinar—Simple Steps to Distributed Deep Learning on Databricks—with Yifan Cao, Senior Product Manager, Machine Learning and Bago Amirbekian, Machine Learning Software engineer at Databricks. In this webinar, we covered some of the latest innovations brought into the Databricks Unified Analytics Platform for Machine Learning, specifically distributed deep learning....
Databricks Runtime 5.2 ML Features Multi-GPU Workflow, Pregel API, and Performant GraphFrames
We are excited to announce the release of Databricks Runtime 5.2 for Machine Learning. This release includes several new features and performance improvements to help developers easily use machine learning on the Databricks Unified Analytics Platform. Continuing our efforts to make developers’ lives easy to build deep learning applications, this release includes the following features...
