Actionable Insight for Engineers and Scientists at Big Data Scale with Databricks and MathWorks

Today, Databricks announced that it is launching a new partnership with MathWorks, whose MATLAB and Simulink product families are used by engineers and scientists globally.

From medical devices to jets and autonomous cars, millions of engineers and scientists use MATLAB and Simulink to build and test autonomous systems using simulation models that interact with physical systems. However, when it comes to running these models at cloud scale, domain experts have to wrestle with operational aspects of setting up compute resources, building complex data pipelines and learning new programming languages to run their models and simulations. This rigidity takes away from the actual simulation and research work that can yield critical discovery and insights at scale.

Simplifying MATLAB Simulations in the Cloud

The Databricks and MathWorks partnership solves this by allowing domain experts to access and analyze big data on the Databricks Unified Data Analytics Platform using the familiar MATLAB interface, greatly simplifying the aspects of running large computational and simulation workloads in the cloud.

The partnership enables core MATLAB users in Engineering Line of Businesses (LOBs) to access central data repositories managed by IT, get accurate results faster using MATLAB, and run algorithms on Databricks-enabled cloud clusters without needing expertise in Virtual Machines, Containers, etc. For Databricks users in IT and other LOBs, MathWorks enables new user communities to use cloud and enables data scientists and data engineers to work more efficiently with their R&D peers who are MATLAB users.

1. Deploy MATLAB Algorithms on Databricks for Large Scale Processing

The joint solution allows engineers and scientists to bring their MATLAB algorithms to Databricks for processing and downstream analytics to turn massive datasets into key insights. The Databricks Unified Analytics Platform provides the speed, scale and simplicity needed to reduce the complexity of cloud infrastructure while staying in a familiar MATLAB environment to deploy their algorithms for large scale processing. The solution allows them to set up fully managed Apache Spark clusters and schedule jobs interactively from the MATLAB command line or as part of their algorithm code.

2. Enable Enterprise-wide Collaboration on Data Science

By staying in the familiar MATLAB interface for Databricks, domain experts can focus on the business logic and leverage verified toolbox capabilities instead of starting a program from scratch. Using pre-built, industry specific MATLAB and Simulink toolboxes for deep learning, predictive maintenance, financial analysis etc., engineers can simply self-deploy their models and applications without having to recode.

3. Make Big Data Quickly Available with Delta Lake

Using Databricks’ DB Connect capability, experts can explore data inside Delta Lake. Delta Lake, an open source project that provides reliable data lakes at scale, allows access to both streaming and archived data from MATLAB built-in interfaces so engineers can run transactions on diverse data types. With features like ACID transactions and schema enforcement, Delta Lake provides the benefits of high-volume data access while preventing data corruption issues so your MATLAB algorithms and Simulink models can retain integrity at cloud scale.

Getting Started with MATLAB and Simulink on Databricks

To learn more about the Databricks and MathWorks partnership, check out the Big Data for Engineers: Processing and Analysis in 5 Easy Steps webinar.

In this webinar, Nauman Prasad, Director of ISV Solutions at Databricks and Arvind Hosagrahara, Chief Solutions Architect at MathWorks discuss how the solution is helping organizations process big data from MATLAB using an in-depth demo of the integration.