I Joined Databricks to Make Data Science a Little Less Scary

Big data and AI has always struck me as useful, but slightly scary.

For example, it’s useful when Waze uses big data to help me outsmart a traffic jam. On the other hand, big data’s ad-targeting is so powerful that millions of people are scared that their smartphones are eavesdropping on them.

Why Big Data and AI Is Intimidating to So Many People

The reason so many of us think big data is scary is because it’s opaque and powerful and we don’t have a way to creatively participate in it. In order to make big data and artificial intelligence truly useful and less opaque, data science tools need to be easier to use and accessible to more of us. Great design is one way to accomplish this.

Databricks is a technical company that is committed to democratizing the power of big data. While it has developed extremely powerful tools, the company is also a big believer in harnessing the power of design to make big data analysis and AI easier for people everywhere. Today, I am thrilled to announce that I’ve decided to join Databricks as VP of Product Design to help achieve this goal.

While technical requirements such as Python and statistical modeling aren’t going away anytime soon, good design can radically reduce the complexity of data science tools and support a more diverse range of technical ability. In fact, job market data suggests that easier-to-use data science tools are going to be the "next big thing."

How Good Design Can Make Data Science More Accessible

In August 2018, LinkedIn reported that there’s a shortage of 151,717 people with data science skills in the U.S., based on data from its platform. Better-designed and easier-to-use data science tools will be key to filling this gap.

According to a recent Gartner study, by 2022 it's predicted that 40% of machine learning model development and scoring will occur outside of tools where machine learning is the primary job-to-be-done. That means that other forms of less-technical business software will become smarter and enable more of us to participate in the predictive powers of machine learning.

For another example of how quickly the barrier-to-entry for machine learning and big data tools is dropping, look at biotech. In the very recent past, the time and cost to do machine learning on genetic data from hundreds of thousands of people was prohibitively expensive. It took months for biological researchers to run a single analysis, and they would often have to restrict their hypotheses to a basic set of questions that they could answer with small datasets using only a couple hundred people. Querying across hundreds of thousands of humans would have required a massive team of data scientists and data engineers, if it was even possible.

Today what was impossible has become the new normal. Performing ML on a massive dataset, like genetic data from a human population, has become dramatically easier and more affordable. Just the other day, a pharma company called Regeneron announced that it would be partnering with the US government to deliver a vaccine for the Coronavirus in a couple of months time. Previously, this would have taken several years.

Regeneron isn’t the only company using Databricks to design solutions to the world’s hardest problems.

A package of medical supplies delivered via a Zipline drone (photo: Zipline International)

Zipline is an on-demand drone delivery company that delivers medicine and blood to clinics in rural Africa. Zipline flies so many drone missions each year, their fleet can quickly generate large data sets. This data can be mined inside Databricks for important insights and business intelligence that help Zipline optimize its operations. This is another example of how big data can literally help save lives.

As a designer, I’ve always loved working with great entrepreneurs to solve big problems and I feel like the decision to join Databricks is the start of a meaningful and impactful journey as a problem-solver. Using the power of great design, Databricks is beginning a journey to make AI and big data easier to manage so that more people can work on solving the world’s hardest problems. And with the mountain of problems facing humanity, it would seem we have little time to waste.

If you’re serious about the potential of design to play a huge role in making AI and big data tools less scary and more accessible, I hope you will join me.

Ryan is the VP of Product Design at Databricks. Find Ryan’s original article on Medium: I joined Databricks to Make Data Science a Little Less Scary.