Matei Zaharia’s 5 predictions about big data and AI in 2018

Over the past few years, the demand for artificial intelligence (AI) and machine learning capabilities has surged with innovations in natural language processing, task automation, and predictions. From autonomous cars to a more personalized shopping experience, big data and artificial intelligence is at the forefront of new solutions that are delighting customers, improving business operations and enabling new products.

But AI is not as widespread as many would like. The reality is that there are only a handful of companies — technology giants such as Facebook, Google, and Uber — that have tapped into the promise of AI and actually accomplishing their goals with it. Most companies continue to struggle with adopting AI for several reasons, including that AI problems in many businesses are often harder than AI problems on the web, and lack of suitable technology platforms and expertise. As AI and big data technologies continue to evolve, however, the barrier to entry for many companies is gradually falling. If we look past the hype around AI, which trends and advances are likely to make a material difference in 2018?

Databricks’ Chief Technologist, Matei Zaharia, hosted a webinar on five key predictions for AI in 2018. Dr. Zaharia started the Apache Spark project, cofounded Databricks, and is also a faculty member at Stanford DAWN, a research lab on usable machine learning, so he will provide a broad view on AI’s impact in business. Matei will discuss five predictions:

1. Data will be the central competitive advantage. AI products can only be as good as the data we feed them, so we are going to see a massive influx in organizations curating datasets for a competitive edge. In addition, cloud platforms that enable users to pay by the dataset or query will create new business opportunities for companies to commercialize their proprietary datasets.
2. AI will find new use cases, starting with verticals. Vertical-specific solutions and libraries will start to incorporate the newest ML techniques and transform existing business processes, as we are already starting to see, for example, with Apache Spark based processing tools in genomics.
3. Data scientists will continue to grow in number. Data science and computer science are currently some of the most popular majors for college students. Although an influx of new graduates will not quench the demand for these roles, we will see more organizations starting data science teams and data products in areas with proven business value.
4. Deep learning frameworks will start to converge and move up in abstraction. Although there are currently a large number of machine learning frameworks, many of them offer similar functionality, and with efforts like ONNX, the basic tasks of defining and serving models will be handled well. Instead, developers will start to focus on making these frameworks easier to use for higher level business applications.
5. The cloud will enable new data application architectures. We are already starting to see that data management systems for the cloud can do much more than on-premise systems deployed by forklift onto virtual machines. We expect new offerings that take advantage of the scalability, elasticity, availability and ease of management of the cloud to simplify the pipeline around big data and AI products.

To dig deeper into Matei’s thoughts on these trends, watch the on-demand webinar (hosted by Data Science Central), as he discusses how centering organizations around high-quality data will be the main driver to AI, which new AI applications are seeing success in practice, and how new technologies including deep learning, data marketplaces and the cloud will affect the computing landscape.

Check out the Webinar

Download the infographic below and share these predictions with your peers!