A Spark-Based Intelligent Assistant: Making Data Exploration in Natural Language Real - Databricks

A Spark-Based Intelligent Assistant: Making Data Exploration in Natural Language Real

Rather than running pre-defined queries embedded in dashboards, business users and data scientists want to explore data in more intuitive ways. Natural language interfaces for data exploration have gained considerable traction in industry. Their success is triggered by advancements in machine learning and by novel big data technologies that enable processing large amounts of data in real-time. However, even though these systems show significant progress, they have not yet reached the maturity level to support real users in data exploration scenarios either due to the lack of supported functionality or the narrow application scope, remaining one of the “holy grails” of the data analytics community. In this talk, we will present a Spark-based architecture of an intelligent data assistant, a system that combines real-time data processing and analytics over large amounts of data with user interaction in natural language, and we will argue why Spark is the right platform for next-gen intelligent data assistants. Our intelligent data assistant (a) enables a more natural interaction with the user through natural language; (b) offers active guidance through explanations and suggestions; (c) constantly learns and improves its performance. To build an intelligent data assistant, there are several challenges. Unlike search engines, users tend to express sophisticated query logics and expect perfect results. The inherent complexity of natural languages complicates things in several ways. The intricacies of the data domain require that the system constantly expands its domain knowledge and its ability to interpret new data and user queries by constantly analyzing data and queries. Our intelligent data assistant brings together several components, including natural language processing for understanding user queries and generating answers in natural language, automatic knowledge base construction techniques for learning about data sources and how to find the information requested, as well as deep learning methods for query disambiguation and domain understanding.



« back
About Georgia Koutrika

Dr. Georgia Koutrika is Research Director at Athena Research Center. She has more than 12 years of experience in multiple roles at HP Labs, IBM Almaden, and Stanford, building innovative solutions for recommendation systems, data analytics and exploration, information extraction and integration. Her work has been incorporated in commercial products, described in 8 granted patents and 18 patent applications in the US and worldwide, and published in more than 80 research papers in top-tier conferences and journals. She serves in various roles in program committees of top-tier conferences. She is an IEEE senior member, ACM member, and ACM Distinguished Speaker.