Learning how to write Apache Spark applications in Databricks with the Integrated Search feature

We built Databricks on top of the Apache Spark framework to make data analysis simple. In the same spirit, we want to make the adoption and usage of Databricks simple. Developers and data scientists need to answer questions about Databricks and Spark quickly in order to get their work done, while project leads need to minimize the time their teams spend on ramping up on a new platform. In this blog post, we present a new feature in Databricks called Integrated Search that helps our users find relevant information quickly and easily.

What is Integrated Search?

The integrated search feature helps users find relevant information about Databricks and Apache Spark from three powerful resources:

• Databricks Guide: An extensive guide with tutorials, sample applications, reference code, and documentation built into Databricks
• Databricks Forum: An online forum where customers can ask questions and find answers provided by our expert field engineers and developers who work on Spark, as well as other customers
• Databricks Workspace: The workspace contains code previously written by you and other users of your Databricks instance, a valuable resource for finding code examples where your team may have solved a similar problem before

For questions about how to use Databricks features and Apache Spark, Integrated Search provides a simple interface that allows users to quickly find answers without leaving Databricks and interrupting their workflow. In the example below, one can easily locate example applications of processing Parquet files in Databricks Guide, or find more in-depth discussions about how to use Parquet in the Databricks forum:

There are also times when users want to find code examples written by others in their team, or they may simply want to locate some code they wrote in the past. For this scenario, Integrated Search can search through code and results in Notebooks, which are interactive documents users create in the Databricks Workspace. In the example below, users can find examples of code where their team used Spark broadcast variables, so they can quickly determine how to do so in their own code:

Benefits for Databricks users

Rather than wasting time crawling irrelevant or out-of-date help sources, and switching between multiple websites to search code, product documentation, and forums, Integrated Search provides a one-stop-shop for getting answers about the nuts-and-bolts of creating Spark applications using Databricks. We hope this feature will help our customers master Apache Spark more quickly, so they can spend their valuable time developing exciting applications in Databricks.

To learn more about Databricks, check out our products page or the overview video

If you’re interested in trying out Databricks, sign up for a trial!