Big data projects are hungry for simpler and more powerful tools: Survey validates Apache Spark is gaining developer traction!

In partnership with Typesafe, we are excited to see the publication of the survey report representing the largest poll of Apache Spark developers to date. Spark is currently the most active open source project in big data and has been rapidly gaining traction over the past few years. This survey of over 2100 respondents further validates the wide variety of use cases and environments where it is being deployed.

The survey results indicate that 13% are already using Spark in production environments with 20% of the respondents with plans to deploy Spark in production environments in 2015, and 31% are currently in the process of evaluating it. In total, the survey covers over 500 enterprises that are using or planning to use Spark in production environments ranging from on-premise Hadoop clusters to public clouds, with data sources including key-value stores, relational databases, streaming data and file systems. Applications range from batch workloads to SQL queries, stream processing and machine learning, highlighting Spark’s unique capability as a simple, unified platform for data processing.

At Databricks and within the Spark community, this type of feedback helps us enhance Spark for more use cases making big data simpler for enterprises of all sizes. As the creators of and most active contributors to Spark, we are happy to see widespread adoption of Spark across a wide range of industries.

In 2015, we are working on a number of Spark initiatives, including integrated SQL and data frame capabilities, improved stream processing capabilities, extending Spark's machine learning library (MLlib), and richer APIs in Python and R. We have also taken Spark to the cloud in our hosted service, Databricks Cloud, that enables users to leverage Apache Spark directly in the cloud giving them a powerful platform to manage clusters, interactively explore and visualize data, and build production data pipelines.

We are excited to partner with Typesafe in this effort and in our shared vision to bring a comprehensive suite of application development tools for developers that let enterprises operate with more agility and speed. The feedback from this survey will be extremely valuable for both companies to collectively enhance the Spark developer experience.