Building an Agile Development Environment for Healthcare Analytics Pipelines in Spark

Collective Health provides an integrated solution that allows self-funded employers to administer plans, control costs, and take care of their people. Dealing with private and financially sensitive healthcare data while considering security and HIPAA compliance requires us to develop data pipelines using robust and well tested infrastructure. The rapidly changing nature of our data and requirements means we need to be ready to update our pipelines with minimal lead time. In this presentation we will discuss how the Collective Health Data Engineering team has addressed data complexity, changing requirements, and compliance in a highly regulated industry by moving our pipelines to a Spark-based infrastructure.

« back