Customer Case Study: Healthdirect Australia – Databricks

Healthdirect Australia

Customer Case Study

Healthdirect Australia

Health Direct Australia is a government-owned, not-for-profit organization tasked with helping Australians manage their health through access to information and advice. Established in 2006, Health Direct Australia has expanded its range of services to include AI-supported tele-medicine across multiple platforms, including social media and voice-activated applications.

Vertical Use Case

  • Predictive analytics to improve the quality of care
  • Digital and Tele-health services to better address patient needs

Technical Use Case

  • Batch and streaming ingestion
  • ETL
  • Regulatory compliance
  • Machine learning / NLP

The Challenges

  • Slow Data Pipeline: A legacy system of multi-node clusters couldn’t process data efficiently, resulting in batch over-runs and up to 24-hour for queries to run.
  • High Volumes of Disparate Data: Healthcare and patient data was siloed in different locations with multiple schemas, making it difficult to aggregate for analytics.
  • Poor Cross-Team Collaboration: Disparate data and teams working in different programming languages wasn’t conducive to streamlined workflows.
  • Poor Data Reliability: The high variety of constantly changing data sources made it difficult to ensure data remained accurate.

The Solution

Databricks provides Health Direct Australia with a Unified Analytics Platform that allows them to process and analyze patient data and develop new machine learning and voice recognition-based products, all while keeping patient data secure and in compliance with government regulations.

  • Fully Managed Platform: A fully managed cloud platform simplifies operations and delivers superior performance.
  • Delta Lake: Enabled read consistency and high data integrity which was not available earlier. Also, allowed analysts to securely experiment with reliable data and act on insights faster in accordance to government regulatory standards.
  • Interactive Workspace Data scientists can collaborate, share, and track data and insights, fostering an environment of transparency and improving patient outcomes.
  • Realized Continuous Data Assurance: Able to ensure data security with access control, data spillage and leakage monitoring.

The Results

  • Unified Approach: Data engineering and data science teams are now solving problems together.
  • 1440X Faster ETL: Processing a million records a minute compared to 24 hours for the same volume before deploying Databricks.
  • Improved accuracy of fuzzy name match algorithm: From less than 80% with manual verification to 95% and no manual intervention.
  • Designed for Scale: Infrastructure is now capable of handling an anticipated exponential growth of patient health records.

Databricks Unified Analytics Platform delivered the time to market as well as the analytics and operational uplift that we needed in order to be able to meet the new demands of the healthcare sector.

Peter James, Chief Architect, Health Direct Australia