Developing personalized care plans to improve patient outcomes

Managing a 10x growth in data

Hinge Health uses custom ingestion and CDC to consume data from 70+ first- and third-party data sources via Fivetran. The company currently has 2,000 Postgres tables spread across 35+ databases. Over the last few years, the amount of data Hinge consumes has grown 10-fold, making it difficult to propagate schema and data at the same time. “When we started building our data platform, we had 20 to 30 terabytes of data. That grew to nearly 300 terabytes within a year,” Veera Mukkanagoudar, Senior Engineering Manager at Hinge Health, said. “As a data engineering leader, cost optimization is an ongoing journey. We have to build things where the costs grow linearly as the data volume and velocity grows. We could not grow our data cost by 10 times.”

The company needed to build an efficient, low-cost, low-latency data pipeline to ingest and transform data from heterogeneous sources at scale.

The journey to an optimized CDC architecture

Hinge Health data engineering leaders evaluated a number of solutions to enable their data engineering mission, including Snowflake, EMR, Redshift and Glue. Ultimately, the company chose Databricks Delta Live Tables. “We liked Databricks the best,” Veera said. “One feature stood out the most — the ability to support extract, load, transform (ELT), data warehousing, ML streaming, batch and generative AI (GenAI) workloads without moving data to meet the needs of the use case. We don’t have to copy the data between different systems to serve different applications. Name a problem in the data space and there is a Databricks tool available to solve it. We can serve everything on a single platform. That adds up to faster application development time.”

Once they selected Databricks, Hinge Health began working to spin up a new data pipeline using DLT to unify batch and streaming workloads to transform 300 terabytes of data. The primary objective was to isolate any large loads coming from large databases so they didn’t impact smaller databases. “We don’t have control over upstream databases,” Veera said. “There is a possibility that an upstream database or source could do a backward compatible change, which could cause a failure. We did not want it to impact other data pipelines.”

Veera and his team built an initial pipeline using Debezium Postgres reader to stream data to Kafka. They then used Amazon S3 to sync the data to an S3 bucket. They used two DLT pipelines: one to ingest the data from the S3 bucket to a staging table, which broadcast that data into a series of history tables, and the other to generate data for the latest version of the table. 

That architecture had a couple of challenges. First, using S3 meant they needed storage, query and additional compute to store and retrieve data from intermediary storage. This had implications for cluster utilization and cost. There were also reliability and latency challenges that made it difficult for the team to meet their SLAs.
Veera and his team realized they needed to move to Kafka as the data source rather than S3. As they did so, they also improved the staging table design which enabled better cluster utilization and allowed them to easily onboard data to pipelines that could accommodate the load rather than to pipelines that already had a taxed driver. The team now mirrors source databases in a lakehouse target by collecting and writing change logs from Aurora to serve data in Delta for GenAI and non-GenAI use cases using DLT. With topics as their new source, the team was able to simplify and consolidate. 

Veera said he especially likes DLT’s Python- and SQL-based parameterization features. “We have 35 data pipelines running, but they all run off a single code base,” he said. “We have parameterized all the table definitions, columns, data types and the database names — everything.”

Lowering costs while scaling

With a new CDC pipeline reading from Kafka instead of S3 and the ability to configure ingestion from multiple databases with a single data pipeline, Hinge Health has dramatically reduced their costs. Veera estimated: “We’ve reduced costs by at least 50% by migrating to DLT compared to when they started this journey, even though they now have 10 times more data.” Veera added, “Taking out S3 as a part of our data pipeline was very powerful.” Additionally, they’ve improved cluster utilization in smaller databases, increased reliability and reduced latency by 80%. “With the first iteration of the pipeline, it was challenging to meet SLAs and we were always running into issues with large data changes in upstream databases,” he explained. “Latency was nearly four times when we were using S3.”  

DLT significantly enhances Hinge Health’s data architecture by providing the benefits of materialized views, streamlining their CDC processes. Business insights from change data are served to the downstream teams as precomputed materialized views. DLT combines streaming tables with incrementally updated materialized views, avoiding the need to completely rebuild the view when new data arrives. This allows Hinge Health to reduce data processing costs and update reports and dashboards in real time. 

Ultimately, Hinge Health’s optimized CDC architecture adds up to happier users. “Data engineers are happy because they have less troubleshooting to do,” Veera said. “Our data science team is happy because we can meet our SLAs. Our engineering leadership is happy because they are always pushing us to optimize our costs. And our ML applications team is happy because they want to see data as fast as possible.”