Building Metrics Store with Incremental Processing

The traditional BI solution, bundling consuming applications with analytical logic running on a data warehouse, faces scalability and maintenance challenges. Complex metric calculation across multiple places leads to redundant reprocessing, missing review and testing, and inconsistent metrics for business teams. This session explores Instacart’s in-house solution-business metrics store with incremental processing. It decouples analytic logic from BI reporting and visualization, offering unparalleled efficiency, reliability, and cost-effectiveness for robust business analytics. Attendees will gain insights from a case study demonstrating this architecture's application in supporting large-scale ads experiment analysis. Key takeaways:

• Design of a metrics store facilitating reuse of metrics across BI and operational applications
• Practical insights into incremental processing with DBX structured streaming
• Future enhancement with Databricks Unity Catalog and Lakehouse Monitoring.