Building a Scalable, Real-Time Concurrency Prediction Service

Dream11's rapid growth has posed critical challenges in scaling infrastructure to handle millions of concurrent users during high-traffic events. Concurrency Prediction Service provides real-time forecasts of peak user activity in 30-minute intervals to optimize resource allocation by the Scaler Service.

This presentation covers the critical aspects of building and optimizing the Concurrency Prediction Service, including:

• Real-time data ingestion and processing to handle spiky data patterns and high-variability traffic
• Anomaly detection mechanisms to identify and adjust for deviations caused by notifications or external events
• Modular and composable architecture for better scalability and maintainability
• Incremental processing with Spark Structured Streaming for real-time insights
• Granular resource tuning to optimize performance and control costs
• Leveraging Databricks for streamlined workflows, enhanced collaboration and efficient pipeline management