How Relogix reduced infrastructure costs by 80% with Delta Lake and Azure Databricks

Get an early preview of O'Reilly's new ebook for the step-by-step guidance you need to start using Delta Lake.

Introduction

Relogix is a leading workspace analytics and sensor-as-a-service provider for workplace monitoring, management, and performance. Founded on a decade of Corporate Real Estate (CRE) data and analytics experience, Relogix specializes in blending data used by CRE professionals with data collected from workspace occupancy and comfort sensors. Blending data enables CRE strategy leaders to tell compelling stories about the culture of the organization which includes the workspace, the workplace, and the workforce. Relogix customers include the most recognizable brands in the world today. They have grown to over 30 corporate and public sector customers and deployed over 100,000 sensors covering over 20 million square feet of office space globally.

Relogix Workspace Occupancy Sensors provide real-time monitoring of workspace occupancy, utilization, densities and dwell times, while Workplace Comfort Sensors monitor ambient temperature, light, noise and humidity in the workplace. Relogix analytics and insights are designed to help CRE professionals understand how employees use the workplace, support organizations that are interested in achieving WELL certification, and prepare for the return to office in a post-COVID world.

Challenges with Cost in a Traditional Data Architecture

The maturity of data and technology has come a long way from descriptive data analysis where simple relational databases, historical data, and spreadsheet expertise delivered all the insight necessary to the business. The question of “what is my portfolio occupancy” or even “is the workspace comfortable?” are no longer sufficient to meet and deliver on competitive business opportunities. Especially relevant today, it was important for Relogix to move forward to answer questions for their customers like “Can I reduce my office footprint while maintaining or improving the employee experience?” or “How can I ensure my workplace comfort levels fall within acceptable WELL ranges?”. In order to answer these questions, the cost associated with development in their current architecture made it nearly impossible to achieve them. Traditional data architecture costs go beyond the high cost of storage and compute. The even larger opportunity lost is from not delivering on innovative and competitive business opportunities. Let’s break these data challenges out even further:

• Data Accessibility - Traditional data servicing architecture makes it too difficult and costly to store all the data at high volumes and reliably retrieve it. This ranges from a wide data variety, including semi-structured and unstructured data and at high velocity, including batch and real-time intervals.
• Inflexibility - Traditional data servicing and BI architecture usually only has limited support with certain proprietary tools and languages. This results in additional costs with introducing new tools and data silos with the lack of a central environment supporting all the necessary capabilities.
• Performance - Ingesting complex, high volume data can add costly processing times that may take hours before incoming data is available for queries. Even then, typically only a static, partial view is available. As a result, analysts and data science teams do not benefit from exploring the latest data.
• Reliability - Complex data pipelines are error-prone and complicated, requiring significant amounts of time and resources. In addition, data schemas evolve as business needs change. As new business opportunities arise every change to the schema may cause unforeseen data errors and failures in downstream dashboards, applications, and analysis.
• Complexity - Combining streaming and batch analytics requires complex and low-level code. Updates to batch and streaming data often interfere with queries or cause inconsistent results. This also results in needing more expensive, skilled resources to develop and maintain, neglecting new development opportunities. The data silos from this lambda architecture resulting from inflexibility make it difficult for resources to maximize each other's skill sets to speed up project development.

Given all of these challenges, Relogix’s data is driving demand for greater business insight, the foundation to deliver it, and the human resources to execute it.

How Azure Databricks and Delta Lake helped Relogix power Conexus

Azure Databricks is the jointly-developed data and AI service from Databricks and Microsoft that enables organizations to unlock value from their data by dramatically accelerating time to insight and by maximizing the productivity of their operations. Delta Lake is an open-source storage layer that brings reliability at scale to data lakes. Together Azure Databricks and Delta Lake are designed to pave the way for cost-effective, fast, and flexible analysis of complex data. This resulted in Relogix being able to reallocate IT resources to higher-value projects and reduce operational costs by up to 10x.

Introducing Azure Databricks

Higher value with the Relogix launch of Conexus

Relogix recently launched Conexus, a workspace intelligence platform powered by Azure Databricks and Delta Lake. Conexus enables organizations to access and leverage insights for the return to office in the post-COVID world and plan for the new workplace. Conexus surfaces insights for executives and decision makers, and provides the supporting details required by those who execute on those strategies, such as space planners and interior designers. Whether the objective is to optimize, rationalize or revolutionize the workplace, the information provided through the Relogix Conexus platform helps organizations transform the experience of work.

I.E: Occupancy rates, Actual Desks Required, Desk Surplus, and Suggested Sharing Ratio for optimizing safe, shared workspace portfolios.

Relogix customers can visualize insights and KPIs via data ‘storyboards’ and quickly get answers to questions, not possible before. The Conexus platform enables CRE professionals to:

• Automate the blending and integration of multiple data sources in any format ( i.e. API, xlsx, doc, txt, csv, IOT, PDF) at scale to improve reporting . Examples of additional data sources include, but are not limited to meeting room and desk reservation systems, IWMS, HRIS, security badging etc.
• Monitor and report on Key Performance Indicators (KPIs) that directly impact cost savings, cost reduction and/or cost avoidance
• Monitor COVID-19 workplace related health and safety protocols and identify potential areas that may be of concern such as high occupancy or high densities (clustering) and inform the availability of safe seats i.e. seats that have not been occupied for more than a predetermined number of days
• Access always up-to-date insights related to their workplace and workspace performance on-demand

With Conexus, CRE professionals are able to extract factual assessments related to the workspace, workplace, and workforce.This eliminates guesswork from reporting and ensures real estate related requirements are always aligned to business objectives and effectively support how people work.

Reduced TCO with a Modern Data Architecture

Prior to their Delta Lake migration, Relogix worked off a single SQL Server instance. Although SQL Server supported their core use case, scaling it to meet the requirements of a modern data company became too expensive. Relogix took advantage of Delta Lake and Azure Databricks to provide a secure and intuitive environment for analytics while limiting compute spend to what is required. Data preparation and ad hoc analysis were moved to Azure Databricks where data could be centrally accessed and clusters could be turned on and off, as needed. Additionally, Delta empowered Relogix to experiment with ML and AI use cases at scale. The implementation of the modern data architecture allowed Relogix to scale back costs on wasted compute resources by 80% while further empowering their data team.

Simple, modern data architecture with Azure Databricks and Delta Lake

Decreased Time to Market with Delta Lake and Databricks Connect

The medallion architecture (as noted in the following diagram) allows for flexible access and extendable data processing. The Bronze tables are for data ingestion and enable quick access (without the need for data modeling) to a single source of truth for incoming IoT and transactional events. As data flows to Silver tables, it becomes more refined and optimized for business intelligence and data science use cases through data transformations and feature engineering. The Bronze and Silver tables also act as Operational Data Store (ODS) style tables allowing for agile modifications and reproducibility of downstream tables. Deeper analysis is done on Gold tables where analysts are empowered to use their method of choice (PySpark, Koalas, SQL, BI, and Excel all enable business analytics at Relogix) to derive new insights and formulate queries. Productized through Databricks Connect, Relogix follows SaaS best practices such as Agile methodologies, DevOps standards, and functional programming principles to create highly resilient tables that their analysts, scientists, and customers can leverage with confidence. By connecting the desired business outcomes with implementation, Relogix takes tables from concept to production in a timely manner, ensuring quality each step of the way.

Architecting your Delta Lake

Increased Productivity with Delta Lake and Collaborative Notebooks

Azure Databricks has become the de-facto platform at Relogix for turning data into insights. From the moment data lands in the Bronze stage of their Delta Lake, Relogix connects the analytical with the technical, providing a secure sandbox-style environment for team members to collaborate in real-time. Subject matter experts propose and experiment with new ideas, analysts quickly answer customer requests, debug issues, or dream up new insights, and data scientists combine data sources to bring advanced analytics to life. Underlying the analysis are data engineers leveraging Delta Lake to securely support data access internally while rapidly serving insights to customers externally. By unifying the disciplines around a common platform, Relogix ensures business relevance and actionable outcomes are at the core of data initiatives.

• Real-Time Delta Pipeline with SQL and Python Collaboration 

A relevant case example is customers relying on real-time social media trends to gain a holistic view of their target market and anticipate their needs and desires. Reacting in an untimely manner or showing poor judgment can result in millions of lost revenue. The video tutorials below will show you real-time ingestion of tweets from Twitter, specifically for COVID-19, through the stages of Bronze to Gold with Databricks and Delta Lake.

Bronze tables will contain raw data.

Silver tables will contain filtered, consistent information after data cleansing

Gold tables will contain business-level aggregations used by our customers for data visualization, business analytics and critical business decision-making

What’s Coming up Next

Stay tuned for the next blog post on Databricks and Relogix giving better context to IoT data by leveraging alternative and big data sources.

Get Started

The Delta Lake quickstart provides an overview of the basics of working with Delta Lake. The quickstart shows how to build a pipeline that reads JSON data into a Delta table, modify the table, read the table, display table history and optimize the table. To see these Delta Lake features in action, see these introductory notebooks.

To learn more about Delta Lake, join our Modern Data Engineering virtual event or our next Azure Databricks Office Hours. To try Delta Lake, sign up for a free trial of Azure Databricks.