MIT Tech Review Study: Building a High-performance Data and AI Organization -- The Data Architecture Matters.

Only 13% of organizations -- super achievers -- are succeeding at their data and AI strategy yet the successful application of data and AI has never been a greater necessity for survival than now¹. In order to remain forward-thinking in today’s landscape, data leaders are looking for the ability to eliminate silos that traditionally separate analytics, data science, and machine learning through lakehouse platforms -- unifying their data, analytics, and AI under a simple, open, and collaborative data architecture. The old adage of “this is how things have always been” is a recipe for failure and the successful use of data and AI by a group of innovative organizations is transforming every industry by force. This blog will provide insight into what the super achievers attribute their success to, as well as what data and technology leaders cite as a critical enabler to building data cultures, their challenges with ML, priority investment areas over the next two years, and what they would focus on if given a redo button.

Growing importance and struggles with data and AI

Research confirms the obsession with data + AI is extending beyond the practitioner and into the board room. Leaders are also shifting their mindset to no longer just think about what data they have, but rather how that data is being used to fuel innovation and growth. In fact, in the 2021 Big Data and AI Executive Survey, NewVantage Partners found 92% of executives report that the pace of Big Data/AI investment in their organization is accelerating — up 40% from the previous year², and McKinsey & Co. estimates that analytics and AI will create over $15 trillion in new business value by 2030³. Yet despite this growing priority, very few organizations actually successfully implement their strategy - only 13%¹. One angle we rarely examine is what the so-called "super-achievers" -- are doing to drive their success?

Spoiler alert, the data architecture matters a lot more than you would think

Based on interviews with 9 data and analytics leaders from brands like McDonald’s, CVS Health, L’Oreal, and Northwestern Mutual, in addition to a survey of 350 CIOs, CDOs, CTOs, and other leaders, MIT Tech Review, in collaboration with Databricks, found in its latest report, "Building a high-performance data and AI organization," the challenge starts with the data architecture. Organizations need to build four different stacks to handle all of their data workloads: business analytics, data engineering, streaming, and ML. All four of these stacks require very different technologies and, unfortunately, they sometimes don’t work well together. The technology ecosystem across data warehouses and data lakes further complicates the architecture. It ends up being expensive and resource-intensive to manage. That complexity impacts data teams. Data and organizational silos can accidentally slow communication, hinder innovation and create different goals amongst the teams. The result is multiple copies of data, no consistent security/ governance model, closed systems, and less productive data teams.

Meanwhile, ML remains an elusive goal. With the emergence of lakehouse architecture, organizations are no longer bound by the confines and complexity of legacy architectures. By combining the performance, reliability, and governance of data warehouses with the scalability, low cost, and workload flexibility of the data lake, lakehouse architecture provides flexible, high-performance analytics, data science, and ML.

At Databricks we bring the lakehouse architecture to life through the Databricks Lakehouse Platform which excels in three ways:

1. It’s simple: Data only needs to exist once to support all workloads on one common platform.
2. It’s open: Based on open source and open standards, it’s easy to work with existing tools and avoid proprietary formats.
3. It’s collaborative: Data engineers, analysts, and data scientists can work together and more efficiently.

The cost savings, efficiencies, and productivity gains offered by the Databricks Lakehouse Platform are already making a bottom-line impact on enterprises in every industry and geography. Freed from overly complex architecture, Databricks provides one common cloud-based data foundation for all data and workloads across all major cloud providers. Data and analytics leaders can foster a data-driven culture that focuses on adding value by relieving the daily grind of planning and all its complexities, with predictive maintenance.

Additional findings from the study

In addition to an effective and efficient data architecture being the prime reason for success, the study also found:

• Open standards are the top requirements of future data architecture strategies. If respondents could build a new data architecture for their business, the most critical advantage over the existing architecture would be a greater embrace of open source standards and open data formats.
• Technology-enabled collaboration is creating a working data culture. The CDOs interviewed for the study ascribe great importance to democratizing analytics and ML capabilities. Pushing these to the edge with advanced data technologies will help end-users to make more informed business decisions — the hallmarks of a strong data culture.
• ML’s business impact is limited by difficulties managing its end-to-end lifecycle. Scaling ML use cases is exceedingly complex for many organizations. According to 55% of respondents, the most significant challenge is the lack of a central place to store and discover ML models.
• Enterprises seek cloud-native platforms that support data management, analytics, and machine learning. Organizations’ top data priorities over the next two years fall into three areas, all supported by broader adoption of cloud platforms: improving data management, enhancing data analytics and ML, and expanding the use of all types of enterprise data, including streaming and unstructured data.

From video streaming analytics to customer lifetime value, and from disease prevention to finding life on Mars, data is part of the solution, to succeed with data and AI, organizations need better tooling to handle the data management fundamentals across the enterprise. Download your copy of the report to dive into the analysis and better understand the interviewees’ viewpoints.

Attribution

¹MIT Tech Review - Building a high-performance data and AI organization
²NewVantage Partners  - Big Data and AI Executive Survey
³McKinsey & Company -  The executive’s AI playbook