Five Challenges CIOs Need to Overcome in the New Year

As IT leaders kick off the new year during one of the most tumultuous times in recent history, CIOs are being forced to revisit their priorities to adapt and emerge through 2023 stronger than ever.

The past few years have been extraordinarily disruptive, and more so for CIOs who have had to accelerate their digital transformation journey to keep pace with changing market trends, while also dealing with cost-cutting measures in an economic downturn, staffing challenges and remote work, and an ever-increasing list of business requirements for new digital services.

With these changes, CIOs must rethink their agenda and reprioritize their goals and investments to stay ahead of evolving business conditions. Here are the top five considerations CIOs should focus on as they prepare for 2023:

Modernization of Data Platforms

For many organizations, the pandemic accelerated cloud adoption - with many spanning two or more cloud providers - but there are still many workloads still residing in on-prem data lakes like Hadoop, as well as Cloud and Enterprise Data Warehouse stores such as Oracle, Netezza, Teradata, and Greenplum that remain a burden on costs, resources and the ability to be agile and innovate.

While the licensing costs of maintaining these legacy systems often reach millions of dollars annually for larger enterprises, licensing alone is typically only part of the equation. Hardware maintenance and utilization can account for as much as 20% of the total cost, as well as the DevOps burden that comes with supporting these systems.

Modernizing these technologies will be vital in 2023. And we already see it happening at some of the largest enterprises. Take note of Nationwide, who improved the productivity of their data engineers and data scientists by as much as 25%, accelerated their data pipelines 9x faster by leveraging modern cloud-based analytics platforms, and migrated to new technologies like Databricks Lakehouse.

Tech Consolidation

It's no secret that the amount of data - both structured and unstructured - continues to explode by unimaginable amounts with the rise of cloud, mobile, social, AI and IoT data. There is an incredible amount of untapped potential for new data monetization and developing new products; as a result, there has been an explosion of systems to manage all of it. Just look at famed Matt Turck, VC at FirstMark, and his 2021 Machine Learning, AI and Data Landscape. As he mentions, the deluge of VC financings, unicorn creation and IPOs have resulted in a dizzying array of new data and AI technologies (see for yourself here).

One of the biggest priorities for CIOs will be to take back control of their data estate and consolidate these systems. Having your entire data team manage all of these bespoke tool chains just isn't sustainable and has resulted in tech bloat, inefficiencies, and missed opportunities. CIOs have more pressure than ever to remain competitive, with the demands to deliver all these technologies at lower costs. Consolidating and optimizing their tech stacks will be key.

The emergence of the lakehouse paradigm is one strategy that more and more companies are utilizing to consolidate their BI and AI infrastructure. In fact, in the Gartner Hype Cycle for Data Management, 2022, Gartner foresees that lakehouse adoption will reach critical mass in the next 2-5 years. Underpinning the lakehouse technology is the open-source technology Delta Lake, a data, analytics and AI workbench powered by the Linux foundation. There is a great podcast on Matt Turck's page featuring Databrick's CEO and founder, Ali Ghodsi, where they dive into the history of lakehouse and its advantages.

Cost Containment

There are a lot of headwinds as we go into the new year, so cost containment will certainly be at the forefront of every CIO. According to Gartner Senior Director Analyst, James Anderson, "It is the role of the CIO to show to stakeholders how certain cuts will impact services and where savings are possible". ¹ IT budgets are increasing for most organizations, but inflation and rising costs will eat into new areas of investment, so there's increasing pressure to reduce costs in existing services. Sunsetting legacy data platforms, consolidating vendors and optimizing cloud spending are just a few of the ways CIOs can save resources and fund new and more impactful innovation projects.

Cloud migration projects and tech consolidation will certainly help with cost containment, but they won't go far enough. IT leaders need to also take a close look at their existing infrastructure and performance, particularly when it comes to rising cloud data warehouse costs. Costs can quickly spiral out of control, leading to sticker shock with more and more data and users running queries against the warehouse.

One way to ensure you are getting the best bang for your buck from your warehouse is to establish the following:

1. Leverage a data warehouse using open formats that don't change as quickly as proprietary data formats
2. Utilize an architecture that takes advantage of massively parallel processing
3. Investigate and adopt technologies that improve both latency and throughput

This is exactly what Databricks Lakehouse and Databricks SQL have done. Impressively, The Barcelona Supercomputing Center, which provides the gold standard performance benchmarks for data warehousing, found that Databricks was 2.7x faster with a 12x better price performance compared to Snowflake. These improvements in performance have a huge impact on your cloud spending as data size increases in production while playing a crucial role in cost containment.

Case in point, companies like Regeneron, who are accelerating drug target identification, can now get faster insights and have reduced the time it takes to run queries on their entire dataset from 30 minutes, down to 3 seconds - a 600x improvement. Jeffrey Reid, Ph.D., Head of Genome Informatics at Regeneron said, "The Databricks platform is enabling everyone in our integrated drug development process – from physician-scientists to computational biologists – to easily access, analyze, and extract insights from all of our data."

Unified Data Management Architecture

In 2023, CIOs will need to develop a unified data management architecture – a need driven by the increasing number of data cloud platforms, as outlined in Matt Turck's research. Orchestrating all of this data is crucial, and recent research shows that 68% of CIOs say unifying their data platform for analytics and AI is crucial.

Without a unified data architecture, organizations often build multiple different stacks to handle all of their data workloads, including data warehousing, data lakes, ETL/data engineering, data streaming, business intelligence (BI) and artificial intelligence (AI). And each of these stacks requires very different technologies that may not work well together, if at all. Those organizations are painfully aware of the high costs associated with maintaining complex system architectures and dealing with vendor lock-in and proprietary solutions that are slow to evolve.

Maintaining such data silos over time with their associated infrastructure and plumbing is now seen as unnecessary overhead that doesn't provide any additional value or competitive advantage. Adopting a single technology vision and architecture in today's world is a must.

In a recent data management podcast, Habsah Nordin, Head of Enterprise Data in Group Digital for PETRONAS Global, agrees. "That's the reason why we identified Databricks as one of the technology stacks we are going to build within the EDH (enterprise data hub). That allows us to actually bring in data and connect to that across different data sets, across all the applications and data platforms."

Risk Management

Risk, governance, compliance and security have long been fundamental data challenges as data leaders strive to build trust in their data models both internally and externally. Bad-quality data leads to inaccurate analytics, poor decision-making, cost overhead, and negative business impact. Gartner estimates that poor data quality costs organizations an average of $12.9 million every year. An effective data governance strategy must include a focus on data quality so that the provenance of data can be known, rules can be enforced on the data, and changes can be tracked to build trust in the results.

Data governance encapsulates the policies and practices to securely manage the data assets within an organization. Leveraging a unified approach to managing data and analytics helps data leaders address the most common challenges when modernizing their risk management practices. They can adopt a more agile approach when their data can serve multiple risk management use cases and they are no longer restricted to the narrow view of their individual use case. Solid data governance can also help optimize costs by preventing users from needlessly starting up large data clusters, and by creating guardrails for using expensive graphics processing unit instances.

At Rolls-Royce, which builds aircraft engines, governance is both external and internal. "The governance that we go through at Rolls-Royce is probably much higher than I could've dreamt of when I was in one of those small agile startups," said Stuart Hughes, Chief Information and Digital Officer for Rolls-Royce.

"We created a data passport process to understand the lineage of all of the data, where it came from, what it can be used for and where we are going to put it. So any data that comes into Rolls-Royce from an engine, from a customer, or some data that we generate from those data sets, goes through a very consistent process where it's reviewed."

Data-driven IT leaders need to focus on the timeliness, completeness and accuracy of their data. Governance capabilities, including auditing, retention and lineage, have become essential, particularly considering recent privacy regulations.

In Summary

CIOs must be attuned to the latest trends in digital transformation, cloud computing, governance, and data management and analytics in order to effectively lead their organization's IT strategy. By staying up to date with these initiatives and prioritizing them in the new year, CIOs can help their organizations stay ahead of the curve and remain competitive in an increasingly disruptive world.

Learn more about how technology executives can transform and scale your organization with data, analytics and ai here.

¹ How CIOs Can Optimize IT Costs - Gartner