This was written in collaboration with Andrew Mullins, Director of Data Science at Kin + Carta.
With the rise of new technologies from telematics to autonomous vehicles, data and AI are steering the wheel of innovation in the automotive industry, as manufacturers are rapidly shifting gears to embrace transformative technologies to navigate the road ahead with precision and efficiency. Leading the charge is one of the largest American automakers with 165,000 employees focused on pushing the envelope of automotive engineering, design, and technology.
Like any resourceful enterprise company looking to fully leverage its data, the U.S. automotive giant knew that it had to turn its raw data — which it had a lot of — into valuable business insights. Not only could the brand have better control over its data by pairing a modern cloud platform with AI-powered data software, but it could also drive innovation where it saw fit with its own offerings.
The automotive giant could no longer deny that thousands of its data scientists, analysts, engineers, and other technical employees needed centralized storage for its petabytes of data to facilitate more fruitful analysis. With a unified interface, the company could gain greater control over its data, build a solid foundation for its data practices, and streamline data workflows for all of its employees.
Although the company's C-suite was committed to fully adopting Azure and cloud infrastructure, organizational and technical issues slowed its migration progress. However, the maturation of cloud platforms and emerging tech evolutions, such as Large Language Models (LLMs), reignited the priority to accelerate the company's modernization efforts.
The successful automotive brand was already well aware of the benefits of the switch. By fully transitioning from on-premises storage to cloud storage, it could lessen the strain on IT by facilitating self-service and rapid prototyping. Plus, a cloud-based architecture would help the company better ingest, process, and store the vast amount of data inherent in its industry to improve vehicle performance, innovate new features and products, foster customer relationships, reduce operational costs, and oversee its relationships with its suppliers.
Yet, the switch from on-premises to cloud wouldn't happen overnight — or without help. The company was aware that it would need to build a successful blueprint to accelerate migration and connect its existing Azure Data Lake, which wasn't being properly utilized, to a data lakehouse platform.
Additionally, the automotive business remained open-minded to bringing in an implementation partner to spearhead the nuanced aspects of onboarding new software. With all of this top of mind, the automotive company decided to buckle down and finally prioritize the project, approaching it formulaically and with a brick-by-brick mentality (no pun intended).
While the automotive company already had an existing Azure Data Lake, it wasn't utilizing the full potential of its data to evolve its business. To remedy this, the first step was for them to partner with Kin + Carta, a digital transformation consultancy focused on driving positive business outcomes through the power of data.
Despite the migration process getting off to a slow start, the behemoth automotive brand already knew that it wanted to connect the Databricks Lakehouse Platform to its existing Azure Data Lake. This way, all resources in the Databricks data plane, where data is processed, could be hosted in its Azure cloud account and connect to data sources from the Databricks File System (DBFS) and of course, its current Azure Data Lake.
The company would invest in other components of Databrick's Lakehouse Platform, including Unity Catalog, a unified governance solution for data and AI assets in the lakehouse. It also wanted to utilize Databricks to manage its ML lifecycle — from model training and experimentation to deployment to production. To wrap a bow on its selection of tools, the big automotive brand would use Azure DevOps to more astutely manage all of the intricacies of this project in the Azure Cloud.
"Kin + Carta partnered with our client to migrate to a unified data platform leveraging Databricks Lakehouse. By building and documenting tooling and reusable patterns, we unlocked modern end-to-end, governed workflows to enable their data practitioners."— Andrew Mullins, Director of Data Science at Kin + Carta.
On the agency side, Kin + Carta would help the brand discern reusable patterns, processes, and tooling to implement common workflows for data pipeline orchestration. Ultimately, this would help uncover a targeted inventory of core data assets to enable projects and teams that were "cloud ready" at the project's initial inception but hindered by platform immaturity.
Next, Kin + Carta planned to work with the automotive manufacturer to create a socialization plan to educate data practitioners on the platform's features and best practices to prepare the team for onboarding. This was a crucial step, as it would enable hundreds of data practitioners, from analysts and scientists to engineers, for daily Azure and Databricks usage.
By connecting to the Lakehouse infrastructure, the automotive business would help technical users unify the expansive amount of business data at its disposal, without worrying about batch processing or requiring additional data transformation and/or integration steps. Now, the company's technical teams could take advantage of real-time processing, allowing for immediate analytics and making it easier to discuss, collaborate, and ideate around the executable data.
The good news was that they wouldn't have to concern themselves with the complexities of data ingestion and management anymore. The Databricks Lakehouse Platform would automate the "extract, transform, and load" (ETL) process to transform the company's Azure Data Lake into a prime destination for its structured, semi-structured, and unstructured data, turning business insights into actionable steps.
Now that the automotive company made the strategic shift from on-premises to the cloud, it has fully positioned itself to modernize its business — all it took was a commitment to emerging technologies and cloud innovation. At the core of this project, building a solid data foundation for increased efficiency was the goal, and the automotive brand is now capitalizing on the impacts of its commitment.
From an operational perspective, the migration has been beneficial in numerous ways. First and foremost, the automotive brand achieved the ability to scale in a cost-effective and timely manner — one of its main objectives when reinvigorating this project. It also centralized key performance indicator (KPI) definitions (e.g., data completeness, accuracy, validity, and freshness) using Azure DataOps, now having a single platform to monitor and manage these KPIs. Finally, and perhaps, most importantly, it has allowed for continual use case activation, as the development of common, reusable patterns becomes clear.
What made this project truly worth it to the automotive company, however, was the confidence its technical users would gain in the integrity of its organizational data, feeling empowered to make informed decisions that most accurately support ever-shifting business goals. It all started with Azure, which provided a secure cloud foundation — essentially, a launching pad for a host of various impactful Databricks products.
But how has the enterprise automotive brand specifically improved employee efficiency? Not only have the technical teams improved their data syndication capabilities, but they have also automated monitoring data quality, cutting down on time and resources devoted to such tasks. Better yet, the business now has well-documented and discoverable data assets, so technical team members no longer have to waste additional time searching for the resources they need to excel in their roles.
For IT, the data project has reduced development and delivery time, with less time spent procuring required data. Not to mention, traceable data lineage enables them to better identify issues and mitigate compliance risk when necessary. Last, but not least, the new, stringent data security processes and requirements limit the exposure of sensitive data (e.g., customer data).
As for what the future holds, the automotive company's technical team can focus its valuable time on strategic tasks that move the needle forward on this business' key objectives. Kin + Carta continues to partner with the client as the brand explores new use cases and further innovation of its data and cloud platforms.
Interested in hearing more about how Databricks' Data Intelligence Platform can help your business? Learn more today.
