Driving freight transportation into the future

The impossibility of progress without unified data

In order to fulfill its mission of creating the most efficient transportation network in North America, J.B. Hunt Transport, Inc. offers asset- and non-asset-based transportation solutions, including dynamic freight matching. This involves connecting a business’s specific shipping needs with available carrier capacity, taking into consideration details such as price, weight and location. The problem? The carrier world is deeply fragmented, with an estimated 3.5 million drivers across the industry. Improving upon a matching system that considers that much fragmentation requires real-time data and meaningful analytics.

In order to achieve this goal, J.B. Hunt had to be able to unlock the value of its data that was stored in legacy enterprise data warehouse (EDW) platforms, which limited its usability for real-time decision making. Its systems struggled to process and store the massive data generated from location pings every 15 minutes from hundreds-of-thousands of loads being moved. It also required high levels of data security to ensure the right users had access to sensitive data. Finally, it needed the ability to support data streams generated by IoT sensors on trucks and containers that are typically not owned by the company. This made telemetry-based use cases leveraging machine learning (ML) and AI nearly impossible.

Building an open, scalable, and unified lakehouse architecture

J.B. Hunt chose to work with Google Cloud and Databricks Data Intelligence Platform to create a unified BI and AI platform that could capture all forms of data and support real-time analytics for data engineers, scientists and others across the business. With Databricks, J.B. Hunt successfully created an open, interoperable and rapid data lakehouse for J.B. Hunt 360°®, that enables the company to offer customers unmatched transportation services to maintain its leading position in North America.

With Delta Lake as a foundation, J.B. Hunt not only has the ability to put all of its data in one place for easy access across the organization, but also to ensure the performance and reliability of streaming data pipelines at any scale. Delta Lake as the open storage layer brought efficiency and portability to its teams as it moved terabytes of its existing data onto the platform. By streaming real-time to Delta Lake, J.B. Hunt can analyze larger, complete datasets to run analytics and ML faster than ever. With MLflow, the data science team is now able to establish reproducibility of code and experiments to ensure it’s reusable by data scientists. “It was critical that we built upon a platform that provides the flexibility to quickly deploy use cases regardless of which cloud or toolsets are being leveraged across our diverse operations” said Joe Spinelle, Director, Engineering & Technology at J.B. Hunt.

Databricks Data Intelligence Platform is also being used in conjunction with Immuta, an automated data governance platform. From a security standpoint, Immuta has added a level of data security that extended the security capabilities of its legacy EDW. It is now able to automate the data governance process to ensure the appropriate users have access to the data needed to make important business decisions. This is achieved through columnar-level data masking, which provides fine-grained security beyond traditional role-based access controls, the ability to establish global and local security policies, and full auditing on who has access to what data and how the data is being used.

“By combining the two solutions, we have the level of flexible data security needed to enable users to have access to reporting without the worry of sensitive data being accessed,” explained Tina Headrick, Senior Manager of Engineering and Technology at J.B. Hunt.

A single source of truth that delivers operational efficiency

In terms of collaboration, Databricks has succeeded in bringing J.B. Hunt’s data teams together to accelerate data science productivity. “With Databricks, everything is in the same repository, the same notebook structure, the same language and the same version, which is key,” explained Douglas Mettenburg, VP of Engineering & Technology.

The success of Databricks at J.B. Hunt is reflected in its phenomenal performance gains, including the ability to train thousands of ML models in less than four hours that can nowdeliver freight recommendations to carriers 99.8% faster than before. “Ultimately, Databricks is now a source of transparency for J.B. Hunt,” added Doug. “It’s showing the real value that data and technology can deliver for the entire company.”