Graduating higher education from traditional to cutting-edge

The challenges of data management without a data and AI platform

The North Dakota University System (NDUS) has five community colleges, four regional universities and two research universities with approximately 80,000 students, faculty and staff. For NDUS to compete in the fierce higher education market, each institution must work together; however, its data teams lacked the infrastructure to collaborate and scale its use cases. Lacking a modernized data and AI infrastructure, NDUS spent time on manual tasks rather than capitalizing on innovation for speed, ease and cost-cutting across all departments.

Ryan Jockers, Assistant Director of Reporting and Analytics for NDUS, described the impact of an overwhelming amount of policy documents, state law, contracts, procedures and code that must regularly be referenced across the system: “Finding what you need among all those texts can take hours, and users constantly need to start fresh searches to know what we can and can’t do, what the actual policy says or what’s the policy on XYZ. We need to reference regulations for just about everything, but without any enablement, users were slow to surface what they needed.” Relying solely on institutional knowledge and people, NDUS was limited in its ability to share these one-of-a-kind resources and ensure their knowledge was retained by the universities.

NDUS needed a unified, user-friendly and scalable data platform to empower data teams and enable productive AI use cases. As a public sector organization, adding new software tools could be daunting given lengthy procurement process timelines. After being introduced to the Databricks Data Intelligence Platform for Azure, NDUS could move forward quickly with advanced AI tools.

Data enablement drives cost-effective innovation with generative AI

For NDUS, the Databricks Data Intelligence Platform was already a proven foundation for its data needs. So expanding into GenAI by jump-starting new applications using unstructured data was an easy and clear transition. “Once we saw the LLM capabilities of the Databricks Platform, we were impressed that most of our data and AI could be done in one place. We already had Databricks in our Azure cloud environment, so it was the most convenient way to hit the ground running,” Jockers said.

Since NDUS had vetted Azure and was an existing Databricks customer, it had instant access to the Databricks GenAI tools. Leaning on an enablement expert at Databricks for guidance, NDUS started testing different open source LLMs on the Databricks Platform. Cordell Wagendorf, Software Developer for Enterprise Services at NDUS, said, “Performance was our number one criteria in choosing an LLM, and then inference time, size and cost. We ended up on Llama 2, but now with DBRX, we’re going to continue consolidating and simplifying on Databricks.” After LLM evaluation, the data team used Foundation Model APIs to quickly build applications that leverage GenAI without the complexity of custom model deployment.

Data governance is a key requirement across NDUS’ GenAI use cases. With Unity Catalog, it can unify access controls and secure collaboration to ensure the right people access the right data and models. To reduce pipeline maintenance, Vector Search enables automatic data synchronization, ensuring LLM outputs are up to date. Using MLflow, NDUS performs local tests and now has a simple method for running ML and GenAI applications.

Cutting-edge technology revolutionizes AI applications for automatic efficiency

Since taking advantage of the Databricks Data Intelligence Platform on Microsoft Azure, NDUS is not only managing its data in a centralized manner but also leveraging GenAI to serve students, faculty and staff. Now that the system operates from a single, centralized lakehouse, data teams have reduced the time it takes to bring new insights to market from one year to six months.

NDUS began its generative AI journey by building a low-risk, automated search and response application called “Policy Assistant” in six months. Over 3,000 public PDFs are now contained and synthesized in Databricks, with the LLM surfacing automatic policy search and response. Users prompt the LLM within the System with plain English via an API to instantly generate accurate results with references, page numbers and links. Jockers said, “Without having to manually search five different sites for one document, Policy Assistant has increased our team productivity.”

With a proven use case in production, NDUS is testing the automatic generation and daily distribution of private, internal audit reports to authorized individuals. Additionally, NDUS is establishing DLT pipelines for systemwide daily enrollment data on an automated daily enrollment tracker. For data-based decision-making, NDUS plans to expand the tracker in conjunction with LLMs for predictive enrollment forecasting.

Moving forward, NDUS continues to capitalize on the efficiency of the Databricks Data Intelligence Platform, fulfilling new and cost-effective use cases for unstructured news data and domain-specific LLMs. Regular educational events help people from the top down understand what, why and how to use Databricks to support GenAI adoption across the university system. Wagendorf said, “Databricks has been an invaluable tool to lean on and allowed us to experiment with confidence. It’s enlightening, engaging and efficient. It has enabled us to streamline processes and better serve our stakeholders.”