Online Learning with Databricks

Recently, a gentleman named Scott Galloway was featured in an article in New York Magazine about how Covid-19 is breaking the higher education business model.  The article led to an invitation to a live  interview with Anderson Cooper (full transcript of that interview can be found here).  When I watched that segment and saw Anderson Cooper’s eyes go wide, it brought me back to a faculty meeting circa 1998, wherein a roomful of tenured faculty members worried about their relevance as pre-millennium venture capital met the business opportunities of online education. Now it appears that the COVID-19 pandemic has brought this concern to our front door.

So here we are now, in uncertain times for our politics, our society, our economy, and for all of the educational systems that will produce the leaders of tomorrow.  Tough problems for a tough time… Where do we go from here?

Making Online Learning Better

Online learning is the new normal, the primary learning environment.  In this digital world, teachers are provided new lenses through which to engage students.  While teachers no longer have a brick and mortar classroom and face-to-face office hours for instruction, the digital world gives them a different set of teaching tools.

In this digital world, every interaction between the student and the teacher, the student and the content, the student and his or her project groups, viewing external information sources, etc, leaves a data footprint.  The digital learning environment can tell us what content students interact with, how long they spend on a page or video replay, if they are writing originally or executing a ‘copy and paste’ function, how they are collaborating with classmates, and a host of other information about their experience.  In a video conference session, it can tell us if they are attentive to the screen as well as if they are interacting with quiz, testing, or survey content.

Data from populations of students can identify how they respond to different combinations of classes, identifying class combinations that are toxic as well as those likely to lead them to flourish in their learning experience.  Their social media feeds can inform satisfaction with their learning experience and their engagement with their education community, allowing educators to make real-time adjustments to teaching and engagement.

COVID-19 has forced this mode of education on everyone everywhere.  We can’t change that reality, but we can certainly learn from it!  It is an educational experiment mediated by technology whose scope will never be repeated (we hope), so let us pose some timely questions:

Can online instruction scale up without losing impact?  At what scale do we have diminishing returns, if any?  One instructor for a physical classroom filled with 30 students is a norm; what is a good norm for online instruction, and how should we leverage technology to make it as effective as possible?

Is there a way to look at student interactions and student grades so that we have early warning systems for identification and intervention with at-risk students?  Can we look at data to help us tailor courses and learning materials to maximize the success potential for students with non-traditional backgrounds? Can we identify mental health concerns to help students get access to resources and support before things get worse?

If the business model of higher education is broken, how must it evolve?  Over the long run, what determines the value of a degree from a specific school with a specific major?  In the short run, how can schools leverage data from across its enterprise to optimize recruiting, retention, and job placement upon graduation?  If a university must focus on scarcity to maintain its brand value, how can it best engage its network of graduates, students, and parents to participate in school activities?

None of these questions are new, but their urgency has certainly spiked, and data analysis at scale can certainly help solve these tough problems, as shown by the following examples.

Real-world Examples

A virtual classroom at Berkeley with global reach and key technology partners

According to Scott Galloway in the transcript of his interview with Anderson Cooper, the University of California, Berkeley will graduate more kids from low-income households this year than the entire Ivy League.  In addition, the university has been using tools and methods for instruction that have a global reach for many years.  This recent article from InformationWeek highlights one such course that has been using technology to run a global classroom in real time since 2014.  Kyle Hamilton, one of the instructors for Data Science W261: Machine Learning at Scale course, teaches students the skills needed to process the big data needed to address the tough problems mentioned above.  The course builds on and goes beyond the collect-and-analyze phase of big data by focusing on how machine learning algorithms can be rewritten and extended to scale to work on petabytes of data, both structured and unstructured, to generate sophisticated models used for real-time predictions.  Databricks is honored that Kyle has chosen us as a technology partner.

Using data at scale to improve the business model of online education

Western Governors University (WGU) was founded in 1997 by 19 US governors as a non-profit, all-online competency-based university offering undergraduate and graduate degrees.  It has more than 180,000 graduates, 123,000 active students and more than 6,800 employees.  WGU has made a commitment to understanding its students, employees, and educational platforms through data.  Databricks helps Western Governors improve student success by providing a one-stop for data access, democratizing data to allow for access to all employees, freeing up employees to do deep-dives on course- and student-related dashboards, and improving time-to-insight via streamlined ETL.

WGU uses data not only to deliver education content, but for real-time assessment of each student’s learning experience, with real-time data systems to inform teachers what is working, and to intervene directly with students where the process is not working.  WGU also applies data analytics over time with AI models that help educators learn to optimize the learning experience.

Using Data to Map the Way Forward

Data has always had the potential to make education better, improving its reach and impact across digital divides, and help the millennia-old model of the classroom adjust to modern-day economic and social constructs.  It is up to us now to map the path forward, using data to find the truths that will help the current educational system efficiently produce a generation of diverse, well-educated and workforce-ready students.

Getting Started

Try Databricks for free Get started

Sign up