I moved from Wisconsin to Utah in 2009 to complete my BS and MS in Information Systems at the University of Utah. I started my work at Western Governors University in 2012, eventually becoming a Senior Big Data Developer for our Enterprise Solutions team. I have been working with Databricks for 4 years and have split my time between working in both Python and Scala. While at WGU, I helped architect and migrate our Enterprise Data Warehouse off of Oracle and into Databricks Delta (Delta.io). I currently write ETL/ELT pipelines into our Delta Lake, administrate all of our 120+ users on the Databricks Unified Platform and help manage our AWS cloud resources for our analytics teams.
May 28, 2021 11:40 AM PT
Implementing tools, let alone an entire Unified Data Platform, like Databricks, can be quite the undertaking. Implementing a tool which you have not yet learned all the ins and outs of can be even more frustrating. Have you ever wished that you could take some of that uncertainty away? Four years ago, Western Governors University (WGU) took on the task of rewriting all of our ETL pipelines in Scala/Python, as well as migrating our Enterprise Data Warehouse into Delta, all on the Databricks platform. Starting with 4 users and rapidly growing to over 120 users across 8 business units, our Databricks environment turned into an entire unified platform, being used by individuals of all skill levels, data requirements, and internal security requirements.
Through this process, our team has had the chance and opportunity to learn while making a lot of mistakes. Taking a look back at those mistakes, there are a lot of things we wish we had known before opening the platform to our enterprise.
We would like to share with you 10 things we wish we had known before WGU started operating in our Databricks environment. Covering topics surrounding user management from both an AWS and Databricks perspective, understanding and managing costs, creating custom pipelines for efficient code management, learning about new Apache Spark snippets that helped save us a fortune, and more. We would like to provide our recommendations on how one can overcome these pitfalls to help new, current and prospective users to make their environments easier, safer, and more reliable to work in.