Architectural Overview of Atlassian's Next-Generation Data Lakehouse

We are rebuilding our data lakehouse from the ground up - a greenfield re-design based on everything we have learned from the last 5 years of working with Databricks. In this talk, we’ll give an architectural overview of how we’re setting up this new lake, covering topics like:

• How we are laying out our Databricks accounts/workspaces and AWS accounts to create the concept of “environments”
• Our “workbench environment” concept that separates insights work from production pipelines
• The benefit we get from doubling down on Unity Catalog, Delta Lake, and Managed Tables
• Creation and enforcement of a consistent information architecture
• Making our data lakehouse declarative, and keeping human users out of the production environment
• Supporting and governing machine learning workloads