Maciej Szpakowski is the Co-Founder of Prophecy – Low code Data Engineering on Apache Spark & Airflow. He’s focused on building Prophecy’s unique Code = Visual IDE on Spark. Previously, he founded a medical research automation company based on Spark ML.
May 26, 2021 11:30 AM PT
Development on Apache Spark can have a steep learning curve. Low-code offers an option to enable 10x more users on Spark and make them 10x more productive from day 1. We’ll show how to quickly develop and test using Visual components and SQL expressions.
Spark Frameworks can standardize development. We’ll show you how to create a framework, using which your entire team will be able to create complex workflows with a few clicks.
May 27, 2021 11:00 AM PT
What do you do after your Spark workflow development is finished? Nowadays, data engineers dread putting their workflows in production. Apache Airflow provides the necessary scheduling primitives but writing the glue scripts, handling Airflow operators and workflow dependencies sucks! Low-Code Development & Deployment can make scheduling Spark workflows much simpler – we’ll show you how.
Additionally, we show column-level lineage for governance such as tracing your PII data, or for production issues such as understanding the downstream impact of changes to a workflow.