Raj Bains is the Founder, CEO of Prophecy.io – focused on Data Engineering on Spark. Previously, Raj was the product manager for Apache Hive at Hortonworks and knows ETL well. He has extensive expertise in programming languages, tools and compilers – as a member of the early CUDA team at NVIDIA, and as part of Microsoft Visual Studio team. He has also developed a language for computable insurance contracts. He knows database internals and has worked on NewSQL databases
Development on Apache Spark can have a steep learning curve. Low-code offers an option to enable 10x more users on Spark and make them 10x more productive from day 1. We’ll show how to quickly develop and test using Visual components and SQL expressions. Spark Frameworks can standardize development. We’ll show you how to create a framework, using which your entire team will be able to create complex workflows with a few clicks.
What do you do after your Spark workflow development is finished? Nowadays, data engineers dread putting their workflows in production. Apache Airflow provides the necessary scheduling primitives but writing the glue scripts, handling Airflow operators and workflow dependencies sucks! Low-Code Development & Deployment can make scheduling Spark workflows much simpler – we’ll show you how. Additionally, we show column-level lineage for governance such as tracing your PII data, or for production issues such as understanding the downstream impact of changes to a workflow.
Enterprises used ETL tools for decades for higher productivity and standardization. Data Engineers see these tools don't work anymore and have moved to code. However, we're back again to ad hoc scripts and frameworks -reminding us of the world before ETL tools. We show how a new generation of tools can be built for Spark development based on code - for productivity, code standardization, metadata and lineage, and for agility via CI/CD.