Raj Bains is the Founder, CEO of Prophecy.io – focused on Data Engineering on Spark. Previously, Raj was the product manager for Apache Hive at Hortonworks and knows ETL well. He has extensive expertise in programming languages, tools and compilers – as a member of the early CUDA team at NVIDIA, and as part of Microsoft Visual Studio team. He has also developed a language for computable insurance contracts. He knows database internals and has worked on NewSQL databases
Enterprises used ETL tools for decades for higher productivity and standardization. Data Engineers see these tools don't work anymore and have moved to code. However, we're back again to ad hoc scripts and frameworks -reminding us of the world before ETL tools. We show how a new generation of tools can be built for Spark development based on code - for productivity, code standardization, metadata and lineage, and for agility via CI/CD.