ETL has been around since the 90s, supporting a whole ecosystem of BI tools and practises. While traditional ETL has proven its value, it’s time to move on to modern ways of getting your data from A to B. Since BI moved to big data, data warehousing became data lakes, and applications became microservices, ETL is next our our list of obsolete terms. Spark provides an ideal middleware framework for writing code that gets the job done fast, reliable, readable. In this session I will support this statement with some nice ‘old vs new’ diagrams, code examples and use cases. Please join if you want to know more about the NoETL paradigm, or just want to be convinced of the possibilities of Spark in this area!
Bas is a programmer, scientist, and IT manager. At ING, he is responsible for the Fast Data chapter within the Analytics department. His academic background is in Artificial Intelligence and Informatics. His research on reference architectures for big data solutions was published at the IEEE conference ICITST 2013. Bas has a background in software development, design and architecture with a broad technical view from C++ to Prolog to Scala and is a Spark Certified Developer. He occasionally teaches programming courses and is a regular speaker on conferences and informal meetings.