If you ever wrote a PySpark job you probably struggled with the same challenges we had in Armis. How do I test my jobs? How do I deploy it? How to monitor my job? How do I make everyone follow the same standard?
It is already common knowledge how to build, test and deploy micro-services and web applications, but Spark jobs are different: They require special dependency management and deployment options. In this talk we will discuss how to correctly package PySpark application and how we package them in Armis using Bazel.
I will explain how to design a “DevOps Interface” that will provide both flexibility and standardization that developers love to use.
To complete the picture I will focus on the PySpark CI flow and how we built it incrementally.
How we made our developers fall in love with PySpark
Full Talk (40 Minutes)
Food & Swag Sponsors
Learn more about each of our Event Sponsors.