Problem
Our shared platform ran Spark jobs against a 20 PB Iceberg data lake. As more teams piled on, the microservice that launches and manages those jobs had to keep up, and our micro-batch ETL pipelines on AWS were slower and pricier than they should have been.