Back to Projects

Apple · 2021 – 2026

Scaling a microservice application to run 2,000+ Spark jobs daily on a 20 PB Iceberg lakehouse

I scaled the service that launches 2,000+ Spark jobs a day on our lakehouse, then tuned Spark to make those jobs faster and cheaper.

Senior Software Engineer, Data Platform

Iceberg data lake

20 PB

Saved per year

$0.5M

Faster micro-batches

75%

Problem

Our shared platform ran Spark jobs against a 20 PB Iceberg data lake. As more teams piled on, the microservice that launches and manages those jobs had to keep up, and our micro-batch ETL pipelines on AWS were slower and pricier than they should have been.

Approach

  • Scaled the microservice that runs 2,000+ Spark jobs a day across the lakehouse.
  • Tuned Spark configs across 20+ ETL pipelines running on Airflow, Spark, and Kubernetes in AWS.
  • Pushed Snowflake and dbt adoption, wired them into Airflow, and migrated legacy pipelines onto them.

Results

  • About $0.5M a year saved on compute.
  • Micro-batch runs got 75% shorter.
  • Internal lakehouse usage grew 20% year over year.

Stack

  • Apache Spark
  • Apache Iceberg (20 PB)
  • Airflow
  • Kubernetes
  • AWS
  • Snowflake
  • dbt
  • Python
  • Microservices