Libin Guo · Senior Software Engineer (Data Platform)
“I have no special talent. I am only passionately curious.”
I'm a Senior Software Engineer (Data Platform) with 9+ years building large-scale data pipelines and data platforms — most recently a 20 PB Iceberg lakehouse at Apple and video training-data pipelines for world models at Reka AI.
Open to Senior Data, Data Platform, AI & Applied AI Engineer rolesAustin, TX

- 2015UT DallasM.S., Information Technology & ManagementWhere it started
- 2016QlarantSenior Data AnalystMedicare fraud analytics
- 2018HealthmineData EngineerHealthcare ETL pipelines and a Postgres analytics warehouse
- 2020JPMorgan ChaseData EngineerReal-time ingestion on AWS
- 2021AppleSenior Software Engineer, Data PlatformScaled the lakehouse platform · Modernized legacy pipelines: on‑prem to cloud, pandas to Spark
- 2026Reka AIMember of Technical Staff, Data PlatformVideo training data for world models
9+
years building data pipelines and platforms
20 PB
Iceberg lakehouse at Apple
2,000+
Spark jobs run daily
$0.5M
saved per year through Spark tuning
- Spark
- Python
- Airflow
- Kubernetes
- AWS
What I Do
Focus areas
01
Spark data pipelines & Iceberg lakehouses
Large-scale Spark pipelines on Apache Iceberg — tuned for speed and cost, and run reliably at thousands of jobs a day.
02
AI training-data pipelines
Ray-on-Kubernetes extraction and batch inference, data-quality gates, and WebDataset delivery for pretraining and fine-tuning.
03
AI agents for data platforms
MCP-based agents and LLM evaluation frameworks that automate platform work, like migration assessment.
Selected Work
Production projects
- Reka AI · 2026Video training-data pipelines for world modelsI build the Ray and PySpark pipelines that turn raw video into clean training data for pretraining, SFT, and distillation.
- Apple · 2021 – 2026Scaling a microservice application to run 2,000+ Spark jobs daily on a 20 PB Iceberg lakehouseI scaled the service that launches 2,000+ Spark jobs a day on our lakehouse, then tuned Spark to make those jobs faster and cheaper.
- Apple · 2021 – 2026An agentic AI application that assesses and migrates lakehouse jobs through a conversational interfaceMCP servers and a chat interface so teams can assess and migrate Snowflake ingestion jobs to the lakehouse by talking to an agent, plus evals so I could trust what it said.
- Dinner Match Lab · 2026 – presentA meal recommendation app that learns your taste from every swipeI designed, built, and run Dinner Match Lab end to end: swipe-based recipe picks, a learned reranker, daily personalized top 10s, and the data pipelines behind them, on web and mobile.
Hiring for a senior data role?
I'd like to hear about the platform you're building and the data problems behind it.