The Software Engineer on the Spark Platform team will work on DoorDash's Apache Spark ecosystem, focusing on runtime upgrades, multi-tenant scheduling, and cluster lifecycle automation. This role involves optimizing distributed systems and enhancing observability and incident automation to ensure platform sustainability. The engineer will collaborate closely with team members and platform consumers to tackle high-leverage problems across various layers of the system. Candidates must be located in specific cities for this hybrid position.

• B.S., M.S., or PhD in Computer Science or equivalent.
• 24+ years of industry experience operating production distributed systems.
• Experience operating Apache Spark at scale on Amazon EMR, Databricks, or an in-house deployment.
• Hands-on experience operating production systems on Kubernetes.
• Familiarity with batch or big-data schedulers and/or with the Spark-on-Kubernetes operator.
• Familiarity with observability stacks and defining SLOs and SLIs.
• Comfort working in a cloud environment, preferably AWS.
• Professional experience with Python, Go, Scala, or Java; SQL fluency.
• A bias toward incremental rollout, measurement, and reducing toil.

• Build and operate an in-house Spark platform that runs at company-wide scale.
• Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement.
• Own pieces of cluster lifecycle automation including provisioning and upgrades.
• Build observability and incident automation for end-to-end debugging.
• Partner with senior engineers on shuffle, runtime, and architecture work.