The Senior Software Engineer on the Spark Platform team at DoorDash will lead the technical direction for the in-house Spark deployment, focusing on architecture that supports data, analytics, and ML workloads. This role involves tackling complex distributed systems challenges and optimizing runtime performance. The engineer will collaborate with the Engineering Manager on the technical roadmap and team development while mentoring other team members. The position requires a strong understanding of Apache Spark and its operational aspects at scale.

• B.S., M.S., or PhD in Computer Science or equivalent.
• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep, hands-on experience with Apache Spark internals and operations.
• Production experience with remote/external shuffle systems or batch/big-data schedulers.
• Strong fluency operating workloads on Kubernetes in production.
• Familiarity with data lake table formats such as Apache Iceberg or Delta Lake.
• Track record of acting as a technical leader on a platform team.
• Professional experience with Scala, Java, Python, or Go; strong SQL.

• Set the multi-year technical direction for the Spark-on-Kubernetes platform.
• Own the distributed-systems problems related to shuffle architecture and multi-tenant scheduling.
• Partner with the Engineering Manager on technical roadmap and team shape.
• Uplevel the team through design reviews and mentorship.
• Represent Spark Platform in cross-team architecture forums.