The Senior Software Engineer on the Spark Platform team at DoorDash will lead the technical direction for the in-house Spark deployment, focusing on architecture that supports data, analytics, and ML workloads. This role involves tackling complex distributed systems challenges and optimizing runtime performance. The engineer will collaborate closely with the Engineering Manager and other teams to shape the technical roadmap and mentor team members. The position requires a strong background in Apache Spark and distributed systems at scale.

• B.S., M.S., or PhD in Computer Science or equivalent.
• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep, hands-on experience with Apache Spark internals and operations at platform scale.
• Production experience with remote/external shuffle systems or batch/big-data schedulers.
• Strong fluency operating workloads on Kubernetes in production.
• Familiarity with data lake table formats such as Apache Iceberg or Delta Lake.
• Track record of acting as a technical leader on a platform team.
• Professional experience with Scala, Java, Python, or Go; strong SQL.

• Set the multi-year technical direction for the Spark-on-Kubernetes platform.
• Own the deepest distributed-systems problems related to shuffle architecture and multi-tenant scheduling.
• Partner with the Engineering Manager on technical roadmap and team growth.
• Uplevel the team through design reviews and mentorship.
• Represent the Spark Platform in cross-team architecture forums.