The Staff Engineer role at Databricks focuses on driving the technical direction of Spark Structured Streaming, which is integral to the Databricks Data Intelligence Platform. The team is dedicated to building advanced stream processing capabilities and enhancing the architecture for improved performance. This position requires a strong background in big data systems and a passion for database technologies. The engineer will also mentor team members and collaborate with product management to ensure alignment with customer needs.

• BS (or higher) in Computer Science or a related technical field, or equivalent practical experience
• 8+ years building related systems such as big-data ecosystems, Apache Spark, or database internals
• A passion for database systems, storage systems, distributed systems, language design, or performance optimization
• Comfortable working toward a multi-year vision with incremental deliverables
• A track record of delivering features while maintaining a high bar for operational excellence and engineering quality
• Comfortable working cross-functionally with product management and directly with customers

• Set and drive the technical vision for Spark Structured Streaming across OSS and the Databricks Data Intelligence Platform
• Design and build core engine capabilities including state management, new operators, and architectural improvements to latency and throughput
• Raise the bar for engineering quality and operability by building software that is high quality and easy to run in production
• Make company-wide impact by driving stream processing adoption across the Databricks product portfolio
• Guide long-term architecture and technical-debt decisions while balancing them against the product roadmap
• Mentor and technically lead engineers on the team and partner with the Engineering Manager to attract and grow top-tier talent