The role is for an engineer on the Ingestion Core Team at Databricks, focusing on simplifying data ingestion processes for both structured and unstructured data. The team aims to build distributed platform systems that efficiently ingest high-volume data from various sources. This position involves optimizing streaming workloads and enhancing monitoring capabilities to improve data availability in Delta Lake. The engineer will also collaborate with partner teams to support innovative use cases in AI and data.

• 5+ years of experience writing production code in Java, Scala, Go, C++, or Python.
• Experience architecting, developing, and deploying large-scale distributed and asynchronous systems.
• Experience with distributed systems, streaming, Spark, databases, data processing, or CDC.
• Experience building or operating systems where scale, throughput, latency, reliability, and cost are important considerations.
• Comfortable using AI tools and defining effective evaluations to shorten the development loop.

• Build distributed infrastructure to ingest data from diverse sources and support streaming ingestion, incremental processing, and replication.
• Reduce end-to-end latency, increase throughput, and reduce costs from the time data appears in source systems to when it is available in Delta Lake.
• Design and optimize streaming and distributed workloads for throughput, cost, latency, reliability, and scale.
• Optimize streaming workloads by exploring and applying machine learning techniques.
• Build monitoring and observability capabilities that provide visibility into ingestion workflows and the systems running them.
• Collaborate with partner teams to enable use cases like RAG and AI agents.
• Build Agentic SDKs with strong evaluations.