The role is focused on driving the applied science foundation for observability in AI agents and multi-agent systems at Microsoft. The team aims to redefine how software is built and experienced using Generative AI. The position involves developing scientific methods and evaluation frameworks to understand and improve the performance of intelligent agents in production. The candidate will work at the intersection of agent architecture, telemetry, evaluation science, and responsible AI.

• Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python.
• Ability to meet Microsoft, customer and/or government security screening requirements.
• 6+ years of experience in applied science, machine learning, evaluation systems, or related technical fields.
• Strong experience designing evaluation methodologies, experiments, or measurement systems for complex intelligent or distributed systems.
• Experience analyzing large-scale production or experimental data to derive actionable insights and drive product or system improvements.
• Strong coding and prototyping skills in Python or similar languages.
• Demonstrated ability to lead cross-team technical direction through scientific depth, influence, and strong problem framing.
• Advanced degree in Computer Science, Machine Learning, Statistics, Applied Mathematics, or related field.

• Develop evaluation and measurement frameworks for single-agent and multi-agent systems.
• Design methodologies that connect offline evaluations, online signals, and production telemetry.
• Define scientifically grounded quality signals and benchmarks for agent systems.
• Build models and analysis techniques to detect regressions and identify root causes.
• Advance observability for AI systems through new approaches to trace analysis and anomaly detection.
• Partner with engineering teams to operationalize evaluation and observability methods in production systems.
• Contribute to instrumentation and semantic standards for agent observability.
• Collaborate with product and platform teams to shape end-to-end experiences for evaluation and monitoring.
• Act as a technical leader by setting scientific direction and mentoring others.

• Employees at Microsoft are often offered comprehensive, “world-class” benefits—including health and mental-wellness programs, competitive pay with bonuses and stock awards, and retirement/savings options. Time-off and flexibility are common, with generous vacation and holidays, parental and caregiver leave, and flexible work schedules, alongside learning support, employee resource groups, product discounts, and matching-gifts/volunteering programs. Specific benefits can vary by region.