The Senior Security Researcher and Principal Security Researcher role on the AI Evaluation Systems team at Microsoft Security focuses on building a common evaluation platform for agentic engineering workflows. This position requires engineers to design benchmarks, implement validation systems, and analyze quality and cost tradeoffs. The team aims to ensure that AI agents are ready for production use in security contexts. The role emphasizes collaboration across various teams to make evidence-based release decisions and improve agentic systems continuously.

• Doctorate in Statistics, Mathematics, Computer Science, Computer Security, or related field, or equivalent experience.
• Master's Degree in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 3+ years of experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection.
• Bachelor's Degree in Statistics, Mathematics, Computer Science, Computer Security, or related field AND 4+ years of experience in software development lifecycle, large-scale computing, threat analysis or modeling, cybersecurity, vulnerability research, and/or anomaly detection.
• Proven software engineering experience building production systems, developer platforms, test infrastructure, automation frameworks, data pipelines, quality systems, or reliability tooling.
• Ability to design and implement evaluation systems end to end, including task definition, dataset creation, harness implementation, scoring, analysis, and operational integration.

• Design and build end-to-end evaluation harnesses for agentic security and engineering workflows.
• Create representative benchmark suites and golden datasets that include various case types.
• Implement deterministic validators, automated graders, and result stores to make evaluations repeatable and actionable.
• Measure task success, correctness, safety and policy compliance, and other relevant metrics.
• Integrate evaluations into engineering workflows and decision processes to support reproducible evidence.
• Connect offline evaluation results with production feedback to enhance system performance.
• Detect regressions and benchmark drift to ensure evaluation accuracy.
• Partner with various teams to turn evaluation results into actionable release recommendations.
• Convert learnings into reusable resources such as playbooks and dashboards.

• Employees at Microsoft are often offered comprehensive, “world-class” benefits—including health and mental-wellness programs, competitive pay with bonuses and stock awards, and retirement/savings options. Time-off and flexibility are common, with generous vacation and holidays, parental and caregiver leave, and flexible work schedules, alongside learning support, employee resource groups, product discounts, and matching-gifts/volunteering programs. Specific benefits can vary by region.