The Annotation Data Scientist role on the Evaluation Integrity team focuses on enhancing the evaluation processes for Siri by designing and executing human-in-the-loop annotation projects. This position combines data science, human annotation engineering, and evaluation methodology to ensure the quality and authenticity of user interactions with Siri. The team is dedicated to creating a reliable quality signal that informs product decisions. The role requires collaboration across various functions to drive annotation initiatives from conception to actionable insights.

• Bachelor's or Master's degree in a quantitative or related field such as Data Science, Computer Science, Linguistics, Statistics, or Cognitive Science.
• 3+ years of hands-on experience with human-annotated datasets or human-in-the-loop evaluation methodologies.
• 3+ years of experience using Python for data processing, analysis, and prototyping.
• Experience designing and communicating annotation schemas, rubrics, or ontologies for machine learning.
• Experience managing multiple dataset curation efforts and monitoring annotator performance metrics.
• Experience specifying or designing custom annotation tooling in collaboration with software engineers.
• Familiarity with statistical methods addressing accuracy and variability in human annotation data.
• Data-querying experience with SQL, Spark, or similar technologies.
• Experience building pre-ship evaluation pipelines for conversational products.
• Excellent written and verbal communication skills.

• Design and run human-in-the-loop annotation projects to evaluate user personae and agent conversations.
• Own annotation initiatives from rubric design to data science analysis.
• Coordinate with in-house or vendor annotators and monitor their performance metrics.
• Collaborate with software engineers to specify or design custom annotation tooling.
• Evaluate LLM-powered systems and implement rubric-based grading methodologies.
• Communicate technical topics clearly to cross-functional partners and stakeholders.

• Employees at Apple are often offered comprehensive benefits that support physical and mental well-being—flexible medical plans, confidential counseling, onsite wellness centers at major campuses, and resources for fitness and daily life. Families typically receive fertility support, paid parental leave with gradual return, caregiving leave, and dependent-care guidance, while financial perks commonly include stock grants (with purchase discounts), 401(k) matching, and income-protection coverage. Employees also see robust time off, Apple University learning and tuition reimbursement, donation matching and paid volunteer hours, and deep product and partner discounts.