About the opportunity
RBC’s AI Group is building trusted AI capabilities for the enterprise. The Staff Data Scientist, AI Evaluations will lead the data science work for measuring model and agent quality, safety, risk, and performance. This includes developing evaluation datasets, scoring methods, human evaluation protocols, benchmarks, and measurement frameworks, while providing technical guidance and mentorship to other data scientists.
What you’ll do
- Design model and agent evaluation methods, including datasets, rubrics, LLM-as-judge methods, deterministic scorers, human evaluation, and measurement frameworks.
- Define standards that translate model risk, responsible AI, product quality, safety, and business expectations into measurable criteria and repeatable methods.
- Own evaluation dataset and scorecard lifecycles, including sourcing, curation, validation, quality checks, versioning, lineage, and ongoing improvement.
- Develop scalable evaluation methods for generative AI and agentic systems at task, workflow, trajectory, and runtime levels.
- Partner with research, engineering, product, risk, governance, and business teams to embed evaluations into development, release, certification, monitoring, and recertification.
- Establish human evaluation and review protocols, and improve scorer accuracy, calibration, robustness, failure-mode coverage, and explainability.
- Provide technical leadership, communicate findings, and mentor junior data scientists.
What you’ll need
- At least 8 years of experience in data science, applied machine learning, AI evaluation, ML quality, or a related technical field, including technical leadership and mentorship.
- Experience designing evaluation frameworks for machine learning, generative AI, or agentic AI systems.
- Practical experience with methods such as LLM-as-judge, deterministic scoring, human evaluation, hallucination and factuality assessment, safety evaluation, or benchmarking.
- Strong foundation in data science, statistics, machine learning, experimentation, data curation, Python, SQL, and modern AI/ML practices.
- Ability to translate governance, model risk, responsible AI, and business requirements into measurable controls and decision-ready evidence.
- Strong communication and stakeholder management skills.
Experience evaluating agentic systems, financial services or regulated AI, and evaluation or observability tooling is an asset. A relevant master’s or Ph.D. is also an asset.
What RBC offers
A comprehensive total rewards program, development and coaching opportunities, challenging work, and a collaborative team.
Position details
Location: 6666 Rue St-Urbain, Floor 3, Montréal, Quebec
Hours: 37.5 per week
Employment type: Full-time, regular, salaried
Application deadline: Applications accepted until 11:59 PM on the day before October 31, 2026.