Client Service Risk Manager - AI Evaluation and Testing

Pracovní nabídka

We are seeking for someone who will be ensuring our firm’s strong ability to evaluate and validate AI systems including those used and built internally and in client contexts by consultants for reliability, safety, security, and regulatory compliance.

What you will do:

You, as the lead in AI evaluation and testing, will define the vision and strategic roadmap for AI control objectives and testing. You will establish frameworks, requirements, and guidance to assess behavior and ensure reliability (including accuracy, bias, and other factors) across various modalities such as text, code, multimodal systems, and agents. Additionally, you will define and prioritize the requirements for tooling solutions that support AI governance, observability, and testing for all relevant AI systems and agents.

You will also develop or validate frameworks and methodologies for conducting red-team attacks on AI systems (jailbreaks, prompt injections, system prompt extraction). You will partner with first line of defense to implement evaluation gates and build a control environment with appropriate technical guardrails to ensure compliance with applicable regulations (e.g., the EU AI Act).

You will report directly to the Associate Director of Client Service Risk, Technology Knowledge & Frameworks, within the firm’s global Risk function. You will partner closely with Legal, Risk teams, Technology and Product teams to develop consistent and scalable RAI control objectives and testing methodologies.

You will support updates to existing robust policies and standards in line with development in AI capabilities (e.g., agentic).

Your collaboration will extend to various roles and functions across the firm, including senior leadership, to whom you will present and gain support for your strategy and program, report progress, and demonstrate impact.

What we are looking for:

• U.S. Bachelor’s degree or equivalent work experience required
• 8+ years in Trust & Safety (AI/ML), risk, or security domains with Data Science background; the scope of the models includes traditional ML models, GenAI and agentic deployments
• 5+ years managing complex cross-functional projects
• Familiarity with AI safety, testing and validation, and governance; experience partnering with engineers to build control and integrity systems
• Knowledge of AI evaluation methods and ability to connect evaluation results to guardrails and controls; Strong knowledge of LLM behavior and jailbreak techniques
• Proven ability to translate complex ethical challenges into actionable policies and frameworks; strong policy development, risk management, and analytical skills
• Experience working with engineering and product teams to create tools, solutions, or automation
• Ability to work in-person (team schedules and office locations may vary)
• Ability to operate office equipment
• Excellent leadership and communication skills, with a proven ability to navigate ambiguity and influence product direction in a fast-changing environment
• Excellent written and oral communication skills, as well as interpersonal skills, including the ability to articulate technical concepts to both technical and non-technical audiences