Senior AI Data Engineer

The Role

This role is responsible for ensuring the reliability, accuracy, and safety of our Veeva AI Agents through rigorous evaluation and systematic validation methodologies. We're looking for experienced candidates with :

1. A meticulous, critical, and curious mindset with a dedication to product quality in a rapidly evolving technological domain2. Exceptional analytical and systematic problem-solving capabilities3. Excellent ability to communicate technical findings to both engineering and product management audiences4. Ability to learn application areas quickly

Thrive in our Work Anywhere environment : We support your flexibility to work remotely or in the office within Canada or the US, ensuring seamless collaboration within your product team's time zone.Join us and be part of a mission-driven organization transforming the life sciences industry.

What You'll Do

• Evaluation Strategy & Planning : Define and establish comprehensive evaluation strategies for new AI Agents. Prioritize the integrity and coverage of test data sets to reflect real-world usage and potential failure modes
• LLM Output Integrity Assessment : Programmatically and manually evaluate the quality of LLM-generated content against predefined metrics (e.g., factual accuracy, contextual relevance, coherence, and safety standards)
• Creating High-Fidelity Datasets : Design, curate, and generate diverse, high-quality test data sets, including challenging prompts and scenarios. Evaluate LLM outputs to proactively identify system biases, unsafe content, hallucinations, and critical edge cases
• Automation of Evaluation Pipelines : Develop, implement, and maintain scalable automated evaluations to ensure efficient, continuous validation of agent behavior and prevent regressions with new features and model updates
• Root Cause Analysis : Understand model behaviors and assist in the trace and root-cause analysis of identified defects or performance degradations
• Reporting & Performance Metrics : Clearly document, track, and communicate performance metrics, validation results, and bug status to the broader development and product teams

Requirements

Perks & Benefits

Compensation