Tech Lead / Manager, Machine Learning Research Scientist- LLM Evals

Overview

Tech Lead Manager, Machine Learning Research Scientist- LLM Evals at Scale AI leads a team of research scientists and research engineers to develop and implement novel evaluation methodologies, metrics, and benchmarks for large language models. The role designs and executes a roadmap that defines best practices in data-driven AI development and accelerates the next generation of generative AI models in partnership with top foundational model labs.

Responsibilities

• Lead a team of research scientists and research engineers on LLM evals.
• Conduct research on the effectiveness and limitations of existing LLM evaluation techniques.
• Design and develop novel evaluation benchmarks for large language models, covering instruction following, factuality, robustness, and fairness.
• Communicate and collaborate with clients and peer teams to facilitate cross-functional projects.
• Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols.
• Implement scalable and reproducible evaluation pipelines using modern ML frameworks.
• Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives.
• Stay up-to-date on research developments, participate in design decisions, and contribute to the research community.
• Thrive in a high-energy, fast-paced startup environment and dedicate time and effort to drive impactful results.

Qualifications

Compensation and Benefits

Compensation packages include base salary, equity, and benefits. The stated salary range for this full-time role in San Francisco, New York, and Seattle is $260,000 – $350,000 USD. Scale employees may also receive equity-based compensation. Benefits include comprehensive health, dental, and vision coverage, retirement benefits, a learning and development stipend, and generous PTO. Some roles may include additional benefits such as a commuter stipend.

About Scale and EEO

Scale AI is an equal opportunity employer. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability status, gender identity, or veteran status.

#J-18808-Ljbffr