Technical Staff Member, Pretraining Evaluations

A company is looking for a Member of Technical Staff, Pretraining Evaluations.

Key Responsibilities

Understand and evaluate tasks in the base model evaluation suite, identifying strengths and limitations

Implement improvements to the evaluation suite by adding new tasks or removing ineffective ones

Enhance statistical understanding and improve the signal-to-noise ratio of evaluations

Required Qualifications

Familiarity with base model evaluations and their differences from post-trained models

Strong statistical skills and experience with scientific experiments related to data collection and model performance

Extremely strong software engineering skills

Proficiency in programming languages such as Python and ML frameworks (e.g., PyTorch, TensorFlow, JAX)

One or more papers published at top-tier venues (e.g., NeurIPS, ICML, ICLR)