Talent.com
The Fountain Group
LLM Evaluation EngineerThe Fountain Group • Mountain View, CA
LLM Evaluation Engineer

LLM Evaluation Engineer

The Fountain Group • Mountain View, CA
30+ days ago
Job type
  • Full-time
Job description
About the Role:
We are seeking a LLM Evaluation Engineer to join a forward-thinking team responsible for developing a sophisticated voice assistant platform. This isn’t your typical QA role – it’s a unique blend of technical engineering, machine learning evaluation, and data analysis. You’ll work closely with cutting-edge conversational AI technology, designing evaluation frameworks, building custom scripts, and creating data visualizations to assess platform performance.

Key Responsibilities:
  • Design and implement evaluation strategies for voice and language models, including automated testing approaches.
  • Analyze unstructured data from log store systems to identify performance gaps and optimize user experiences.
  • Build and maintain custom Python scripts to streamline data processing and generate actionable insights.
  • Develop visual reports to communicate findings and drive continuous improvement.
  • Collaborate with cross-functional teams globally to identify and address pain points in conversational AI performance.
  • Use prompt engineering techniques to refine LLM outputs and articulate system health.

Ideal Candidate:
  • 3+ years of experience in machine learning evaluation, data analysis, or related technical roles.
  • Intermediate to advanced Python scripting, including log parsing and API testing.
  • Familiarity with GenAI and LLMs, including automated workflows and API integrations.
  • Strong analytical mindset, capable of working independently and identifying innovative solutions.
  • Excellent communication skills, able to present complex findings clearly to both technical and non-technical stakeholders.

Create a job alert for this search

LLM Evaluation Engineer • Mountain View, CA

Similar jobs

Forward Deployment Engineer (Inference & RL POC)

Glint Tech Solutions LLCMountain View, California, United States
Full-time
Quick Apply

Bay area (frequent customer interaction).Inference & Reinforcement Learning Platform.Forward Deployment Engineer (FDE).You'll be embedded with customers during early-stage deployments—turning r... Show more

Senior Software Engineer, Simulator Evaluation & Metrics

WaymoMountain View, CA, United States
Full-time

A leading autonomous driving technology company is seeking a Senior Software Engineer to architect evaluation methodologies for their hybrid simulator.Responsibilities include developing high-throu... Show more

 • Promoted

Machine Learning Engineer, LLM Fine-Tuning

First Soft Solutions LLCSan Jose, California, US
Full-time

Machine Learning Engineer, LLM Fine‑Tuning.Find out more about the daily tasks, overall responsibilities, and required experience for this opportunity by scrolling down now.LLM fine‑tuning for Veri... Show more

 • Promoted

ML Eng TL, Behavior Planning - Lead Autonomous Driving

Australian Competition and Consumer CommissionMountain View, CA, United States
Full-time

A leading technology company in Mountain View, California, is seeking a Machine Learning Engineering Team Lead.This role involves defining architecture for planning models, mentoring junior enginee... Show more

 • Promoted

Software Engineer, LLM Compilation

OpenReqCupertino, CA, United States
Full-time

Etched is building AI chips that are hard-coded for individual model architectures.Our first product (Sohu) only supports transformers, but has an order of magnitude more throughput and lower laten... Show more

 • Promoted

Senior ML Engineer, Gmail Abuse & Safety Protections

GoogleSunnyvale, CA, United States
Full-time

Google is seeking an AI/ML Software Engineer to help shape the frontier of enterprise technology with a mission to enhance Gmail users' experience.In this role, you will leverage your expertise in ... Show more

 • Promoted

Remote Equity Research Analyst - AI Trainer ($50-$60 per hour)

Data AnnotationCapitola, California
$50.00 hourly
Remote
Full-time +1

DataAnnotation is committed to creating high-quality AI.Join our team to help train the next generation of AI while enjoying the flexibility of remote work and the freedom to set your own&nbsp... Show more

 • Promoted

Lead ML Engineer – LLM Training & Network Data

TeslaFremont, CA, United States
Full-time

Tesla is looking for a Senior Software Engineer in Fremont, California, to lead the development of a platform combining large-scale LLM training with advanced network data processing.The role deman... Show more

 • Promoted

Agile PLM Support Engineer

TechDigital GroupSanta Clara, CA, United States
Full-time

Guide the business in developing requirements that fit well with Agile PLMs out of the box capability.Elaborate and map business requirements into functional and technical solutions.Serve as an int... Show more

 • Promoted

LLM Training Infra Lead — Scale & Post-Training

LinkedInMountain View, CA, United States
Full-time

Software Engineering Manager in Mountain View, CA to lead the Post-Training Infra team responsible for architecting high-throughput systems for LLM training.The role involves overseeing sophisticat... Show more

 • Promoted

ML Engineer: LLM Research & Production

Apple Inc.Cupertino, CA, United States
Full-time

A leading technology company in Cupertino seeks a Research Scientist in AI to conduct groundbreaking research in deep learning and large language models.The ideal candidate will develop and fine-tu... Show more

 • Promoted

Multimodal LLM Engineer for Robotics & Autonomous Driving

XPENG & Volkswagen GroupSanta Clara, CA, United States
Full-time

A leading smart technology company seeks to hire a specialist to fine-tune pre-trained language models for Humanoid Robots and Autonomous Driving Cars.The role emphasizes strong Python skills and r... Show more

 • Promoted

Senior ML Serving Engineer for LLMs & Inference

AlldusSan Jose, CA, United States
Full-time

A tech company in AI/ML is seeking a Senior Software Engineer specializing in ML Serving to build robust infrastructure for ML models.The ideal candidate has 5+ years of experience in software engi... Show more

 • Promoted

Senior ML Evaluation Engineer for Personal Q&A at Scale

Apple Inc.Cupertino, CA, United States
Full-time

A leading technology company located in Cupertino is seeking an experienced Machine Learning Engineer to develop data generation methodologies and quality assessment systems.This role involves desi... Show more

 • Promoted

Staff Machine Learning Engineer, LLM Fine‑Tuning (Verilog/RTL Applications)

Highbrow Technology IncSan Jose, California, US
Full-time

Staff Machine Learning Engineer, LLM Fine‑Tuning (Verilog/RTL Applications).Check all associated application documentation thoroughly before clicking on the apply button at the bottom of this descr... Show more

 • Promoted

Foundational ML Researcher / Engineer - Remote, Equity

Pathway Genomics CorporationPalo Alto, CA, United States
Remote
Full-time

A cutting-edge AI startup is seeking R&D Engineers for groundbreaking work in attention-based machine learning models.Candidates should have a strong research background and experience in model... Show more

 • Promoted

Lead ML Engineer: Scale, Deploy & Validate Models

Intuit Inc.Mountain View, CA, United States
Full-time

A leading financial software firm is seeking a Staff Machine Learning Engineer to join their vibrant team in Mountain View, CA.The ideal candidate will have over 6 years of experience and strong kn... Show more

 • Promoted

Production AI/ML Engineer: LLMs, MLOps & Cloud

Cloud Hybrid Technologies, LLCSan Jose, CA, United States
Full-time

A leading AI solutions provider is seeking a Mid-Level AI/ML Engineer in San Jose, California.The ideal candidate will oversee the full ML lifecycle, design LLM-based solutions, and integrate cloud... Show more

 • Promoted

LLM Infrastructure Engineer - Ultra-Low Latency AI Chips

OpenReqCupertino, CA, United States
Full-time

A technology company in Cupertino is seeking a Software Engineer specialized in LLM infrastructure.You will build software to improve latency performance and work collaboratively with our teams to ... Show more

 • Promoted

LLM Inference Engineering Manager — Hybrid | Equity

NVIDIA CorporationSanta Clara, CA, United States
Full-time

A leading technology company in California is seeking an Engineering Manager to lead the development of cutting-edge LLM/VLM technologies.In this hands-on leadership role, you will manage a team re... Show more