AI Agent Evaluation Analyst

A company is looking for an AI Agent Evaluation Analyst (Freelance).

Key Responsibilities

Review evaluation tasks and scenarios for logic, completeness, and realism

Identify inconsistencies, missing assumptions, or unclear decision points

Help define clear expected behaviors (gold standards) for AI agents

Required Qualifications

Excellent analytical thinking regarding complex systems and logical implications

Familiarity with structured data formats (ability to read JSON / YAML)

Experience with policy evaluation, logic puzzles, or structured scenario design

Background in consulting, academia, or research

Exposure to LLMs, prompt engineering, or AI-generated content