Talent.com
Evaluation Scenario Writer - AI Agent Testing Specialist
Evaluation Scenario Writer - AI Agent Testing SpecialistMindrift • New York, NY, US
Evaluation Scenario Writer - AI Agent Testing Specialist

Evaluation Scenario Writer - AI Agent Testing Specialist

Mindrift • New York, NY, US
[job_card.30_days_ago]
[job_preview.job_type]
  • [job_card.part_time]
  • [filters.remote]
  • [filters_job_card.quick_apply]
[job_card.job_description]

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.

At Mindrift , innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.

What we do

The Mindrift platform, launched and powered by Toloka , connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

About the Role

We’re looking for someone who can design realistic and structured evaluation scenarios for LLM-based agents. You’ll create test cases that simulate human-performed tasks and define gold-standard behavior to compare agent actions against. You’ll work to ensure each scenario is clearly defined, well-scored, and easy to execute and reuse. You’ll need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Although every project is unique, you might typically :

  • Designing structured test scenarios based on real-world tasks.
  • Defining the golden path and acceptable agent behavior.
  • Annotating task steps, expected outputs, and edge cases.
  • Working with devs to test your scenarios and improve clarity.
  • Reviewing agent outputs and adapting tests accordingly

How to get started

Simply apply to this post, qualify, and get the chance to contribute to projects aligned with your skills, on your own schedule. From creating training prompts to refining model responses, you’ll help shape the future of AI while ensuring technology benefits everyone.

Requirements

  • Bachelor's and / or Master’s Degreein Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
  • Background in QA, software testing, data analysis, or NLP annotation.
  • Good understanding of test design principles (e.g., reproducibility, coverage, edge cases).
  • Strong written communication skills in English.
  • Comfortable with structured formats like JSON / YAML for scenario description.
  • Can define expected agent behaviors (gold paths) and scoring logic.
  • Basic experience with Python and JS.
  • Curious and open to working with AI-generated content, agent logs, and prompt-based behavior.
  • You are ready to learn new methods, able to switch between tasks and topics quickly and sometimes work with challenging, complex guidelines.
  • Our freelance role is fully remote so, you just need a laptop, internet connection, time available and enthusiasm to take on a challenge.

    • Nice to Have

  • Experience in writing manual or automated test cases.
  • Familiarity with LLM capabilities and typical failure modes.
  • Understanding of scoring metrics (precision, recall, coverage, reward functions).

    • Benefits

    Contribute on your own schedule, from anywhere in the world. This opportunity allows you to :

  • Get paid for your expertise, with  rates that can go up to $60 / hour  depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.
    • [job_alerts.create_a_job]

    Evaluation Testing • New York, NY, US

    [internal_linking.related_jobs]
    AI Engineer - NYC

    AI Engineer - NYC

    Aegis Ai • New York, New York, United States
    [job_card.full_time]
    We’re a team of ex-Google engineers who built some of the largest defensive platforms on the planet —.Now, we’re striking out on our own to tackle an even bigger challenge : stopping the new wave of...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Remote M&A Associate - AI Trainer ($50-$60 / hour)

    Remote M&A Associate - AI Trainer ($50-$60 / hour)

    Data Annotation • Clifton, New Jersey
    [filters.remote]
    [job_card.full_time] +1
    We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the quality of ...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    AI Platform Engineer IV

    AI Platform Engineer IV

    Capital Group • New York, NY, United States
    [job_card.full_time]
    I can succeed as an AI Platform Engineer at Capital Group.As an AI Platform Engineer, you will design, build, and operate the foundational components of Capital Group's enterprise AI platform, enab...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Applied AI Engineer - Agent

    Applied AI Engineer - Agent

    The General Intelligence Company Of New York • New York, NY, United States
    [job_card.full_time]
    We’re hiring an Applied AI Engineer to push the boundaries of our Cofounder agent.You’ll own core backend systems and applied LLM work : advancing agent reliability and autonomy, building evaluation...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    AI Engineer - Site Reliability Researcher

    AI Engineer - Site Reliability Researcher

    Traversal • New York, NY, United States
    [job_card.full_time]
    AI Site Reliability Engineer (SRE) for the enterprise—already trusted by some of the largest companies in the world to troubleshoot, remediate, and even prevent the most complex production incident...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Applied AI Engineer

    Applied AI Engineer

    Thread Ai • New York, NY, United States
    [job_card.full_time] +1
    Thread AI is focused on building an AI-native workflow orchestration engine looking for dedicated individuals to join its growing team. Our goal is to make infrastructure simple for enterprises and ...[show_more]
    [last_updated.last_updated_30] • [promoted]
    AI Engineer (New York)

    AI Engineer (New York)

    Finster Ai • New York, NY, United States
    [job_card.full_time]
    Compensation : $120,000-$200,000 + Equity .We’re a Series A stage firm, redefining the future of finance with our AI-native research and task automation platform, backed by leading, global venture i...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    AI Research Engineer, Enterprise Evaluations

    AI Research Engineer, Enterprise Evaluations

    Scale AI, Inc. • New York, NY, United States
    [job_card.full_time]
    Scale AI is seeking a technically rigorous and driven.This high-impact role is critical to our mission of delivering the industry's leading. You will be a hands-on contributor to the core systems th...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    AI Engineer - Agents

    AI Engineer - Agents

    Traversal • New York, NY, United States
    [job_card.full_time]
    AI Site Reliability Engineer (SRE) for the enterprise—already trusted by some of the largest companies in the world to troubleshoot, remediate, and even prevent the most complex production incident...[show_more]
    [last_updated.last_updated_30] • [promoted]
    AI Engineer

    AI Engineer

    Zone It Solutions • New York, New York, United States
    [job_card.full_time]
    We are on the lookout for an innovative and driven.In this role, you will be responsible for designing, developing, and deploying AI models that will enhance our products and improve our services.B...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Senior AI Engineer

    Senior AI Engineer

    OneStream Software • New York, NY, United States
    [job_card.full_time]
    Senior AI Engineer (REACT / Front End).Hybrid, USA (Birmingham, MI, Chicago, IL, New York, NY, Fort Lauderdale, FL).Range applies to US candidates only) + Benefits / Variable Comp / Equity - Range may va...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Applied AI Research Engineer

    Applied AI Research Engineer

    Norm Ai • New York, New York, United States
    [job_card.full_time]
    Norm Ai is the Compliance AI Platform for legal standards-based reasoning & workflow automation.We developed the first Domain Specific Language (DSL) for fully representing regulatory requirements ...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Generative AI Engineer

    Generative AI Engineer

    Regard • New York, New York, United States
    [job_card.full_time]
    As a Generative AI Engineer at Regard, you’ll work across the full lifecycle of developing and deploying AI-driven features, from ideation and design to prototyping, implementation, evaluation, and...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Remote Market Research Evaluator (Hiring Immediately)

    Remote Market Research Evaluator (Hiring Immediately)

    Maxion Research • Bradley Gardens, New Jersey, US
    [filters.remote]
    [job_card.part_time]
    Join Our Part-Time Research Studies Today!.Are you eager to earn extra income from the comfort of your home? We are looking for motivated individuals to participate in paid research studies includi...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Forward Deployed AI Engineer

    Forward Deployed AI Engineer

    Palantir Technologies • New York, New York, United States
    [job_card.full_time]
    Palantir builds the world’s leading software for data-driven decisions and operations.By bringing the right data to the people who need it, our platforms empower our partners to develop lifesaving ...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Applied AI, Technical Lead, Forward Deployed AI Engineer - NYC

    Applied AI, Technical Lead, Forward Deployed AI Engineer - NYC

    Mistral Ai • New York, NY, United States
    [job_card.full_time]
    At Mistral AI, we believe in the power of AI to simplify tasks, save time, and enhance learning and creativity.Our technology is designed to integrate seamlessly into daily working life.We democrat...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Applied AI Engineer

    Applied AI Engineer

    Tandem • New York, New York, United States
    [job_card.full_time]
    Tandem is a generational opportunity to rethink how we bring new therapies to market, and our path to doing so is significantly de-risked – we have : . We have product-market fit and are growing rapid...[show_more]
    [last_updated.last_updated_30] • [promoted]
    AI researcher

    AI researcher

    Graphite • New York, New York, United States
    [job_card.full_time]
    Graphite builds consumer-quality tools for modern software engineering teams, so they can ship faster and create amazing products. Anyone can start using Graphite individually without needing their ...[show_more]
    [last_updated.last_updated_30] • [promoted]