Talent.com
Freelance Agent Evaluation Analyst
Freelance Agent Evaluation AnalystMindrift • New York, NY, US
Freelance Agent Evaluation Analyst

Freelance Agent Evaluation Analyst

Mindrift • New York, NY, US
[job_card.variable_days_ago]
[job_preview.job_type]
  • [job_card.part_time]
  • [job_card.permanent]
  • [filters.remote]
  • [filters_job_card.quick_apply]
[job_card.job_description]

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.

At Mindrift , innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.

What we do

The Mindrift platform, launched and powered by Toloka , connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

Who we're looking for :

We’re looking for curious and intellectually proactive contributors, the kind of person who double-checks assumptions and plays devil’s advocate.

Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?

This is a flexible, project-based opportunity well-suited for :

  • Analysts, researchers, or consultants with strong critical thinking skills.
  • Students (senior undergrads / grad students) looking for an intellectually interesting gig.
  • People open to a part-time and non-permanent opportunity.

About the project :

We’re on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you’ll have to balance quality assurance, research, and logical problem-solving. This project opportunity is ideal for people who enjoy looking at systems holistically and thinking through scenarios, implications, and edge cases.

You do not need a coding background, but you must be curious, intellectually rigorous, and capable of evaluating the soundness and consistency of complex setups. If you’ve ever excelled in things like consulting, CHGK, Olympiads, case solving, or systems thinking — you might be a great fit.

What you’ll be doing :

  • Reviewing evaluation tasks and scenarios for logic, completeness, and realism.
  • Identifying inconsistencies, missing assumptions, or unclear decision points.
  • Helping define clear expected behaviors (gold standards) for AI agents.
  • Annotating cause-effect relationships, reasoning paths, and plausible alternatives.
  • Thinking through complex systems and policies as a human would to ensure agents are tested properly.
  • Working closely with QA, writers, or developers to suggest refinements or edge case coverage.

    • How to get started :

    Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone.

    Requirements

  • Excellent analytical thinking : Can reason about complex systems, scenarios, and logical implications.
  • Strong attention to detail : Can spot contradictions, ambiguities, and vague requirements.
  • Familiarity with structured data formats : Can read, not necessarily write JSON / YAML.
  • Ability to assess scenarios holistically : What's missing, what’s unrealistic, what might break?
  • Good communication and clear writing (in English) to document your findings.

    • We also value applicants who have :

  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design.
  • Background in consulting, academia, olympiads (e.g. logic / math / informatics), or research.
  • Exposure to LLMs, prompt engineering, or AI-generated content.
  • Familiarity with QA or test-case thinking (edge cases, failure modes, “what could go wrong”).
  • Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.).

    • Benefits

  • Get paid for your expertise, with  rates that can go up to $80 / hour  depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.
    • [job_alerts.create_a_job]

    Agent • New York, NY, US

    [internal_linking.similar_jobs]
    Remote Legal Expert - AI Trainer

    Remote Legal Expert - AI Trainer

    SuperAnnotate • Passaic, New Jersey, US
    [filters.remote]
    [job_card.full_time]
    In this hourly, remote contractor role, you will review AI-generated legal analyses and / or generate expert legal content, evaluating reasoning quality and step-by-step issue-spotting while providin...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Remote Side Hustle Evaluator - Flexible Online Gig Work

    Remote Side Hustle Evaluator - Flexible Online Gig Work

    Finance Buzz • Keansburg, New Jersey, US
    [filters.remote]
    [job_card.temporary]
    Are you looking to earn extra income from the comfort of your home? We're seeking motivated individuals to explore and test a variety of remote side hustle opportunities featured on FinanceBuzz.Thi...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Remote Investment Analyst – AI Trainer ($50-$60 / hour)

    Remote Investment Analyst – AI Trainer ($50-$60 / hour)

    Data Annotation • Hackensack, New Jersey
    [filters.remote]
    [job_card.full_time] +1
    We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the quality of ...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Analyst, MS&T

    Analyst, MS&T

    SK Life Science • Paramus, NJ, United States
    [job_card.full_time]
    Manage Contract Manufacturing Organization (CMO) in the following areas : commercial product manufacturing (Drug Substance, Drug Product & Packaging), and testing laboratories (Quality Control for ...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Data Analyst - Actions Analytics

    Data Analyst - Actions Analytics

    Zeta Global • New York, New York, United States
    [job_card.full_time]
    Zeta Global (NYSE : ZETA) is the AI-Powered Marketing Cloud that leverages advanced artificial intelligence (AI) and trillions of consumer signals to make it easier for marketers to acquire, grow, a...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Data Analyst - Claims & Loss Containment

    Data Analyst - Claims & Loss Containment

    Arlo • New York, New York, United States
    [job_card.full_time]
    Arlo is rebuilding health insurance from the ground up using AI.The healthcare experience today is expensive, confusing, and often so frustrating that people delay the care they need.We’re changing...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    AI Research Engineer, Enterprise Evaluations

    AI Research Engineer, Enterprise Evaluations

    Scale AI, Inc. • New York, NY, United States
    [job_card.full_time]
    Scale AI is seeking a technically rigorous and driven.This high-impact role is critical to our mission of delivering the industry's leading. You will be a hands-on contributor to the core systems th...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Online Survey Participant : Work Remote and Earn Up To $25 Per Survey

    Online Survey Participant : Work Remote and Earn Up To $25 Per Survey

    Earn Haus • Long Branch, NJ, US
    [filters.remote]
    [job_card.full_time] +1
    Looking for people to participate in taking online surveys for Fortune 500 brands.All you need to do is complete online surveys by sharing your opinion. You will help influence brand decisions on se...[show_more]
    [last_updated.last_updated_variable_hours] • [promoted] • [new]
    Data Analyst

    Data Analyst

    Confidential • New York, New York, United States
    [job_card.full_time]
    SundaySky has a unique and exciting opportunity to join an analytics group for a rapidly growing startup.Our product combines digital technology, advertising, consumer relationship management, and ...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Data Analyst

    Data Analyst

    Troveo Ai • New York, New York, United States
    [job_card.full_time]
    Troveo is building the next-generation data platform to train AI video models.We offer the world’s largest library of AI video training data, offering millions of hours of licensed, training-ready...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Forward Deployed Data Analyst

    Forward Deployed Data Analyst

    Valon Tech • New York, NY, United States
    [job_card.full_time]
    Valon is building the AI-native operating system for regulated finance, starting with mortgage servicing.We're a Series C company backed by a16z, transforming industries that others have written of...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Associate Data Analyst

    Associate Data Analyst

    Sagesure • Neptune City, New Jersey, United States
    [job_card.full_time]
    SageSure, a leader in catastrophe-exposed property insurance, is seeking a.This is a great opportunity for an early-career analyst to build hands-on experience with enterprise data systems, reporti...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Data Analyst

    Data Analyst

    Quality Talent Group • Town of Clarkstown, New York, United States
    [job_card.full_time]
    Our client is a leading force in advancing safer, smarter AI technology.Their work has been featured in.They have built a global community of expert contributors and have paid more than.No contract...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Senior Development Evaluation Analyst

    Senior Development Evaluation Analyst

    VirtualVocations • Bronx, New York, United States
    [job_card.full_time]
    Development Evaluation Analyst.Key Responsibilities Design and maintain dashboards, scorecards, KPIs, and reports using data visualization tools Gather business requirements and conduct advanced...[show_more]
    [last_updated.last_updated_variable_hours] • [promoted] • [new]
    Pricing Integrity Agent

    Pricing Integrity Agent

    RDSolutions formerly RetailData • Ossining, NY, US
    [job_card.part_time]
    The retail industry continues to see unprecedented dynamics as it pivots to a true omni-channel shopping experience.Informed retailers are succeeding, and RDSolutions is providing them with the con...[show_more]
    [last_updated.last_updated_variable_hours] • [promoted] • [new]
    Algebra Private Tutoring Jobs Middletown

    Algebra Private Tutoring Jobs Middletown

    Superprof • Middletown, New Jersey, US
    [job_card.full_time] +1
    Superprof is the leading tutoring platform in the USA, connecting passionate individuals with eager learners.We operate in over 41 countries, making education accessible to everyone, everywhere.Sup...[show_more]
    [last_updated.last_updated_variable_hours] • [promoted] • [new]
    Data Analyst / Junior Analyst (Remote – Entry-Level)

    Data Analyst / Junior Analyst (Remote – Entry-Level)

    GOLD GATE • Jersey City, New Jersey
    [filters.remote]
    [job_card.full_time]
    [filters_job_card.quick_apply]
    Data Analyst / Junior Analyst (Remote – Entry-Level).We are seeking a Remote Junior Data Analyst to assist with data review, reporting, and basic analysis to support business decisions.Collec...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Compensation Analyst

    Compensation Analyst

    Columbia University • New York, NY, United States
    [job_card.full_time]
    Job Type : Officer of Administration.Salary Range : $80,000 - $90,000.The salary of the finalist selected for this role will be set based on a variety of factors, including but not limited to departm...[show_more]
    [last_updated.last_updated_30] • [promoted]