Talent.com
Freelance Agent Evaluation Analyst
Freelance Agent Evaluation AnalystMindrift • New York, NY, US
Freelance Agent Evaluation Analyst

Freelance Agent Evaluation Analyst

Mindrift • New York, NY, US
[job_card.variable_days_ago]
[job_preview.job_type]
  • [job_card.part_time]
  • [job_card.permanent]
  • [filters.remote]
  • [filters_job_card.quick_apply]
[job_card.job_description]

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.

At Mindrift , innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI.

What we do

The Mindrift platform, launched and powered by Toloka , connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

Who we're looking for :

We’re looking for curious and intellectually proactive contributors, the kind of person who double-checks assumptions and plays devil’s advocate.

Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?

This is a flexible, project-based opportunity well-suited for :

  • Analysts, researchers, or consultants with strong critical thinking skills.
  • Students (senior undergrads / grad students) looking for an intellectually interesting gig.
  • People open to a part-time and non-permanent opportunity.

About the project :

We’re on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you’ll have to balance quality assurance, research, and logical problem-solving. This project opportunity is ideal for people who enjoy looking at systems holistically and thinking through scenarios, implications, and edge cases.

You do not need a coding background, but you must be curious, intellectually rigorous, and capable of evaluating the soundness and consistency of complex setups. If you’ve ever excelled in things like consulting, CHGK, Olympiads, case solving, or systems thinking — you might be a great fit.

What you’ll be doing :

  • Reviewing evaluation tasks and scenarios for logic, completeness, and realism.
  • Identifying inconsistencies, missing assumptions, or unclear decision points.
  • Helping define clear expected behaviors (gold standards) for AI agents.
  • Annotating cause-effect relationships, reasoning paths, and plausible alternatives.
  • Thinking through complex systems and policies as a human would to ensure agents are tested properly.
  • Working closely with QA, writers, or developers to suggest refinements or edge case coverage.

    • How to get started :

    Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone.

    Requirements

  • Excellent analytical thinking : Can reason about complex systems, scenarios, and logical implications.
  • Strong attention to detail : Can spot contradictions, ambiguities, and vague requirements.
  • Familiarity with structured data formats : Can read, not necessarily write JSON / YAML.
  • Ability to assess scenarios holistically : What's missing, what’s unrealistic, what might break?
  • Good communication and clear writing (in English) to document your findings.

    • We also value applicants who have :

  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design.
  • Background in consulting, academia, olympiads (e.g. logic / math / informatics), or research.
  • Exposure to LLMs, prompt engineering, or AI-generated content.
  • Familiarity with QA or test-case thinking (edge cases, failure modes, “what could go wrong”).
  • Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.).

    • Benefits

  • Get paid for your expertise, with  rates that can go up to $80 / hour  depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.
    • [job_alerts.create_a_job]

    Agent • New York, NY, US

    [internal_linking.similar_jobs]
    Technical Business Analyst

    Technical Business Analyst

    Relativity • New York, NY, United States
    [job_card.full_time]
    Relativity's Problem Management is seeking a Technical Business Analyst who excels at using data analytics to uncover trends in quality, client workflows, product performance and efficiency.Your in...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Data Analyst

    Data Analyst

    Share Local Media • New York, NY, United States
    [job_card.full_time]
    Share Local Media, or SLM, is a rapidly growing startup reimagining the world of offline marketing for tech and e-commerce companies. We started as e-commerce marketers ourselves, and launched SLM w...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Data Analyst

    Data Analyst

    Diverse Lynx • New York, NY, United States
    [job_card.full_time]
    Data Governance, Data Management, or Business Analysis (financial-services preferred).Familiarity with governance frameworks (DAMA, DCAM) and stewardship best practices. Working knowledge of metadat...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Netsuite Freelancer (NetSuite implementation experience required)

    Netsuite Freelancer (NetSuite implementation experience required)

    Maspeth Contracting • Maspeth, New York, United States
    [job_card.part_time]
    If the following job requirements and experience match your skills, please ensure you apply promptly.We are looking for an experienced NetSuite ERP Technical & Functional Freelancer to support the ...[show_more]
    [last_updated.last_updated_variable_hours] • [promoted] • [new]
    Application Analyst I - Epic BeaconEmpty heading

    Application Analyst I - Epic BeaconEmpty heading

    Memorial Sloan • New York, NY, United States
    [job_card.full_time]
    The people of Memorial Sloan Kettering Cancer Center (MSK) are united by a singular mission : ending cancer for life.Our specialized care teams provide personalized, compassionate, expert care to pa...[show_more]
    [last_updated.last_updated_1_day] • [promoted]
    Remote - Senior Data Analyst / Entry Level

    Remote - Senior Data Analyst / Entry Level

    Recruit Monitor • Yonkers, NY, United States
    [filters.remote]
    [job_card.full_time]
    About the job Remote - Senior Data Analyst / Entry Level.This role will work closely with teams across the organization, including product, growth, editorial, ad ops, and sales to develop actionabl...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Data Analyst

    Data Analyst

    Howden Group Holdings • New York, NY, United States
    [job_card.permanent]
    Howden is a global insurance group with employee ownership at its heart.Together, we have pushed the boundaries of insurance. We are united by a shared passion and no-limits mindset, and our strengt...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Energy Program Data Analyst

    Energy Program Data Analyst

    Equiliem • New York, NY, US
    [job_card.full_time]
    Energy Program Data Analyst (EV Programs).This role involves reviewing applications for eligibility, managing program documentation, analyzing charging data, and preparing incentive payment calcula...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    BIC Compliance Analyst

    BIC Compliance Analyst

    Cooley LLP • New York, NY, United States
    [job_card.full_time]
    Cooley is seeking a BIC Compliance Analyst to join the Business Intake and Conflicts team.The BIC Compliance Analyst is responsible for ensuring compliance with risk management policies related to ...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    AI Research Engineer, Enterprise Evaluations

    AI Research Engineer, Enterprise Evaluations

    Scale AI, Inc. • New York, NY, United States
    [job_card.full_time]
    Scale AI is seeking a technically rigorous and driven.This high-impact role is critical to our mission of delivering the industry's leading. You will be a hands-on contributor to the core systems th...[show_more]
    [last_updated.last_updated_30] • [promoted]
    M&A Analyst

    M&A Analyst

    NFP • New York, NY, United States
    [job_card.full_time]
    NFP, an Aon company, is a multiple Best Places to Work award winner in Business Insurance.We are an organization of consultative advisors and problem solvers. We help companies and individuals aroun...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Forward Deployed Data Analyst

    Forward Deployed Data Analyst

    Supper • New York, NY, United States
    [job_card.full_time]
    Supper is an AI-native data platform that makes your company's information as easy to use as a conversation.Business teams at most high-growth companies are held back by data request backlogs, over...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Remote Investment Analyst – AI Trainer ($50-$60 / hour)

    Remote Investment Analyst – AI Trainer ($50-$60 / hour)

    Data Annotation • Passaic, New Jersey
    [filters.remote]
    [job_card.full_time] +1
    We are looking for a finance professional to join our team to train AI models.You will measure the progress of these AI chatbots, evaluate their logic, and solve problems to improve the quality of ...[show_more]
    [last_updated.last_updated_30] • [promoted]
    M&A Analyst

    M&A Analyst

    National Financial Partners • New York, NY, United States
    [job_card.full_time]
    NFP, an Aon company, is a multiple Best Places to Work award winner in Business Insurance.We are an organization of consultative advisors and problem solvers. We help companies and individuals aroun...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Compliance Analyst

    Compliance Analyst

    Zenex Partners • New York, NY, US
    [job_card.full_time]
    Provide Compliance Analyst services in New York, New York, United States learn more about this role and apply.[show_more]
    [last_updated.last_updated_30] • [promoted]
    EPIC APPLICATIONS ANALYST

    EPIC APPLICATIONS ANALYST

    New Bridge Medical Center • Paramus, NJ, United States
    [job_card.full_time]
    Join Our Team at New Bridge Medical Center! • •.We are dedicated to providing high-quality, compassionate care to our diverse community. As a leading healthcare provider, we offer a supportive and inc...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Compensation Analyst

    Compensation Analyst

    Columbia University • New York, NY, United States
    [job_card.full_time]
    Job Type : Officer of Administration.Salary Range : $80,000 - $90,000.The salary of the finalist selected for this role will be set based on a variety of factors, including but not limited to departm...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Analyst

    Analyst

    NYC Staffing • New York, NY, US
    [job_card.full_time]
    Heidrick & Struggles is the world's foremost advisor on executive leadership, driving superior client performance through premier human capital leadership advisory services.For more than 70 years, ...[show_more]
    [last_updated.last_updated_30] • [promoted]