Talent.com
Mindrift
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)Mindrift • New York, NY, US
Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Senior Python Engineer - AI Coding Agent Evaluation (Freelance)

Mindrift • New York, NY, US
16 days ago
Job type
  • Part-time
  • Permanent
  • Remote
  • Quick Apply
Job description

Please submit your CV in English and indicate your level of English proficiency.


Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not permanent employment.What this opportunity involves:
We're building a dataset to evaluate AI coding agents - how well a model handles real-world developer tasks.


You'll create challenging tasks and evaluation criteria within realistic simulated environments:

  • Build realistic developer environments - a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history
  • Design tasks from intermediate states of these environments - craft the prompt, define what "solved" means, and ensure the task is solvable by an AI agent
  • Write tests that verify agent solutions - accept all valid approaches and reject incorrect ones, neither too strict nor too lenient
  • Iterate on tasks and tests based on QA feedback - review agent solutions, analyze failures, and refine until the evaluation is fair and robust

What this is NOT:

  • Not data labeling
  • Not prompt engineering
  • Not writing code from scratch - the agent writes most of the code; you guide and evaluate

What we look for:

  • 8+ years in software development
  • Core stack: Python (FastAPI), JavaScript/TypeScript (React), Docker, Postgres, Kafka, Redis
  • Experience writing tests (functional, integration)
  • English proficiency - B2+

Why this is hard:
Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions - writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.

How it works
Apply → Pass qualification(s) → Join a project → Complete tasks → Get paidEffort estimate


Tasks for this project are estimated to take 30 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.

Compensation:
Up to $200/hr equivalent, depending on level and pace. Tasks are estimated at ~30 hours each; you set your own schedule.

Create a job alert for this search

Senior Python Engineer - AI Coding Agent Evaluation (Freelance) • New York, NY, US

Similar jobs

Senior Software Engineer AI-Driven Hiring Platform (Remote)

RippleMatchNew York City, NY, United States
Remote
Full-time

A leading recruitment platform is seeking a Senior Software Engineer to join their innovative team focused on building an AI-driven hiring platform.This position involves writing high-quality code,... Show more

 • Promoted

Remote Physics Expert (PhD) - $100+ per hour

Turing Eatontown, New Jersey
Remote
Full-time
Quick Apply

Remote contract for PhDs in Physics, Applied Physics, or related fields.Work on cutting-edge projects with top AI labs while earning $100+/hour, fully remote, with flexible weekly hours.Help fine-t... Show more

 • Promoted

Data Engineer / Python Developer / Remote

Apetan Consulting llcNew Milford, NJ, United States
Remote
Full-time

Key ResponsibilitiesDesign, build, and maintain ETL / ELT data pipelinesDevelop Python-based data processing applicationsWork with structured and unstructured data at scaleIntegrate data from multi... Show more

 • Promoted

Remote AI Full-Stack Engineer: From ML to UI

BjakNew York City, NY, United States
Remote
Full-time

A fast-growing AI-driven financial services company is seeking an AI Developer to work remotely from anywhere in the US.This role involves full-stack development to bridge machine learning and soft... Show more

 • Promoted

Earn up to $400 per Day by Playing Games and TakingSurveys

Freecash.comLong Branch, New Jersey, United States
Full-time

Since its launch 6 years ago, over 60MN users have earned and withdrawn over $250MN! The platform is rated 4.TrustPilot with over 230k+ reviews, establishing Freecash as one of the highest-rated op... Show more

 • Promoted

Lead Machine Learning Engineer (Gen AI, Python, Go, AWS)

Capital oneNew York City, NY, United States
Full-time +1

Lead Machine Learning Engineer (Gen AI, Python, Go, AWS)As a Capital One Machine Learning Engineer (MLE) on the GenAI Workflows Serving team, you'll be part of an Agile team dedicated to designing,... Show more

 • Promoted

Senior Full Stack Engineer (AI / Voice AI)

Resourceful Talent GroupNew York, NY, US
$200,000.00 yearly
Full-time

Our client, a rapidly growing AI startup based in New York City, is seeking a.AI-powered voice agents transforming how B2C businesses engage with customers.The company develops AI solutions that au... Show more

Tech Lead (AI)

VeracityNew York City, NY, United States
Full-time

Role: Tech Lead (AI)Department: Wealth ManagementLocation: NYC, NY & Alpharetta, GA (3 days onsite in a week)Position Type: FulltimeNote- Look for visa independent only.Position OverviewWe are ... Show more

 • Promoted

Remote Mathematics Expert (PhD) - $100+ per hour

Turing Eatontown, New Jersey
Remote
Full-time
Quick Apply

Remote contract for PhDs in Mathematics, Statistics, or related fields.Work on cutting-edge projects with top AI labs while earning $100+/hour, fully remote, with flexible weekly hours.Help fine-tu... Show more

 • Promoted

Consumer Research Associate (Remote)

GL IncRed Bank, New Jersey
$15.00 hourly
Remote
Part-time +1

Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more

 • Promoted

Senior Python Engineer - Backend & Data (Remote)

Bold BusinessNew York, NY, US
Remote
Full-time

Bold Business is an AI-first U.AI agents, and production software for clients like AT&T, JP Morgan, and Verizon.We ship real systems that deliver impact — not experiments.Senior Python Engineer who... Show more

Senior Principal Consultant-Python FS + Gen AI Developer

GenpactNew York City, NY, United States
Full-time

Ready to build the future with AI?At Genpact, we don't just keep up with technology-we set the pace.AI and digital innovation are redefining industries, and we're leading the charge.Genpact's AI Gi... Show more

 • Promoted

Senior Data Engineer-AI/BI Analytics

Bank of AmericaJersey City, NJ, United States
Full-time

Bank Of America Gen Ai Inferencing CapabilitiesAt Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection.We do this by driving ... Show more

 • Promoted

Senior Backend Engineer - AI-Driven Risk (Remote)

BitSightNew York City, NY, United States
Remote
Full-time

A technology leader in cyber risk management is seeking a Senior Software Engineer to join their team.This role focuses on backend development, requiring expertise in Python and experience with RES... Show more

 • Promoted

Python Software Engineer - Remote

S&P GlobalNew York City, NY, United States
Remote
Full-time

About the RoleGrade Level (for internal use) :S&P Global Automotive InsightsJob title :Software EngineerLocation :Mexico - Remote (Only candidates based out within Mexico will be considered)The... Show more

 • Promoted

Remote Chemistry Expert (PhD) - $100+ per hour

Turing Tinton Falls, New Jersey
Remote
Full-time
Quick Apply

Remote contract for PhDs in Chemistry, Chemical Engineering, or related fields.Work on cutting-edge projects with top AI labs while earning up to $100+/hour, fully remote, with flexible weekly hour... Show more