Talent.com
Mercor
Remote Shopify Shop App Agent Annotator - AI Trainer ($40-$50 per hour)Mercor • Bellflower, California, US
No longer accepting applications
Remote Shopify Shop App Agent Annotator - AI Trainer ($40-$50 per hour)

Remote Shopify Shop App Agent Annotator - AI Trainer ($40-$50 per hour)

Mercor • Bellflower, California, US
7 days ago
Salary
$40.00 hourly
Job type
  • Full-time
  • Remote
Job description
## About the role Shopify's **Shop App Agent** is the AI assistant buyers talk to inside the Shop app — product discovery and search, orders, shipping and tracking, account questions, and Shop Cash. When a shopper is unhappy with one of its answers, they tap thumbs-down — and that's all they leave. No comment, no reason, no survey. Your job is to work out *why*, from the transcript alone. ## What you'll do - Read a full shopper ↔ agent conversation, including the tool calls the agent made and what came back - Find the specific turn that received the thumbs-down - Classify it with one primary tag — the *domain* of the complaint: a disliked recommendation, a failed order lookup, an action the agent couldn't take, a rejection of AI itself, or a genuinely broken response — and one specific secondary tag within it - Write a short comment: what the shopper wanted, what the agent did or failed to do, and why those tags fit - **Commitment:** ~5 hours/week ## What makes someone good at this The hardest part is not guessing. Plenty of thumbs-downs have no visible cause — the agent did nothing wrong, or the shopper simply didn't like the answer. There is an explicit `unknown` label for exactly that, and using it honestly matters more than producing a confident-sounding reason. We would rather record "we don't know why" than invent a cause that sends the wrong signal to the model. ## Qualifications **Required:** - **Rule discipline** — you can apply a fixed taxonomy the same way across hundreds of conversations, and you notice when a case sits between two labels rather than forcing it. - **Comfort reading structured traces** — tool calls, their inputs, and their raw outputs, so you can tell what the agent actually did from what it merely claimed. - **Restraint under ambiguity** — you are willing to label something `unknown` rather than reach for a plausible-sounding cause. - **Clear, concise written English** — enough to explain in two or three sentences why a conversation got the labels you gave it. **Strongly preferred:** - **Consumer e-commerce fluency** — orders, tracking, returns, refunds, and a working sense of where a merchant's responsibility ends and the platform's begins. - **Prior annotation, labelling, or model-evaluation work** against a defined rubric or taxonomy. **Preferred (nice to have — we'll ramp you on the specifics):** - Hands-on experience with the Shop app or other AI shopping assistants as a shopper. - Familiarity with customer-support operations and escalation paths. ## What this is not This is not a customer-support role and not an engineering role. You are not fixing the agent and not replying to shoppers — you are diagnosing why a real shopper was unhappy, precisely and repeatably, so the team can measure where the agent falls short.
Create a job alert for this search

Remote Shopify Shop App Agent Annotator - AI Trainer ($40-$50 per hour) • Bellflower, California, US

Similar jobs

Customer Account Manager - Tooling (Remote)

Howmet AerospaceAltadena, CA, United States
Remote
Full-time

Basic Qualifications :Bachelor's Degree and 3 years of experience in sales OR 5 years of experience in sales in lieu of a bachelor's degreeEmployees must be legally authorized to work in the United... Show more

 • Promoted

Video Editor (Remote) (Los Angeles)

Core Code ioAltadena, CA, United States
Remote
Full-time

A Video Editor's main task is to take the raw footage shot by a film crew and Director and turn it into the final product.Examples of duties a Video Editor may need to perform include :Assembling r... Show more

 • Promoted

Consumer Insights Analyst

Earn HausMount Wilson, California, United States
Full-time +1

We are urgently seeking people interested in taking market research studies for well known brands.If you are a self-starter, looking for flexible hours throughout the week, this may be for you! Ear... Show more

 • Promoted

Paid Online Survey Participant - No Experience

Clubshop | USAltadena, California
Full-time

Discover The System Surveoo That’s Helping Newbies Earn Daily.Turn your free time into daily earnings by answering simple surveys.No product or tech skills needed.Join For Free And Start Earn... Show more

BSA / AML Investigator, Associate II - Remote

Bank of the WestAltadena, CA, United States
Remote
Full-time

BSA / AML Investigator, Associate II - RemoteDescriptionAt Bank of the West, our people are having a positive impact on the world.We're investing where we feel we can make the most impact, like adv... Show more

 • Promoted

Senior Data Analyst | HR & Compensation Team (Remote Southern California)

Veritas Search GroupAltadena, CA, United States
Remote
Full-time

This role requires candidates who are currently authorized to work in the U.C2C arrangements are not accepted.While this role is primarily remote, you must live in Southern California due to occasi... Show more

 • Promoted

Work from home - Market Research Study

Earn HausValyermo, California, United States
Remote
Full-time +1

We are urgently seeking people interested in taking market research studies for well known brands.If you are a self-starter, looking for flexible hours throughout the week, this may be for you! Ear... Show more

 • Promoted

Work From Home Fast & Serious 100% Free - No experience

Clubshop | USAltadena, California
Remote
Full-time

Discover The System Surveoo That’s Helping Newbies Earn Daily.Turn your free time into daily earnings by answering simple surveys.No product or tech skills needed.Join For Free And Start Earn... Show more

Remote Product Tester - Flexible Studies

BuzzTestersAltadena, CA
Remote
Full-time

Earn up to $400/week, depending on the number and type of studies you complete.Share honest feedback on products and services from independent brands.Assignments include short online surveys (10–20... Show more

 • Promoted

Licensed Auto Insurance Agent- REMOTE

Los Angeles StaffingAltadena, CA, United States
Remote
Full-time

Licensed Auto Insurance Agent- RemoteBy joining Sedgwick, you'll be part of something truly meaningful.It's what our 33,000 colleagues do every day for people around the world who are facing the un... Show more

 • Promoted

Website Designer and Developer (remote)

Community Literature InitiativeAltadena, CA, United States
Remote
Full-time

Quick Overview :Community Literature Initiative (CLI) seeks a web developer to help with our website.You will oversee the functionality of the website, evaluate and manage website performance, faci... Show more

 • Promoted

Facebook Ads Specialist (Media Buyer) - REMOTE (Los Angeles)

Golden HippoAltadena, CA, United States
Remote
Full-time

We're an employee-owned, vertically-integrated builder of some of America's most popular health & wellness brands.With over 1,400 team members across multiple companies, we create best-in-class... Show more

 • Promoted

Project Manager

Resume-LibraryAltadena, CA, United States
Full-time

This is a test job please ignore.Thats whom we impact, and thats what we are building.Want to be a part of it ? To determine whether youd be a good fit, ask yourself:.Would you like to go to work e... Show more

 • Promoted

Website Depot is hiring: Senior Web Developer (Remote or Hybrid) in Los Angeles

Website DepotAltadena, CA, United States
Remote
Full-time

OverviewWe are a digital marketing agency based in Los Angeles, California.We are currently seeking a talented and experienced web developer to join our team for remote work.The preferred candidate... Show more

 • Promoted

Market Research Associate - No Experience

Clubshop | USAltadena, California
Full-time

Discover The System Surveoo That’s Helping Newbies Earn Daily.Turn your free time into daily earnings by answering simple surveys.No product or tech skills needed.Join For Free And Start Earn... Show more

Audit Senior Remote Home Based & Hybrid Work Schedule

HR Talent AllianceAltadena, CA, United States
Remote
Full-time

Audit Senior - Assurance and Advisory Services -Remote - Home Based / Hybrid Schedule AvaialbleWell established highly profitable regional CPA firm seeks an Audit Senior to specialize in exciting d... Show more

 • Promoted

Work From Home

Remote JobsAltadena, CA
Remote
Full-time

Remote Jobs are now available in your area.This position is a work from home position.You will not need to come into the office.Benefits: Health Insurance, 401K, Vacation& PTO.Apply now! Work f... Show more

 • Promoted

Backend Engineer (Multiple Roles, Remote & On-Site)

MLabsAltadena, CA, United States
Remote
Full-time

We're hiring Backend Engineers to join some of the most ambitious startups, including Basis, Conduit AI, Dashly, and Rilla.These teams are building distributed systems, data-heavy products, and rel... Show more

 • Promoted

Key Account Manager Beauty Professional (Remote)

HenkelAltadena, CA, United States
Remote
Full-time

At Henkel, you'll be part of an organization that's shaping thefuture through innovation, sustainability and collaboration.With our trustedbrands like Persil , 'all , Loctite , Snuggle , and Schwar... Show more

 • Promoted

Assistant Store Manager

Super King MarketAltadena, CA, United States
Full-time

Assistant Store Manager - 3rd Key.At Super King, we are shaping the international food market culture within the grocery industry.We are proud to be the leader of international foods at the lowest ... Show more