## About the role Shopify's **Shop App Agent** is the AI assistant buyers talk to inside the Shop app — product discovery and search, orders, shipping and tracking, account questions, and Shop Cash. When a shopper is unhappy with one of its answers, they tap thumbs-down — and that's all they leave. No comment, no reason, no survey. Your job is to work out *why*, from the transcript alone. ## What you'll do - Read a full shopper ↔ agent conversation, including the tool calls the agent made and what came back - Find the specific turn that received the thumbs-down - Classify it with one primary tag — the *domain* of the complaint: a disliked recommendation, a failed order lookup, an action the agent couldn't take, a rejection of AI itself, or a genuinely broken response — and one specific secondary tag within it - Write a short comment: what the shopper wanted, what the agent did or failed to do, and why those tags fit - **Commitment:** ~5 hours/week ## What makes someone good at this The hardest part is not guessing. Plenty of thumbs-downs have no visible cause — the agent did nothing wrong, or the shopper simply didn't like the answer. There is an explicit `unknown` label for exactly that, and using it honestly matters more than producing a confident-sounding reason. We would rather record "we don't know why" than invent a cause that sends the wrong signal to the model. ## Qualifications **Required:** - **Rule discipline** — you can apply a fixed taxonomy the same way across hundreds of conversations, and you notice when a case sits between two labels rather than forcing it. - **Comfort reading structured traces** — tool calls, their inputs, and their raw outputs, so you can tell what the agent actually did from what it merely claimed. - **Restraint under ambiguity** — you are willing to label something `unknown` rather than reach for a plausible-sounding cause. - **Clear, concise written English** — enough to explain in two or three sentences why a conversation got the labels you gave it. **Strongly preferred:** - **Consumer e-commerce fluency** — orders, tracking, returns, refunds, and a working sense of where a merchant's responsibility ends and the platform's begins. - **Prior annotation, labelling, or model-evaluation work** against a defined rubric or taxonomy. **Preferred (nice to have — we'll ramp you on the specifics):** - Hands-on experience with the Shop app or other AI shopping assistants as a shopper. - Familiarity with customer-support operations and escalation paths. ## What this is not This is not a customer-support role and not an engineering role. You are not fixing the agent and not replying to shoppers — you are diagnosing why a real shopper was unhappy, precisely and repeatably, so the team can measure where the agent falls short.
Remote Shopify Shop App Agent Annotator - AI Trainer ($40-$50 per hour) • East Lansing, Michigan, US
Looking for people to participate in taking online surveys for Fortune 500 brands.All you need to do is complete online surveys by sharing your opinion.You will help influence brand decisions on se... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
How about TODAY being the day you take back your life and secure your future? .The finance industry makes more millionaires than any other industry in the World!.My name is Beau, founder and owner ... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Review and label digital content including text, images, and documents.Every task you complete helps improve how technology interprets information and performs in practical settings.Detail-oriented... Show more
Join our distributed team and help evaluate online platforms through interactive testing assignments.This is a flexible, remote opportunity with full training provided and no prior experience requi... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Registered Nurse (RN) | Obstetrics/Gynecology.Ready to start your next travel adventure? LRS Healthcare offers a full benefits package, 24/7 support, and a responsive, traveler-first culture.What a... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home in the UK to fulfill upcoming contracts with local and international companies.We guarantee 15-25 hours per week with an hourly pay of between £18/hr.Th... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies.We guarantee 15-25 hours per week with an hourly pay of bet... Show more
Account Manager - State Farm Agent Team Member with Al Shattuck - State Farm Agent.As a Account Manager - State Farm Agent Team Member with Al Shattuck - State Farm Agent, you will play a key role ... Show more
