AI Agent Testing SpecialistVirtualVocations • Austin, Texas, United States
AI Agent Testing Specialist
VirtualVocations • Austin, Texas, United States
[job_card.variable_days_ago]
[job_preview.job_type]
- [job_card.full_time]
[job_card.job_description]
A company is looking for an Evaluation Scenario Writer - AI Agent Testing Specialist.
Key Responsibilities
Create structured test cases that simulate complex human workflows
Define gold-standard behavior and scoring logic to evaluate agent actions
Analyze agent logs, failure modes, and decision paths
Required Qualifications
Bachelor's and / or Master's Degree in relevant fields such as Computer Science or Data Science
Background in QA, software testing, data analysis, or NLP annotation
Good understanding of test design principles
Comfortable with structured formats like JSON / YAML for scenario description
Basic experience with Python and JS
[job_alerts.create_a_job]
Testing Specialist • Austin, Texas, United States
[internal_linking.similar_jobs]
At Cloudera, we empower people to transform complex data into clear and actionable insights.With as much data under management as the hyperscalers, we're the preferred data partner for the top comp...[show_more]
Shape the future of AI infrastructure! Join AWS as we scale our Manufacturing Test Engineering team developing comprehensive testing solutions for our innovative AI acceleration hardware, deployed ...[show_more]
Cross Screen Media empowers political and public affairs organizations to plan, activate, and measure video advertising campaigns.
We combine proprietary software and best-in-class campaign executio...[show_more]
At Oracle Analytics, we are building the next generation of enterprise AI products to enable intelligent data analysis at scale.
Leveraging our foundational strengths in data management and enterpri...[show_more]
Develop pipelines to parse documents, chunk, vectorise, and store vector data.Design schemas for vector stores to optimise retrieval efficiency.
Create prompt templates using advanced prompt enginee...[show_more]
Join our expert bench — fractional CTO advisors wanted (hourly).Title : Fractional CTO / AI Strategy Council — Remote (Contractor).
Location : Remote - Global (offshore-friendly to Western Europe, U.C...[show_more]
Humata is a seed-stage startup based in Austin, Texas who was first to market with our Chat with Documents app in early 2023.
We are now on a mission to build the best document- and knowledgebase-an...[show_more]
This role requires a mandatory in person interview at our headquarters in Austin, Texas during the interview process, no exceptions.
SailPoint is the leader in identity security for the cloud enterp...[show_more]
Generative Search Engine Optimization Manager.We're hiring a senior AI Search Engine Optimization manager to drive our shift from traditional SEO to AI-first content discovery.You'll design, build,...[show_more]
Through its robust suite of tools, Realtor.Join us on our mission to empower more people to find their way home by breaking barriers to entry, making the right connections, and building confidence ...[show_more]
Tesla's AI Hardware team is looking for a Design-for-Test (DFT) Design Verification Engineer to work on custom ASICs.You will drive state of the art in the area of testability and contribute to pro...[show_more]![Generative AI Engineer [5918]](https://cdn-logos.talent.com/v1/logo/image?feedcode=lensa-6-api&company_name=Skyelia)
Develop pipelines to parse documents, chunk, vectorize, and persist vector stores.Develop prompt templates with various prompt engineering techniques to interact with LLMs.Experience in prompting w...[show_more]
At Teradata, we believe that people thrive when empowered with better information.That's why we built the most complete cloud analytics and data platform for AI.
By delivering harmonized data, trust...[show_more] The Customer Success Services team needs an AI Engineer that can design, develop, configure, evaluate, monitor and optimize generative AI and agent solutions for Oracle applications, as well as dev...[show_more] 
Primary skills : AI, ML, Generative AI, GitHub CoPilot, Prompt Engineering,.We are looking for an experienced AI Engineer with a strong background Artificial Intelligence, Machine Learning, Generati...[show_more]
Through its robust suite of tools, Realtor.Join us on our mission to empower more people to find their way home by breaking barriers to entry, making the right connections, and building confidence ...[show_more]
Technology|Cloud Platform|Google Big Data.Software Development Life Cycle including Requirements Elicitation, Application Architecture definition and Design.
You will play an important role in creat...[show_more]
Through its robust suite of tools, Realtor.Join us on our mission to empower more people to find their way home by breaking barriers to entry, making the right connections, and building confidence ...[show_more]
Principal Product Manager - Agentic AI Platform
Cloudera • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Systems Test Engineer - AI Hardware Manufacturing, Annapurna Labs
Amazon • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
AI Engineer
Cross Screen Media • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior Research Engineer - Agentic AI
Oracle • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Generative AI Engineer
Macpower Digital Assets Edge • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Fractional CTO / AI Strategy Council - Remote (Contractor)
Shae Group • Austin, TX, US
[filters.remote]
[job_card.full_time]
[filters_job_card.quick_apply]
[last_updated.last_updated_variable_days]
AI Engineer
Humata • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
AI Engineer
SailPoint Technologies Holdings, Inc. • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Generative Search Engine Optimization Manager
Unity Technologies • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_hours] • [promoted] • [new]
Senior AI Solutions Engineer
News Corp. Class A • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Sr. DFT Verification Engineer, AI Hardware
Tesla • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Generative AI Engineer [5918]
Skyelia • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
AI Engineer
Teradata • Austin, TX, United States
[job_card.permanent]
[last_updated.last_updated_variable_days] • [promoted]
AI Engineer
Oracle • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
AI Engineer
My3Tech Inc • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior AI Solutions Engineer
Realtor.com Careers • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Google ADK AI Engineer
Infosys Limited Digital • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior AI Solutions Engineer
realtor.com • Austin, TX, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]