Talent.com
Senior AI Engineer - APM Experiences
Senior AI Engineer - APM ExperiencesDatadog • Syracuse, New York, United States
Senior AI Engineer - APM Experiences

Senior AI Engineer - APM Experiences

Datadog • Syracuse, New York, United States
[job_card.variable_days_ago]
[job_preview.job_type]
  • [job_card.full_time]
[job_card.job_description]

The opportunity

Datadog’s APM Experiences team owns the core product experience for Application Performance Monitoring — including distributed tracing, service representation, and more. We’re building a new wave of AI-powered capabilities that help customers detect, resolve, and prevent performance issues faster. In this role, you will lead end‑to‑end development of LLM- and Agent‑based features that can :

  • Debug and investigate application performance issues down to the root cause, as both a developer assistant and a fully autonomous agent
  • Proactively recommend performance and reliability-based optimizations to prevent the next incident
  • Automatically create intelligent monitors and SLOs for the most important business flows and critical paths

This is a highly product‑minded engineering role : you’ll work from problem discovery and UX all the way to reliable, scalable production systems.

What you’ll do

  • Shape AI experiences for APM. Design and ship LLM / agentic workflows that analyze traces, metrics, logs, and other telemetry to generate diagnoses, explanations, and guided fixes.
  • Own the full loop. Prototype quickly, define success metrics and evals, run experiments, iterate, and ultimately productionize for scale and reliability.
  • Build robust agent systems. Develop tools, retrieval and planning strategies, and guardrails; manage prompts / evals; design fallbacks and human‑in‑the‑loop paths.
  • Integrate with Datadog’s platform. Leverage surfaces like Trace Explorer, Service Catalog, monitors, and workflows to deliver end‑to‑end value in the APM UI.
  • Partner deeply. Collaborate with PM, Design, and partner teams to build cohesive experiences.
  • Raise the bar on engineering. Write performant, maintainable backend code, own services in production, and improve reliability for high‑throughput, low‑latency data systems.

    • Who you are

    Product‑minded engineer who ships AI to production

  • 4+ years building backend or real-time ML systems; you value simplicity, correctness, and performance
  • Proven experience delivering LLM / agent features to production (prompting, tooling, evals, safety / guardrails)
  • Comfortable owning user journeys, iterating from prototype → alpha → GA, and measuring impact with clear product metrics

    • Strong ML / applied science fundamentals

  • Solid grasp of the ML lifecycle (task definition, dataset collection, modeling, evaluation, deployment, iteration) and statistics (experiment design, confidence intervals)
  • Experience choosing / modeling the right technique for the job (e.g., anomaly detection, ranking / recommendation, NLP), and knowing when a heuristic beats a model
  • Fluency with offline / online evals for AI systems; can build reliable golden sets and automatic regressions

    • Distributed systems & observability savvy

  • Experience with microservices performance : tracing, latency breakdowns, concurrency, and resiliency patterns
  • Proficient in Go, Java, or Python; strong API / service design; production ops (monitoring, alerting, on‑call rotation)

    • Nice to have

  • Hands‑on with distributed tracing stacks (OpenTelemetry / Datadog APM), profilers, and logs / metrics pipelines
  • Exposure to planning / agent frameworks, tool‑use orchestration, RAG, and retrieval / indexing for observability data
  • Familiarity with SLO / SLA practices and incident response

    • Benefits and Growth :

  • Get to build tools for software engineers, just like yourself. And use the tools we build to accelerate our development.
  • Have a lot of influence on product direction and impact on the business.
  • Work with skilled, knowledgeable, and kind teammates who are happy to teach and learn.
  • Competitive global benefits.
  • Continuous professional development.

    • Benefits and Growth listed above may vary based on the country of your employment and the nature of your employment with Datadog.

    [job_alerts.create_a_job]

    Senior Engineer Ai • Syracuse, New York, United States

    [internal_linking.related_jobs]
    Hiring Now - Work from Home - No Experience

    Hiring Now - Work from Home - No Experience

    OCPA • DeRuyter village, New York, us
    [filters.remote]
    [job_card.part_time] +1
    Product Testers are wanted to work from home nationwide in the US to fulfill upcoming contracts with national and international companies. We guarantee 15-25 hours per week with an hourly pay of bet...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Remote PK / PD Modeling / Pharmacometrics Lead - AI Trainer ($150-$200 per hour)

    Remote PK / PD Modeling / Pharmacometrics Lead - AI Trainer ($150-$200 per hour)

    Mercor • Syracuse, New York, US
    [filters.remote]
    [job_card.part_time]
    This person complements the client’s “Translational / Clinical Pharmacology Decision-Maker” team by grounding dose selection and exposure–response analysis in • •quantitative structure and parameter...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Sales Engineer

    Sales Engineer

    SMV Recruiting • Syracuse, NY, US
    [job_card.full_time]
    [filters_job_card.quick_apply]
    SMV Recruiting is partnering with a smaller yet thriving engineering firm that acts as a manufacturer's representative supporting the defense and aerospace engineering field in Upstate, NY.Our clie...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Backend Data Engineer

    Backend Data Engineer

    Artemis • Syracuse, New York, United States
    [job_card.full_time]
    We believe that the global financial system is being built on blockchains, and Artemis will power the new financial system. Artemis is the analytics layer supercharging liquid token funds, asset ma...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Neuroscience Advanced Practice Provider

    Neuroscience Advanced Practice Provider

    SUNY Upstate Medical University • Syracuse, NY, United States
    [job_card.full_time]
    The mission of SUNY Upstate Medical University is to improve the health of the communities we serve through Education, Biomedical Research, and Health Care. Beyond our downtown campus, we provide ca...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Senior Software Engineer - AI Code Gen - Agent Engineer

    Senior Software Engineer - AI Code Gen - Agent Engineer

    Datadog • Syracuse, New York, United States
    [filters.remote]
    [job_card.full_time]
    We’re a new team building AI-assisted tools to make Datadog developers more effective, by autonomously generating tests, fixing bugs, and improving performance. We’re looking for a product-minded ge...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Senior Data Engineer

    Senior Data Engineer

    Perchwell • Syracuse, New York, United States
    [job_card.full_time]
    Perchwell is the modern real estate listings platform built for agents to search smarter, collaborate better, and close deals faster. We are building the future of residential real estateʼs critical...[show_more]
    [last_updated.last_updated_30] • [promoted]
    CRNA - Anesthesiology job available in Cicero, New York

    CRNA - Anesthesiology job available in Cicero, New York

    Odyssey Locums • Cicero, NY, US
    [job_card.full_time]
    Looking for flexibility, competitive pay, and the chance to shape your own career path? At Odyssey Locums, we connect CRNAs at every stage of their career - from new graduates to seasoned clinician...[show_more]
    [last_updated.last_updated_variable_hours] • [promoted] • [new]
    Staff AI Engineer - Notebooks

    Staff AI Engineer - Notebooks

    Datadog • Syracuse, New York, United States
    [job_card.full_time]
    Datadog Notebooks provide customers with a collaborative surface for ad-hoc data analysis, technical documentation, incident postmortems and runbooks. The power and flexibility of Notebooks also mak...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Staff Data Scientist

    Staff Data Scientist

    Unity Technologies • Syracuse, New York, United States
    [job_card.full_time]
    At Unity, we foster a culture of collaboration and innovation.In our fast-paced environment, we tackle complex challenges that create real impact for creators and users across our ecosystem.Our Ads...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    CRNA - Anesthesiology job available in Cicero, New York

    CRNA - Anesthesiology job available in Cicero, New York

    Odyssey Staffing • Cicero, NY, US
    [job_card.full_time]
    Locum CRNAs needed to start ASAP and work ongoing near Cicero, NY.K COMPLETION BONUS - 520 hours worked within 17 consecutive weeks • •. Shift Type : 10-hour shift : 7 : 00 AM – 5 : 30 PM.All ne...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Remote Machine Learning Engineer - AI Trainer ($80-$120 per hour)

    Remote Machine Learning Engineer - AI Trainer ($80-$120 per hour)

    Mercor • Syracuse, New York, US
    [filters.remote]
    [job_card.part_time]
    At Mercor, we’re building the talent engine that helps leading labs and research orgs move AI forward.Our latest initiative focuses on benchmarking and improving model performance and training spee...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Remote Engineering Expert (PhD, Master's, or Olympiad Participants) - AI Trainer ($60-$80 per hour)

    Remote Engineering Expert (PhD, Master's, or Olympiad Participants) - AI Trainer ($60-$80 per hour)

    Mercor • Syracuse, New York, US
    [filters.remote]
    [job_card.full_time]
    Role Overview Mercor is collaborating with a leading AI research lab on a project to advance engineering problem-solving at the frontier of artificial intelligence. We are seeking experienced engine...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Engineer, Analyst

    Engineer, Analyst

    Constellation Energy • Parish, New York, United States
    [job_card.full_time]
    As the nation's largest producer of clean, carbon-free energy, Constellation is focused on our purpose : accelerating the transition to a carbon-free future. We have been the leader in clean energy p...[show_more]
    [last_updated.last_updated_variable_hours] • [promoted]
    1st Shift Full-Time Warehouse Associate

    1st Shift Full-Time Warehouse Associate

    ALDI • Tully, NY, US
    [job_card.full_time]
    Shift Full-Time Warehouse Associate.Our warehouse operations make sure that products are properly received, selected and delivered to our stores for our ever-growing number of customers to enjoy.Wh...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Remote Market Research Contributor (Hiring Immediately)

    Remote Market Research Contributor (Hiring Immediately)

    Maxion Research • Cicero, New York, US
    [filters.remote]
    [job_card.part_time]
    Join Our Part-Time Research Studies Today!.Are you eager to earn extra income from the comfort of your home? We are looking for motivated individuals to participate in paid research studies includi...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Neuroscience Per Diem Advanced Practice Provider

    Neuroscience Per Diem Advanced Practice Provider

    Upstate University Hospital • Syracuse, NY, United States
    [job_card.full_time]
    Neuroscience Advanced Practice Provider.The APP will provide inpatient coverage at.Provide expert level clinical care and coordination of care via a full range of services that will improve quality...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Lead AI Engineer - New York

    Lead AI Engineer - New York

    Distyl Ai • Syracuse, New York, United States
    [job_card.full_time]
    AI native technologies for humans & AI to collaborate to power the operations of the Global Fortune 1000.In just 24 months, we’ve rapidly grown to partner with some of the world’s largest enterpris...[show_more]
    [last_updated.last_updated_30] • [promoted]