Principal Data Engineer for AI Platform

Title and Summary

Principal Data Engineer for AI Platform

Role

Drive modernization from legacy and on-prem systems to modern, cloud-native, and hybrid data platforms.

Architect and lead the development of a Multi-Agent ETL Platform for batch and event streaming, integrating AI agents to autonomously manage ETL tasks such as data discovery, schema mapping, and error resolution.

Define and implement data ingestion, transformation, and delivery pipelines using scalable frameworks (e.g., Apache Airflow, Nifi, dbt, Spark, Kafka, or Dagster).

Leverage LLMs, and agent frameworks (e.g., LangChain, CrewAI, AutoGen) to automate pipeline management and monitoring.

Ensure robust data governance, cataloging, versioning, and lineage tracking across the ETL platform.

Define project roadmaps, KPIs, and performance metrics for platform efficiency and data reliability.

Establish and enforce best practices in data quality, CI / CD for data pipelines, and observability.

Collaborate closely with cross‑functional teams (Data Science, Analytics, and Application Development) to understand requirements and deliver efficient data ingestion and processing workflows.

Establish and enforce best practices, automation standards, and monitoring frameworks to ensure the platform’s reliability, scalability, and security.

Build relationships and communicate effectively with internal and external stakeholders, including senior executives, to influence data‑driven strategies and decisions.

Continuously engage and improve teams’ performance by conducting recurring meetings, knowing your people, managing career development, and understanding who is at risk.

Oversee deployment, monitoring, and scaling of ETL and agent workloads across multi cloud environments.

Continuously improve platform performance, cost efficiency, and automation maturity.

All About You

Hands‑on experience in data engineering, data platform strategy, or a related technical domain.

Proven experience leading global data engineering or platform engineering teams.

Proven experience in building and modernizing distributed data platforms using technologies such as Apache Spark, Kafka, Flink, NiFi, and Cloudera / Hadoop.

Strong experience with one or more of data pipeline tools (Nifi, Airflow, dbt, Spark, Kafka, Dagster, etc.) and distributed data processing at scale.

Experience building and managing AI‑augmented or agent‑driven systems will be a plus.

Proficiency in Python, SQL, and data ecosystems (Oracle, AWS Glue, Azure Data Factory, BigQuery, Snowflake, etc.).

Deep understanding of data modeling, metadata management, and data governance principles.

Proven success in leading technical teams and managing complex, cross‑functional projects.

Passion for staying current in a fast‑paced field with proven ability to lead innovation in a scaled organization.

Excellent communication skills, with the ability to tailor technical concepts to executive, operational, and technical audiences.

Expertise and ability to lead technical decision‑making considering scalability, cost efficiency, stakeholder priorities, and time to market.

Proven track leading high‑performing teams with experience leading and coaching director‑level reports and experienced individual contributors.

Advanced degree in Data Science, Computer Science, Information Technology, Business Administration, or a related field. Equivalent experience will also be considered.

Why Join Us?

At Mastercard, you’ll help shape the future of AI in global commerce—solving complex challenges at scale, driving financial inclusion, and reinforcing the trust and security that define our brand. You’ll work with world‑class talent, cutting‑edge technologies, and will make a lasting impact.

Equal Employment Opportunity

Mastercard is a merit‑based, inclusive, equal‑opportunity employer that considers applicants without regard to gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law. We hire the most qualified candidate for the role. In the US or Canada, if you require accommodations or assistance to complete the online application process or during the recruitment process, please contact reasonable_accommodation@mastercard.com and identify the type of accommodation or assistance you are requesting. Do not include any medical or health information in this email. The Reasonable Accommodations team will respond to your email promptly.

Corporate Security Responsibility

• Abide by Mastercard’s security policies and practices;
• Ensure the confidentiality and integrity of the information being accessed;

Report any suspected information security violation or breach, and

Complete all periodic mandatory security trainings in accordance with Mastercard’s guidelines.

Job Information

Seniority Level : Director

Employment Type : Full‑time

Job Function : Information Technology

Industries : Financial Services, IT Services and IT Consulting, and Technology, Information and Internet

Pay Ranges

Arlington, Virginia : $179,000 - $305,000 USD

Purchase, New York : $179,000 - $305,000 USD

San Francisco, California : $187,000 - $318,000 USD

#J-18808-Ljbffr