Senior Data Engineer - Spark, AirflowSigmaways Inc • Sunnyvale, CA, United States

Senior Data Engineer - Spark, Airflow

Sigmaways Inc • Sunnyvale, CA, United States

[job_card.variable_days_ago]

[job_preview.job_type]

[job_card.full_time]

[job_card.job_description]

We are seeking an experienced Data Engineer to design and optimize scalable data pipelines that drive our global data and analytics initiatives.

In this role, you will leverage technologies such as Apache Spark , Airflow , and Python to build high performance data processing systems and ensure data quality, reliability, and lineage across Mastercard’s data ecosystem.

The ideal candidate combines strong technical expertise with hands-on experience in distributed data systems, workflow automation, and performance tuning to deliver impactful, data-driven solutions at enterprise scale.

Responsibilities :

Design and optimize Spark-based ETL pipelines for large-scale data processing.
Build and manage Airflow DAGs for scheduling, orchestration, and checkpointing.
Implement partitioning and shuffling strategies to improve Spark performance.
Ensure data lineage, quality, and traceability across systems.
Develop Python scripts for data transformation, aggregation, and validation.
Execute and tune Spark jobs using spark-submit.
Perform DataFrame joins and aggregations for analytical insights.
Automate multi-step processes through shell scripting and variable management.
Collaborate with data, DevOps, and analytics teams to deliver scalable data solutions.

Qualifications :

Bachelor’s degree in Computer Science, Data Engineering, or related field (or equivalent experience).

At least 7 years of experience in data engineering or big data development.

Strong expertise in Apache Spark architecture, optimization, and job configuration.

Proven experience with Airflow DAGs using authoring, scheduling, checkpointing, monitoring.

Skilled in data shuffling, partitioning strategies, and performance tuning in distributed systems.

Expertise in Python programming including data structures and algorithmic problem-solving.

Hands-on with Spark DataFrames and PySpark transformations using joins, aggregations, filters.

Proficient in shell scripting, including managing and passing variables between scripts.

Experienced with spark submit for deployment and tuning.

Solid understanding of ETL design, workflow automation, and distributed data systems.

Excellent debugging and problem-solving skills in large-scale environments.

Experience with AWS Glue, EMR, Databricks, or similar Spark platforms.

Knowledge of data lineage and data quality frameworks like Apache Atlas.

Familiarity with CI / CD pipelines, Docker / Kubernetes, and data governance tools.

[job_alerts.create_a_job]

Senior Data Engineer • Sunnyvale, CA, United States

[internal_linking.similar_jobs]

Lead Data Engineer

Momento USA • San Jose, California, USA

[job_card.full_time] +1

Momento USA is a global technology consulting talent acquisition and creative development firm that addresses clients most pressing needs and challenges. We are currently looking for a.Architect con...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior Software Engineer, Data

Anvilogic Inc • Palo Alto, CA, United States

[job_card.full_time]

Anvilogic is a Palo Alto-based AI cybersecurity startup founded in 2019 by security veterans and data scientists from Fortune 500 companies. Our mission is to democratize threat detection and huntin...[show_more]

[last_updated.last_updated_30] • [promoted]

Senior Software & Data Engineer – AI Pipelines & Impact

Apple • Santa Clara, CA, United States

[job_card.full_time]

An innovative company is seeking a Senior Software and Data Engineer to join their dynamic Data Team.This role offers the chance to work on cutting-edge machine learning and AI technologies that im...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior / Staff Data Engineer

Balbix • San Jose, California, United States

[job_card.full_time]

The Balbix Security Cloud uses AI and automation to reinvent how the World's leading organizations reduce their cyber risk. With Balbix, security teams can accurately inventory their cloud and on-pr...[show_more]

[last_updated.last_updated_30] • [promoted]

Lead Data Platform Engineer, Big Data & AI

Intuit • Mountain View, CA, United States

[job_card.full_time]

A tech financial services company in Mountain View, CA, is looking for a Senior Staff Software Engineer to lead the Data Engineering Team. The ideal candidate will drive software development, mentor...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior Data Platform Engineer, Analytics & AI

Apple Inc. • Cupertino, CA, United States

[job_card.full_time]

A leading technology company in Cupertino, California, seeks a Senior Software Engineer for its Data Science & Analytics Platform team. The ideal candidate will lead platform engineering efforts, en...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior Data Engineer - GCP Expert (Onsite)

SRI Tech Solutions Inc. • Santa Clara, CA, United States

[job_card.full_time]

We are looking for a highly skilled and motivated Data Engineer to join our team.The ideal candidate will be responsible for designing, building, and maintaining scalable data infrastructure that d...[show_more]

[last_updated.last_updated_1_hour] • [promoted] • [new]

Senior Data Pipelines & Performance Engineer

xage, inc • Palo Alto, CA, United States

[job_card.full_time]

A leading zero trust security company in Palo Alto, CA seeks a Principal Software Engineer specializing in data pipelines. This role involves collaboration with engineers to build internal data syst...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Big Data Engineer Onsite

Saransh Inc • Mountain View, California, USA

[job_card.full_time]

Min of 7 years working with Apache Flink and Apache Spark.Expertise developing new pipelines.Adept at supporting and enhancing existing pipelines. Strong experience with AWS Stack.Apache Hive,S3,Ha...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior Data Systems Engineer

CoStar Group • Sunnyvale, California, USA

[job_card.full_time]

NASDAQ : CSGP) is a leading global provider of commercial and residential real estate information analytics and online marketplaces. Included in the S&P 500 Index and the NASDAQ 100 CoStar Group ...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior Data Engineer : 24-01769 (No C2C)

Akraya Inc • Sunnyvale, California, United States

[job_card.full_time]

[filters_job_card.quick_apply]

Primary Skills : SQL (Expert), Data Modeling (Advanced), Python (Advanced), AWS (Intermediate), Data Visualization (Intermediate). Duration : 11+ Months (Possible Extension).Location : Sunnyvale, CA (H...[show_more]

[last_updated.last_updated_30]

Senior Data Engineer

Two95 International Inc. • Sunnyvale, CA, US

[job_card.full_time]

[filters_job_card.quick_apply]

Provide business insights, while leveraging internal tools and systems, databases and industry data.Ability to document requirements, data lineage, subject matter in both business and technical ter...[show_more]

[last_updated.last_updated_30]

Senior Data Engineer

VirtualVocations • Fremont, California, United States

[job_card.full_time]

A company is looking for a Senior Data Engineer to help scale and optimize their analytics and data infrastructure.Key Responsibilities Develop and maintain scalable data pipelines and ETL proces...[show_more]

[last_updated.last_updated_30] • [promoted]

Senior Data Platform Engineer – Hybrid

General Motors • Mountain View, CA, United States

[job_card.full_time]

A leading automotive company in California is seeking an experienced engineer to architect and automate data processing systems. The role involves building data pipelines and ensuring data quality f...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior Data Science Engineer

Uare.ai • Los Altos, CA, US

[job_card.full_time]

Robert LoCascio (former CEO of LivePerson for 28 years), is an AI startup launched in May 2024 with a mission to empower people to do more with their memories. AI-driven personal digital twins, enab...[show_more]

[last_updated.last_updated_30] • [promoted]

Senior AI Engineer

Commerceiq • Mountain View, California, United States

[job_card.full_time]

CommerceIQ’s AI-powered digital commerce platform is revolutionizing the way brands sell online.Our unified ecommerce management solutions empower brands to make smarter, faster decisions through i...[show_more]

[last_updated.last_updated_30] • [promoted]

Senior Engineer - HashiCorp Core Data / Infragraph

IBM Computing • San Jose, CA, United States

[job_card.full_time]

A career in IBM Software means you'll be part of a team that transforms our customer's challenges into industry‑leading solutions. We are an infinitely curious team, always seeking new possibilities...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior Data Engineer

Adobe • San Jose, California, USA

[job_card.full_time]

Changing the world through digital experiences is what Adobes all about.We give everyonefrom emerging artists to global brandseverything they need to design and deliver exceptional digital experien...[show_more]

[last_updated.last_updated_variable_days] • [promoted]