Senior Data Engineer - Spark, AirflowSigmaways Inc • Sunnyvale, CA, United States
Senior Data Engineer - Spark, Airflow
Sigmaways Inc • Sunnyvale, CA, United States
[job_card.variable_days_ago]
[job_preview.job_type]
- [job_card.full_time]
[job_card.job_description]Bachelor’s degree in Computer Science, Data Engineering, or related field (or equivalent experience). At least 7 years of experience in data engineering or big data development. Strong expertise in Apache Spark architecture, optimization, and job configuration. Proven experience with Airflow DAGs using authoring, scheduling, checkpointing, monitoring. Skilled in data shuffling, partitioning strategies, and performance tuning in distributed systems. Expertise in Python programming including data structures and algorithmic problem-solving. Hands-on with Spark DataFrames and PySpark transformations using joins, aggregations, filters. Proficient in shell scripting, including managing and passing variables between scripts. Experienced with spark submit for deployment and tuning. Solid understanding of ETL design, workflow automation, and distributed data systems. Excellent debugging and problem-solving skills in large-scale environments. Experience with AWS Glue, EMR, Databricks, or similar Spark platforms. Knowledge of data lineage and data quality frameworks like Apache Atlas. Familiarity with CI / CD pipelines, Docker / Kubernetes, and data governance tools.
We are seeking an experienced Data Engineer to design and optimize scalable data pipelines that drive our global data and analytics initiatives.
In this role, you will leverage technologies such as Apache Spark , Airflow , and Python to build high performance data processing systems and ensure data quality, reliability, and lineage across Mastercard’s data ecosystem.
The ideal candidate combines strong technical expertise with hands-on experience in distributed data systems, workflow automation, and performance tuning to deliver impactful, data-driven solutions at enterprise scale.
Responsibilities :
- Design and optimize Spark-based ETL pipelines for large-scale data processing.
- Build and manage Airflow DAGs for scheduling, orchestration, and checkpointing.
- Implement partitioning and shuffling strategies to improve Spark performance.
- Ensure data lineage, quality, and traceability across systems.
- Develop Python scripts for data transformation, aggregation, and validation.
- Execute and tune Spark jobs using spark-submit.
- Perform DataFrame joins and aggregations for analytical insights.
- Automate multi-step processes through shell scripting and variable management.
- Collaborate with data, DevOps, and analytics teams to deliver scalable data solutions.
Qualifications :
[job_alerts.create_a_job]
Senior Data Engineer • Sunnyvale, CA, United States
[internal_linking.related_jobs]%20Inc.)
The Annapurna Labs team at Amazon Web Services (AWS) builds AWS Neuron, the software development kit used to accelerate deep learning and GenAI workloads on Amazon’s custom machine learning a...[show_more]
Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale - unleashing the potential of businesses and people.The Elastic Search AI...[show_more]
Momento USA is a global technology consulting talent acquisition and creative development firm that addresses clients most pressing needs and challenges.
We are currently looking for a.Architect con...[show_more]
The ideal candidate will excel at collaborating with business users cross-functional teams and offshore teams to deliver high-quality data solutions.
A solid understanding and experience with.Design...[show_more]
An innovative company is seeking a Senior Software and Data Engineer to join their dynamic Data Team.This role offers the chance to work on cutting-edge machine learning and AI technologies that im...[show_more]
The Balbix Security Cloud uses AI and automation to reinvent how the World's leading organizations reduce their cyber risk.
With Balbix, security teams can accurately inventory their cloud and on-pr...[show_more]
A tech financial services company in Mountain View, CA, is looking for a Senior Staff Software Engineer to lead the Data Engineering Team.
The ideal candidate will drive software development, mentor...[show_more]
A leading technology company in Cupertino, California, seeks a Senior Software Engineer for its Data Science & Analytics Platform team.
The ideal candidate will lead platform engineering efforts, en...[show_more]
We are seeking an experienced Senior Enterprise Datawarehouse Developer to join our team.As a Senior Datawarehouse Engineer, you will provide advanced data warehousing design solutions , support, m...[show_more] If you’re hands on with modern data platforms, cloud tech, and big data tools and you like building solutions that are secure, repeatable, and fast, this role is for you.
As a Senior Data Engineer, ...[show_more] 
An innovative tech and logistics company is seeking a Senior Data Engineer to enhance their data infrastructure and tools.
In this role, you'll collaborate with stakeholders to gather data requireme...[show_more]
A leading zero trust security company in Palo Alto, CA seeks a Principal Software Engineer specializing in data pipelines.
This role involves collaboration with engineers to build internal data syst...[show_more]
Expected skills : Python, Golang / Rust (nice to have).Data Engineering tools : pyiceberg, daft to name a few.The candidate should be familiar with data engineering supporting and building systems at P...[show_more]
Location : Sunnyvale CA ( 3days Hybrid).The ideal candidate will excel at collaborating with business users cross-functional teams and offshore teams to deliver high-quality data solutions.A solid un...[show_more]
Provide business insights, while leveraging internal tools and systems, databases and industry data.Ability to document requirements, data lineage, subject matter in both business and technical ter...[show_more]
A leading automotive company in California is seeking an experienced engineer to architect and automate data processing systems.
The role involves building data pipelines and ensuring data quality f...[show_more]
CommerceIQ’s AI-powered digital commerce platform is revolutionizing the way brands sell online.Our unified ecommerce management solutions empower brands to make smarter, faster decisions through i...[show_more]
Changing the world through digital experiences is what Adobes all about.We give everyonefrom emerging artists to global brandseverything they need to design and deliver exceptional digital experien...[show_more]
Senior Software Development Engineer, AI / ML, AWS Neuron, Model Inference
Annapurna Labs (U.S.) Inc. • Cupertino, CA, US
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Staff Data Engineer
Elastic • Mountain View, CA, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Lead Data Engineer
Momento USA • San Jose, California, USA
[job_card.full_time] +1
[last_updated.last_updated_variable_days] • [promoted]
Senior Data Engineer
Cloudious LLC • Sunnyvale, California, USA
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior Software & Data Engineer – AI Pipelines & Impact
Apple • Santa Clara, CA, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior / Staff Data Engineer
Balbix • San Jose, California, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Lead Data Platform Engineer, Big Data & AI
Intuit • Mountain View, CA, United States
[job_card.full_time]
[last_updated.last_updated_1_day] • [promoted]
Senior Data Platform Engineer, Analytics & AI
Apple Inc. • Cupertino, CA, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior Enterprise Datawarehouse Developer
Fortinet • Sunnyvale, CA, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Senior Data Engineer
Sigmaways Inc • Santa Clara, CA, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior Data Engineer : Scale Data Pipelines & Warehouses
DoorDash • Sunnyvale, California, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior Data Pipelines & Performance Engineer
xage, inc • Palo Alto, CA, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
MlOps / Data Engineer
TEKsystems • Cupertino, CA, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior Data Engineer
Fixity Technologies LLC • Sunnyvale, California, USA
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior Data Engineer
Two95 International Inc. • Sunnyvale, CA, US
[job_card.full_time]
[filters_job_card.quick_apply]
[last_updated.last_updated_30]
Senior Data Platform Engineer – Hybrid
General Motors • Mountain View, CA, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Senior AI Engineer
Commerceiq • Mountain View, California, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Senior Data Engineer
Adobe • San Jose, California, USA
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]