Data Engineer (Philadelphia)EXL • Philadelphia, PA, US

Data Engineer (Philadelphia)

EXL • Philadelphia, PA, US

[job_card.variable_days_ago]

[job_preview.job_type]

[job_card.part_time]

[job_card.job_description]

Job Title : Data Engineer

Location : Philadelphia, PA (Hybrid)

Experience : 5+ years

Role Summary :

We are seeking an experienced Data Engineer with strong expertise in PySpark and data pipeline operations . This role focuses heavily on performance tuning Spark applications , managing large-scale data pipelines, and ensuring high operational stability. The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements.

Key Responsibilities :

Data Pipeline Management & Support

Operate and support Business-as-Usual (BAU) data pipelines, ensuring stability, SLA adherence, and timely incident resolution.
Identify and implement opportunities for optimization and automation across pipelines and operational workflows.

Spark Development & Performance Tuning

Design, develop, and optimize PySpark jobs for efficient large-scale data processing.

Diagnose and resolve complex Spark performance issues such as data skew, shuffle spill, executor OOM errors, slow-running stages, and partition imbalance.

Platform & Tool Management

Use Databricks for Spark job orchestration, workflow automation, and cluster configuration.

Debug and manage Spark on Kubernetes , addressing pod crashes, OOM kills, resource tuning, and scheduling problems.

Work with MinIO / S3 storage for bucket management, permissions, and large-volume file ingestion and retrieval.

Collaboration & Communication

Partner with onshore business stakeholders to clarify requirements and convert them into well-defined technical tasks.

Provide daily coordination and technical oversight to offshore engineering teams.

Participate actively in design discussions and technical reviews.

Documentation & Operational Excellence

Maintain accurate and detailed documentation, runbooks, and troubleshooting guides.

Contribute to process improvements that enhance operational stability and engineering efficiency.

Required Skills & Qualifications :

Primary Skills (Must-Have)

PySpark : Advanced proficiency in transformations, performance tuning, and Spark internals.

SQL : Strong analytical query design, performance tuning, and foundational data modeling (relational & dimensional).

Python : Ability to write maintainable, production-grade code with a focus on modularity, automation, and reusability.

Secondary Skills (Highly Desirable)

Kubernetes : Experience with Spark-on-K8s, including pod diagnostics, resource configuration, and log / monitoring tools.

Databricks : Hands-on experience with cluster management, workflow creation, Delta Lake optimization, and job monitoring.

MinIO / S3 : Familiarity with bucket configuration, policies, and efficient ingestion patterns.

DevOps : Experience with Git, CI / CD, and cloud environments (Azure preferred).

[job_alerts.create_a_job]

Data Engineer • Philadelphia, PA, US

[internal_linking.similar_jobs]

Data Engineer, Penn Wharton Budget Model

InsideHigherEd • Philadelphia, Pennsylvania, United States

[job_card.temporary]

Data Engineer, Penn Wharton Budget Model.The University of Pennsylvania, the largest private employer in Philadelphia, is a world-renowned leader in education, research, and innovation.This histori...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Snowflake dbt Data Migration Engineer (Philadelphia)

Elsevier • Philadelphia, PA, United States

[job_card.full_time]

Are you an experienced Data Migration Engineer with an expertise in.About the role, we are seeking a Data Migration Specialist with extensive experience in dbt (Data Build Tool) and Snowflake, and ...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Data Scientist / AI Engineer

Innovative Defense Technologies (IDT) • Mount Laurel, NJ, US

[job_card.full_time]

Innovative Defense Technologies.IDT), provider of cutting-edge cloud-based integration, automated testing and data analysis for complex, mission-critical systems in the US Department of Defense (DO...[show_more]

[last_updated.last_updated_30]

Data Architect

Novocure Inc • Philadelphia, PA, United States

[job_card.full_time]

We are seeking an experienced and innovative Data Architect to lead the design, development, and optimization of our enterprise data architecture. This individual will play a critical role in aligni...[show_more]

[last_updated.last_updated_30] • [promoted]

Backend Engineer

Comcast • Philadelphia, Pennsylvania, US

[job_card.full_time]

We’re looking for a Backend Engineer (Cloud) who’s passionate about building amazing cloud-based systems and solving tough problems. If you love coding, thrive on challenges, and want to...[show_more]

[last_updated.last_updated_variable_hours] • [promoted] • [new]

Senior Data Engineer (Philadelphia)

Eigen X • Philadelphia, PA, US

[job_card.part_time]

We are seeking a passionate and skilled Senior Data Engineer to join our dynamic team in Philadelphia, PA.In this role, you will lead the design and implementation of advanced data pipelines for Bu...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Senior Data Engineer

Gopuff • Philadelphia, Pennsylvania, United States, 19123

[job_card.full_time]

At Gopuff, data sits at the heart of our strategy.We are reimagining how people purchase everyday essentials, from snacks to household goods to alcohol, all delivered in minutes.To fuel this missio...[show_more]

[last_updated.last_updated_variable_days]

Data Engineer (Philadelphia)

Realtime Recruitment • Philadelphia, PA, United States

[job_card.full_time]

Data Engineer Job Opportunity.Please note this role is open for US citizens or Green Card Holders only.Were looking for a Data Engineer to help build and enhance scalable data systems that power a...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Data Engineer

CGS • Philadelphia, Pennsylvania, United States, 19102

[job_card.full_time]

Employment Type : Full-Time, Mid-level.Department : Business Intelligence.CGS is seeking a passionate and driven Data Engineer to support a rapidly growing Data Analytics and Business Intelligence pl...[show_more]

[last_updated.last_updated_30]

Senior Healthcare Data Analyst (Philadelphia)

hMetrix • Philadelphia, PA, United States

[job_card.full_time]

At hMetrix, we are transforming healthcare through data-driven solutions.We're looking for a Senior Healthcare Data Analyst to collaborate with and mentor talented hybrid teams of up to five health...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Analytics & Data Engineer

Weidenhammer • Philadelphia, Pennsylvania, United States, 19610

[job_card.full_time]

The Analytics & Data Engineer role is engaged in architecture, consulting and development of Data Engineering and Business Intelligence solutions related to client projects.This role will also part...[show_more]

[last_updated.last_updated_variable_days]

Solutions Engineer (Philadelphia)

Oscar • Philadelphia, PA, United States

[job_card.full_time] +1

Oscar is working with a leading data transformation organization that is looking for an experienced.Act as the primary technical resource for the sales organization, providing support throughout th...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Lead Data QA (Philadelphia)

e&e IT Consulting Services, Inc. • Philadelphia, PA, United States

[job_card.full_time]

Lead Data QA for a hybrid contract opportunity in Philadelphia, PA!.The Lead Data QA is responsible for defining and driving the overall Data Quality Assurance strategy for enterprise-scale data pl...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

MLOps Engineer (Philadelphia)

ValueMomentum • Philadelphia, PA, US

[job_card.part_time]

Location : Erie, PA (Day 1 Complete Onsite).DevOps, MLOps, platform engineering, or cloud infrastructure.Strong skills in containerization (Docker, Kubernetes), API hosting, and cloud-native servic...[show_more]

[last_updated.last_updated_variable_hours] • [promoted] • [new]

Business Data Architect (Philadelphia)

e&e Technical Consultants, LLC • Philadelphia, PA, United States

[job_card.full_time]

Business Data Architect for a remote contract opportunity in Philadelphia, PA!.The Business Data Architect is a hybrid role combining expertise in data architecture, business architecture, and busi...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Data Center Chief Engineer, Data Center Eng Ops

Amazon Data Services, Inc. • Fairless Hills, PA, US

[job_card.full_time]

Amazon Web Services (AWS) is looking for a detail-oriented individual to join our Data Center Engineering Operations Team. This committed group works to maintain the critical physical infrastructure...[show_more]

[last_updated.last_updated_30] • [promoted]

Junior Data Support Engineer

Scale AI • Philadelphia, Pennsylvania, United States

[job_card.full_time]

Join a global community of talented professionals to shape the future of AI.Earn up to $15 USD / hr and additional rewards based on quality of submission. Outlier is committed to improving the intelli...[show_more]

[last_updated.last_updated_variable_hours] • [promoted] • [new]

Data Engineer-1

eTeam Inc • Philadelphia, Pennsylvania, United States

[job_card.full_time]

[filters_job_card.quick_apply]

We are seeking a skilled and experienced Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our data platform. As a technical team member, you will lever...[show_more]

[last_updated.last_updated_30]