Data Engineer (Philadelphia)EXL • Philadelphia, PA, US

Data Engineer (Philadelphia)

EXL • Philadelphia, PA, US

[job_card.variable_days_ago]

[job_preview.job_type]

[job_card.part_time]

[job_card.job_description]

Job Title : Data Engineer

Location : Philadelphia, PA (Hybrid)

Experience : 5+ years

Role Summary :

We are seeking an experienced Data Engineer with strong expertise in PySpark and data pipeline operations . This role focuses heavily on performance tuning Spark applications , managing large-scale data pipelines, and ensuring high operational stability. The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements.

Key Responsibilities :

Data Pipeline Management & Support

Operate and support Business-as-Usual (BAU) data pipelines, ensuring stability, SLA adherence, and timely incident resolution.
Identify and implement opportunities for optimization and automation across pipelines and operational workflows.

Spark Development & Performance Tuning

Design, develop, and optimize PySpark jobs for efficient large-scale data processing.

Diagnose and resolve complex Spark performance issues such as data skew, shuffle spill, executor OOM errors, slow-running stages, and partition imbalance.

Platform & Tool Management

Use Databricks for Spark job orchestration, workflow automation, and cluster configuration.

Debug and manage Spark on Kubernetes , addressing pod crashes, OOM kills, resource tuning, and scheduling problems.

Work with MinIO / S3 storage for bucket management, permissions, and large-volume file ingestion and retrieval.

Collaboration & Communication

Partner with onshore business stakeholders to clarify requirements and convert them into well-defined technical tasks.

Provide daily coordination and technical oversight to offshore engineering teams.

Participate actively in design discussions and technical reviews.

Documentation & Operational Excellence

Maintain accurate and detailed documentation, runbooks, and troubleshooting guides.

Contribute to process improvements that enhance operational stability and engineering efficiency.

Required Skills & Qualifications :

Primary Skills (Must-Have)

PySpark : Advanced proficiency in transformations, performance tuning, and Spark internals.

SQL : Strong analytical query design, performance tuning, and foundational data modeling (relational & dimensional).

Python : Ability to write maintainable, production-grade code with a focus on modularity, automation, and reusability.

Secondary Skills (Highly Desirable)

Kubernetes : Experience with Spark-on-K8s, including pod diagnostics, resource configuration, and log / monitoring tools.

Databricks : Hands-on experience with cluster management, workflow creation, Delta Lake optimization, and job monitoring.

MinIO / S3 : Familiarity with bucket configuration, policies, and efficient ingestion patterns.

DevOps : Experience with Git, CI / CD, and cloud environments (Azure preferred).

[job_alerts.create_a_job]

Data Engineer • Philadelphia, PA, US

[internal_linking.related_jobs]

Data Engineer, Penn Wharton Budget Model

InsideHigherEd • Philadelphia, Pennsylvania, United States

[job_card.temporary]

Data Engineer, Penn Wharton Budget Model.The University of Pennsylvania, the largest private employer in Philadelphia, is a world-renowned leader in education, research, and innovation.This histori...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Snowflake dbt Data Migration Engineer (Philadelphia)

Elsevier • Philadelphia, PA, US

[job_card.part_time]

Are you an experienced Data Migration Engineer with an expertise in.About the role, we are seeking a Data Migration Specialist with extensive experience in dbt (Data Build Tool) and Snowflake, and ...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Data Scientist / AI Engineer

Innovative Defense Technologies (IDT) • Mount Laurel, NJ, US

[job_card.full_time]

Innovative Defense Technologies.IDT), provider of cutting-edge cloud-based integration, automated testing and data analysis for complex, mission-critical systems in the US Department of Defense (DO...[show_more]

[last_updated.last_updated_30]

Data Architect

Novocure Inc • Philadelphia, PA, United States

[job_card.full_time]

We are seeking an experienced and innovative Data Architect to lead the design, development, and optimization of our enterprise data architecture. This individual will play a critical role in aligni...[show_more]

[last_updated.last_updated_30] • [promoted]

Data Engineer (Philadelphia)

Realtime Recruitment • Philadelphia, PA, US

[job_card.full_time] +1

Data Engineer Job Opportunity.Please note this role is open for US citizens or Green Card Holders only.Were looking for a Data Engineer to help build and enhance scalable data systems that power a...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Lead Data Engineer - Capital One Software (Remote)

Capital One • Feasterville, Pennsylvania, United States

[filters.remote]

[job_card.full_time] +1

Lead Data Engineer - Capital One Software (Remote) Capital One Software is seeking a Lead Data Engineer who is passionate about marrying innovation with emerging technologies.In 2022, we publicly a...[show_more]

[last_updated.last_updated_variable_hours] • [promoted] • [new]

Senior Healthcare Data Analyst (Philadelphia)

hMetrix • Philadelphia, PA, US

[job_card.part_time]

At hMetrix, we are transforming healthcare through data-driven solutions.We're looking for a Senior Healthcare Data Analyst to collaborate with and mentor talented hybrid teams of up to five health...[show_more]

[last_updated.last_updated_30] • [promoted]

Senior Data Engineer (Philadelphia)

Eigen X • Philadelphia, PA, US

[job_card.part_time]

We are seeking a passionate and skilled Senior Data Engineer to join our dynamic team in Philadelphia, PA.In this role, you will lead the design and implementation of advanced data pipelines for Bu...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Lead Data QA (Philadelphia)

e&e IT Consulting Services, Inc. • Philadelphia, PA, US

[job_card.part_time]

Lead Data QA for a hybrid contract opportunity in Philadelphia, PA!.The Lead Data QA is responsible for defining and driving the overall Data Quality Assurance strategy for enterprise-scale data pl...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Data Engineer

Comcast • Philadelphia, Pennsylvania, US

[job_card.full_time]

This role is responsible for developing data structures and pipelines aligned to established standards and guidelines to organize, collect, standardize, and transform data that helps generate insig...[show_more]

[last_updated.last_updated_variable_hours] • [promoted] • [new]

Data Engineer (Philadelphia)

EXL • Philadelphia, PA, United States

[job_card.full_time]

Location : Philadelphia, PA (Hybrid).The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements. Data Pipeline Management & S...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Data Pipeline, ETL Engineer

IntegriChain • Philadelphia, Pennsylvania, USA

[job_card.full_time]

We are seeking a Data Pipeline / ETL Engineer to design build and maintain the data ingestion and transformation processes that power IntegriChains analytics and SaaS platform.Youll work closely wi...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Solutions Engineer (Philadelphia)

Oscar • Philadelphia, PA, US

[job_card.full_time] +2

Oscar is working with a leading data transformation organization that is looking for an experienced.Act as the primary technical resource for the sales organization, providing support throughout th...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Analytics & Data Engineer

Weidenhammer • Philadelphia, Pennsylvania, United States, 19610

[job_card.full_time]

The Analytics & Data Engineer role is engaged in architecture, consulting and development of Data Engineering and Business Intelligence solutions related to client projects.This role will also part...[show_more]

[last_updated.last_updated_variable_days]

Lead Data Engineer (AWS, Azure, GCP)

Captech Consulting • Philadelphia, Pennsylvania, United States

[job_card.full_time]

CapTech is an award-winning consulting firm that collaborates with clients to achieve what’s possible through the power of technology. At CapTech, we’re passionate about the work we do and the resul...[show_more]

[last_updated.last_updated_30] • [promoted]

Application Data Platform Engineer (Philadelphia)

MKS2 Technologies • Philadelphia, PA, United States

[job_card.full_time] +1

Application Data Platform Engineer.Philadelphia, PA - hybrid schedule - 2 days on-site per week.Pay : $150,000 (negotiable) - full benefits. Seeking to hire an Oracle Application Data Platform Engine...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

AI / ML Data Engineer

Chabez Tech • Berlin, New Jersey, United States

[job_card.temporary]

We are seeking a highly skilled and motivated AI Engineer to join our data and analytics team.This role is ideal for someone passionate about building intelligent data-driven solutions using cuttin...[show_more]

[last_updated.last_updated_variable_days] • [promoted]

Data Engineer-1

eTeam Inc • Philadelphia, Pennsylvania, United States

[job_card.full_time]

[filters_job_card.quick_apply]

We are seeking a skilled and experienced Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our data platform. As a technical team member, you will lever...[show_more]

[last_updated.last_updated_30]