Talent.com
Data Engineer (Philadelphia)
Data Engineer (Philadelphia)EXL • Philadelphia, PA, US
Data Engineer (Philadelphia)

Data Engineer (Philadelphia)

EXL • Philadelphia, PA, US
[job_card.variable_days_ago]
[job_preview.job_type]
  • [job_card.part_time]
[job_card.job_description]

Job Title : Data Engineer

Location : Philadelphia, PA (Hybrid)

Experience : 5+ years

Role Summary :

We are seeking an experienced Data Engineer with strong expertise in PySpark and data pipeline operations . This role focuses heavily on performance tuning Spark applications , managing large-scale data pipelines, and ensuring high operational stability. The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements.

Key Responsibilities :

Data Pipeline Management & Support

  • Operate and support Business-as-Usual (BAU) data pipelines, ensuring stability, SLA adherence, and timely incident resolution.
  • Identify and implement opportunities for optimization and automation across pipelines and operational workflows.

Spark Development & Performance Tuning

  • Design, develop, and optimize PySpark jobs for efficient large-scale data processing.
  • Diagnose and resolve complex Spark performance issues such as data skew, shuffle spill, executor OOM errors, slow-running stages, and partition imbalance.

    • Platform & Tool Management

  • Use Databricks for Spark job orchestration, workflow automation, and cluster configuration.
  • Debug and manage Spark on Kubernetes , addressing pod crashes, OOM kills, resource tuning, and scheduling problems.
  • Work with MinIO / S3 storage for bucket management, permissions, and large-volume file ingestion and retrieval.

    • Collaboration & Communication

  • Partner with onshore business stakeholders to clarify requirements and convert them into well-defined technical tasks.
  • Provide daily coordination and technical oversight to offshore engineering teams.
  • Participate actively in design discussions and technical reviews.

    • Documentation & Operational Excellence

  • Maintain accurate and detailed documentation, runbooks, and troubleshooting guides.
  • Contribute to process improvements that enhance operational stability and engineering efficiency.

    • Required Skills & Qualifications :

    Primary Skills (Must-Have)

  • PySpark : Advanced proficiency in transformations, performance tuning, and Spark internals.
  • SQL : Strong analytical query design, performance tuning, and foundational data modeling (relational & dimensional).
  • Python : Ability to write maintainable, production-grade code with a focus on modularity, automation, and reusability.

    • Secondary Skills (Highly Desirable)

  • Kubernetes : Experience with Spark-on-K8s, including pod diagnostics, resource configuration, and log / monitoring tools.
  • Databricks : Hands-on experience with cluster management, workflow creation, Delta Lake optimization, and job monitoring.
  • MinIO / S3 : Familiarity with bucket configuration, policies, and efficient ingestion patterns.
  • DevOps : Experience with Git, CI / CD, and cloud environments (Azure preferred).
    • [job_alerts.create_a_job]

    Data Engineer • Philadelphia, PA, US

    [internal_linking.related_jobs]
    Data Engineer, Penn Wharton Budget Model

    Data Engineer, Penn Wharton Budget Model

    InsideHigherEd • Philadelphia, Pennsylvania, United States
    [job_card.temporary]
    Data Engineer, Penn Wharton Budget Model.The University of Pennsylvania, the largest private employer in Philadelphia, is a world-renowned leader in education, research, and innovation.This histori...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Snowflake dbt Data Migration Engineer (Philadelphia)

    Snowflake dbt Data Migration Engineer (Philadelphia)

    Elsevier • Philadelphia, PA, US
    [job_card.part_time]
    Are you an experienced Data Migration Engineer with an expertise in.About the role, we are seeking a Data Migration Specialist with extensive experience in dbt (Data Build Tool) and Snowflake, and ...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Data Scientist / AI Engineer

    Data Scientist / AI Engineer

    Innovative Defense Technologies (IDT) • Mount Laurel, NJ, US
    [job_card.full_time]
    Innovative Defense Technologies.IDT), provider of cutting-edge cloud-based integration, automated testing and data analysis for complex, mission-critical systems in the US Department of Defense (DO...[show_more]
    [last_updated.last_updated_30]
    Data Architect

    Data Architect

    Novocure Inc • Philadelphia, PA, United States
    [job_card.full_time]
    We are seeking an experienced and innovative Data Architect to lead the design, development, and optimization of our enterprise data architecture. This individual will play a critical role in aligni...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Data Engineer (Philadelphia)

    Data Engineer (Philadelphia)

    Realtime Recruitment • Philadelphia, PA, US
    [job_card.full_time] +1
    Data Engineer Job Opportunity.Please note this role is open for US citizens or Green Card Holders only.Were looking for a Data Engineer to help build and enhance scalable data systems that power a...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Lead Data Engineer - Capital One Software (Remote)

    Lead Data Engineer - Capital One Software (Remote)

    Capital One • Feasterville, Pennsylvania, United States
    [filters.remote]
    [job_card.full_time] +1
    Lead Data Engineer - Capital One Software (Remote) Capital One Software is seeking a Lead Data Engineer who is passionate about marrying innovation with emerging technologies.In 2022, we publicly a...[show_more]
    [last_updated.last_updated_variable_hours] • [promoted] • [new]
    Senior Healthcare Data Analyst (Philadelphia)

    Senior Healthcare Data Analyst (Philadelphia)

    hMetrix • Philadelphia, PA, US
    [job_card.part_time]
    At hMetrix, we are transforming healthcare through data-driven solutions.We're looking for a Senior Healthcare Data Analyst to collaborate with and mentor talented hybrid teams of up to five health...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Senior Data Engineer (Philadelphia)

    Senior Data Engineer (Philadelphia)

    Eigen X • Philadelphia, PA, US
    [job_card.part_time]
    We are seeking a passionate and skilled Senior Data Engineer to join our dynamic team in Philadelphia, PA.In this role, you will lead the design and implementation of advanced data pipelines for Bu...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Data Engineer

    Data Engineer

    Comcast • Philadelphia, Pennsylvania, US
    [job_card.full_time]
    This role is responsible for developing data structures and pipelines aligned to established standards and guidelines to organize, collect, standardize, and transform data that helps generate insig...[show_more]
    [last_updated.last_updated_1_day] • [promoted]
    Data Engineer (Philadelphia)

    Data Engineer (Philadelphia)

    EXL • Philadelphia, PA, United States
    [job_card.full_time]
    Location : Philadelphia, PA (Hybrid).The ideal candidate is a strong technical problem-solver, highly collaborative, and proactive in automation and process improvements. Data Pipeline Management & S...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Solutions Engineer (Philadelphia)

    Solutions Engineer (Philadelphia)

    Oscar • Philadelphia, PA, US
    [job_card.full_time] +2
    Oscar is working with a leading data transformation organization that is looking for an experienced.Act as the primary technical resource for the sales organization, providing support throughout th...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Lead Data QA (Philadelphia)

    Lead Data QA (Philadelphia)

    e&e IT Consulting Services, Inc. • Philadelphia, PA, United States
    [job_card.full_time]
    Lead Data QA for a hybrid contract opportunity in Philadelphia, PA!.The Lead Data QA is responsible for defining and driving the overall Data Quality Assurance strategy for enterprise-scale data pl...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Analytics & Data Engineer

    Analytics & Data Engineer

    Weidenhammer • Philadelphia, Pennsylvania, United States, 19610
    [job_card.full_time]
    The Analytics & Data Engineer role is engaged in architecture, consulting and development of Data Engineering and Business Intelligence solutions related to client projects.This role will also part...[show_more]
    [last_updated.last_updated_variable_days]
    Principal, Data Engineering (Remote)

    Principal, Data Engineering (Remote)

    Jazz Pharmaceuticals • Philadelphia, PA, United States
    [filters.remote]
    [job_card.full_time]
    If you are a current Jazz employee please apply via the Internal Career site.Jazz Pharmaceuticals is a global biopharma company whose purpose is to innovate to transform the lives of patients and ...[show_more]
    [last_updated.last_updated_30] • [promoted]
    Associate Principal Engineer, Detection & Response

    Associate Principal Engineer, Detection & Response

    Unisys Corporation • Philadelphia, PA, United States
    [job_card.full_time]
    What success looks like in this role : .Develop and Implement Custom Detections : .Design, develop, and maintain high-fidelity detection rules, signatures, and analytics for a diverse array of enterpri...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Time-Series Data Engineer

    Time-Series Data Engineer

    Kane Partners LLC • Doylestown, Pennsylvania, United States
    [job_card.full_time]
    Local Candidates Only – No Sponsorship • •.A growing technology company in the Warrington, PA area is seeking a Data Engineer to join its analytics and machine learning team.This is a hands-on, engin...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]
    Data Engineer-1

    Data Engineer-1

    eTeam Inc • Philadelphia, Pennsylvania, United States
    [job_card.full_time]
    [filters_job_card.quick_apply]
    We are seeking a skilled and experienced Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our data platform. As a technical team member, you will lever...[show_more]
    [last_updated.last_updated_30]
    Business Data Architect (Philadelphia)

    Business Data Architect (Philadelphia)

    e&e Technical Consultants, LLC • Philadelphia, PA, US
    [job_card.part_time]
    Business Data Architect for a remote contract opportunity in Philadelphia, PA!.The Business Data Architect is a hybrid role combining expertise in data architecture, business architecture, and busi...[show_more]
    [last_updated.last_updated_variable_days] • [promoted]