Senior Data Engineer

Job Description:

• Design, develop, and maintain production data pipelines on Databricks using Python, SQL, Apache Spark, and Delta Lake
• Build Bronze-layer ingestion from APIs, relational databases, flat files, cloud storage, and SaaS platforms using dlt and Databricks-native ingestion
• Develop Silver-layer transformations in dbt and Python over Delta Lake to cleanse, standardize, deduplicate, validate, conform, and enrich data
• Create Gold-layer data products including dimensional models, slowly changing dimensions, fact and bridge tables, aggregates, and serving tables
• Produce curated feature and training tables for ML engineering that are versioned and reproducible
• Author and maintain data contracts using the Open Data Contract Standard
• Implement data quality tests for uniqueness, not-null, referential integrity, accepted values, freshness, and business rules
• Orchestrate ingestion and transformation in Dagster Cloud across development, branch, and production deployments
• Apply governance through Unity Catalog and manage credentials through Azure Key Vault
• Implement incremental and merge-based Delta Lake processing and tune Spark jobs, table layouts, and compute
• Troubleshoot production failures, data-quality issues, source-system changes, late-arriving or duplicate data, backfills, and recovery
• Participate in the on-call rotation and perform root-cause analysis
• Build and maintain CI/CD for data assets in Azure DevOps
• Instrument pipelines for freshness, volume, quality, latency, and cost observability with SLA/SLO alerting
• Follow least-privilege access, secrets management, change management, and audit controls
• Participate in code reviews and document architecture, runbooks, and data products
• Collaborate with data architects, analysts, product owners, and business stakeholders to translate requirements into maintainable data solutions

Requirements:

• Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field; equivalent practical experience considered
• 3+ years building and supporting production data pipelines in a cloud data platform environment
• Strong hands-on Python and SQL
• Hands-on experience with Databricks or a comparable Spark-based lakehouse, including Delta Lake tables, MERGE, and incremental load patterns
• Practical understanding of medallion / multi-layer lakehouse design
• Experience ingesting data from APIs, relational databases, files, or SaaS applications, including incremental and state-management problems
• Working knowledge of dimensional modeling, ELT design patterns, and data quality practice
• Experience with orchestration and scheduling using Dagster, Databricks Workflows, Airflow, Azure Data Factory, or similar
• Git-based source control, pull request review, automated testing, and CI/CD; Azure DevOps or comparable
• Experience troubleshooting production data failures, performance bottlenecks, and source-system changes
• Experience with pipeline monitoring and alerting, including freshness or quality SLAs
• Ability to explain technical designs and trade-offs to technical and non-technical partners
• Databricks certification or equivalent demonstrated depth preferred
• Unity Catalog experience preferred
• dbt on Databricks or another Spark transformation framework preferred
• Python-based modeling frameworks over Delta Lake, Type 2 history, surrogate keys, and merge strategies preferred
• Experience with dlt, Airbyte, Meltano, or Fivetran preferred
• Dagster experience with assets, asset checks, sensors, schedules, and branch deployments preferred

Benefits:

• Comprehensive health, dental, and vision insurance
• Mental health benefits
• Employee assistance program
• Paid time off
• Paid parental leave
• Short-term disability
• Cultural observance day
• Contributions to healthcare accounts
• Pension plan
• 401(k) plan with Company matching
• ProHealth Rewards wellness platform with cash rewards
• Annual incentive based on individual and Company performance