Big data engineer Jobs in Allen, TX

Senior Data Engineer –Plano, TX | Full-Time / HybridPosition OverviewWe are seeking a highly skilled and experienced Senior Data Engineer to help design, develop, and optimize a modern cloud-based data platform.This role will focus heavily on Azure Databricks, PySpark, Spark SQL, scalable ETL/ELT pipelines, and data lakehouse architecture. The ideal candidate will have experience working in high-volume data environments and collaborating with data architects, data scientists, business stakeholders, and cloud infrastructure teams to deliver reliable, secure, and high-performance data solutions.Key ResponsibilitiesData Architecture & Engineering

• Design and implement scalable ETL/ELT pipelines using Azure Databricks, PySpark, and Spark SQL.
• Build robust data ingestion workflows from diverse sources, including APIs, event-streaming platforms, relational databases, and cloud storage.
• Develop and maintain Bronze, Silver, and Gold data layers using a modern medallion architecture.
• Establish best practices for Delta Lake, schema evolution, data versioning, and governance.
• Optimize large-scale data pipelines for performance, reliability, and cost efficiency through partitioning, caching, clustering, and Spark configuration tuning.

Cloud Data Platform

• Integrate Databricks with Azure cloud services, including cloud data lakes, data integration services, event-streaming platforms, data warehouses, secrets management, and CI/CD tooling.
• Implement secure and governed access to enterprise data using managed identities, role-based access controls, and data protection techniques.
• Support the development and evolution of a scalable cloud-native data platform.

Collaboration & Technical Leadership

• Partner with data analysts, data scientists, business stakeholders, and other technical teams to understand data requirements.
• Build trusted datasets, reporting structures, and business-critical KPIs.
• Provide technical leadership and mentorship to junior and mid-level engineers.
• Conduct code reviews and contribute to architectural and engineering standards.
• Collaborate with architecture, security, and infrastructure teams to ensure data solutions meet organizational security and compliance requirements.

Data Quality & Monitoring

• Implement data quality checks, validation frameworks, alerts, and lineage tracking.
• Develop monitoring and observability solutions for mission-critical data pipelines.
• Establish automated failure detection, alerting, and recovery processes.
• Continuously improve data reliability, scalability, and operational performance.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related technical field.
• 5+ years of professional data engineering experience, including 3+ years working extensively with Azure Databricks and Apache Spark.
• Strong expertise with Delta Lake, Unity Catalog, Databricks Notebooks, and SQL-based data warehousing.
• Hands-on experience with Azure data services and cloud infrastructure.
• Strong programming skills in Python and SQL;familiarity with Scala is a plus.
• Solid understanding of data modeling, including OLTP/OLAP environments and star/snowflake schemas.
• Experience with performance optimization and tuning of large-scale data workloads.
• Experience implementing CI/CD for data engineering environments using tools such as Azure DevOps, Terraform, or Databricks CLI.
• Strong understanding of data security, governance, and access-control practices.

Preferred Qualifications

• Databricks certification or comparable professional certification.
• Experience building real-time or streaming data pipelines using technologies such as Structured Streaming, Event Hubs, or Kafka.
• Experience supporting machine learning workflows or ML platforms.
• Familiarity with business intelligence and visualization platforms.
• Experience with data cataloging, lineage, and enterprise governance tools.
• Knowledge of data privacy, security, and compliance frameworks.
• Experience working in high-volume, production data environments.

Work Arrangement

• Location: Plano, TX
• Schedule: Full-Time
• Work Arrangement: Hybrid — 4 days per week on-site
• Remote: 1 day per week
• Relocation Assistance: Not Provided
• Work Authorization: Candidates must be authorized to work in the United States. Sponsorship is not available for this position.
• Local Requirement: Candidates must be able to commute to the Plano/Dallas–Fort Worth area and work on-site four days per week.

Compensation & BenefitsThe organization offers competitive compensation and a comprehensive benefits package. Additional details regarding compensation and benefits will be discussed during the interview process.