Databricks Lead - Data Engineering (Dallas)

Job Title : Databricks Lead - Data Engineering

Location : Dallas, TX

Job Summary :

As a Databricks Lead, you will be a critical member of our data engineering team, responsible for designing, developing, and optimizing our data pipelines and platforms on Databricks, primarily leveraging AWS services. You will play a key role in implementing robust data governance with Unity Catalog and ensuring cost-effective data solutions. This role requires a strong technical leader who can mentor junior engineers, drive best practices, and contribute hands-on to complex data challenges.

Responsibilities :

• Databricks Platform Leadership :
• Lead the design, development, and deployment of large-scale data solutions on the Databricks platform.
• Establish and enforce best practices for Databricks usage, including notebook development, job orchestration, and cluster management.
• Stay abreast of the latest Databricks features and capabilities, recommending and implementing improvements.
• Data Ingestion and Streaming (Kafka) :
• Architect and implement real-time and batch data ingestion pipelines using Apache Kafka for high-volume data streams.
• Integrate Kafka with Databricks for seamless data processing and analysis.
• Optimize Kafka consumers and producers for performance and reliability.
• Data Governance and Management (Unity Catalog) :
• Implement and manage data governance policies and access controls using Databricks Unity Catalog.
• Define and enforce data cataloging, lineage, and security standards within the Databricks Lakehouse.
• Collaborate with data governance teams to ensure compliance and data quality.
• AWS Cloud Integration :
• Leverage various AWS services (S3, EC2, Lambda, Glue, etc.) to build a robust and scalable data infrastructure.
• Manage and optimize AWS resources for Databricks workloads.
• Ensure secure and compliant integration between Databricks and AWS.
• Cost Optimization :
• Proactively identify and implement strategies for cost optimization across Databricks and AWS resources.
• Monitor DBU consumption, cluster utilization, and storage costs, providing recommendations for efficiency gains.
• Implement autoscaling, auto-termination, and right-sizing strategies to minimize operational expenses.
• Technical Leadership & Mentoring :
• Provide technical guidance and mentorship to a team of data engineers.
• Conduct code reviews, promote coding standards, and foster a culture of continuous improvement.
• Lead technical discussions and decision-making for complex data engineering problems.
• Data Pipeline Development & Optimization :
• Develop, test, and maintain robust and efficient ETL / ELT pipelines using PySpark / Spark SQL.
• Optimize Spark jobs for performance, scalability, and resource utilization.
• Troubleshoot and resolve complex data pipeline issues.
• Collaboration :
• Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.
• Communicate technical concepts effectively to both technical and non-technical stakeholders.

Qualifications :

Added Advantage (Bonus Skills) :

What We Offer :