Lead Databricks Engineer

Who we are

Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers.

Job Description:

Job Title : Lead Databricks Engineer

Job Type : C2C / W2

Experience : 8-15 Years

Location : Remote

Role Summary:

We are seeking an experienced Lead Databricks Engineer to lead the design, development, and implementation of scalable, high-performance data engineering solutions using the Databricks Lakehouse Platform. The ideal candidate will possess strong hands-on expertise in Databricks, Apache Spark, PySpark, SQL, Delta Lake, and modern data architecture.

Key Responsibilities :

• Own the technical roadmap for Databricks-based data engineering initiatives, aligning platform architecture with enterprise data strategy and business goals.
• Lead architecture reviews and make high-impact technical decisions involving data platform design, technology selection, scalability, and maintainability.
• Define reusable engineering frameworks, reference architectures, coding standards, and development guidelines across multiple data engineering projects.
• Drive the implementation of domain-oriented data architecture and data product strategies to improve data accessibility, ownership, and reuse.
• Establish architectural patterns for multi-environment data platforms, including development, testing, staging, and production.
• Lead the adoption of Infrastructure as Code and automated platform provisioning to standardize Databricks environment management.
• Partner with security, governance, and compliance teams to establish enterprise-wide policies for sensitive data, auditability, and regulatory requirements.
• Manage technical dependencies, delivery risks, and architectural trade-offs across multiple teams and concurrent data initiatives.

Required Skills and Experience :

• 8 15 years of experience in designing and delivering enterprise data platforms using Databricks and cloud-native technologies.
• Strong proficiency in Apache Spark, PySpark, Python, SQL, and distributed data processing.
• Expertise in lakehouse architecture, Delta Lake, data architecture patterns, and large-scale data platform design.
• Proven experience establishing data governance, metadata management, access controls, and data lineage practices.
• Strong understanding of cloud architecture, networking, identity management, and security across Azure, AWS, or GCP.
• Experience with platform automation, Infrastructure as Code, CI/CD, and production deployment strategies.
• Ability to evaluate complex technical requirements and translate them into scalable architectural solutions.
• Knowledge of Databricks Asset Bundles and automated deployment practices.
• Understanding of data governance, security, data quality, and monitoring.

Qualification:

• Bachelor's degree or equivalent combination of education and experience.