| Job Summary | - We are seeking an experienced Azure Databricks Data Engineer with 8+ years of hands-on experience in data engineering, ETL/ELT, distributed data processing, and large-scale data pipeline development.
- The ideal candidate will have strong expertise in Databricks, Apache Spark, Scala, PySpark, SQL, Azure, and data lakehouse architectures, with proven experience designing and optimizing batch and streaming data solutions.
- The role involves developing scalable data ingestion, transformation, and processing pipelines using Azure Databricks, implementing Delta Lake and Medallion architectures, optimizing Spark workloads, and supporting high-volume streaming environments.
- The candidate will also work closely with DevOps teams to troubleshoot pipelines, Azure services, AKS environments, and production Databricks workloads.
|
| Key Responsibilities | - Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, Spark, Scala, PySpark, and SQL.
- Develop complex data ingestion, transformation, aggregation, and processing workflows for large-volume structured and unstructured datasets.
- Build and optimize Databricks notebooks, Spark jobs, DataFrames, Datasets, and SQL-based solutions.
- Implement Delta Lake, Unity Catalog, Auto Loader, Lakehouse, and Medallion architecture patterns.
- Develop real-time and near-real-time data pipelines using Spark Structured Streaming.
- Design streaming solutions capable of processing high-volume data with appropriate performance, scalability, fault tolerance, and data-quality controls.
- Configure and optimize Databricks clusters, including sizing, autoscaling, compute configuration, and workload optimization.
- Develop and manage Databricks job orchestration, scheduling, dependencies, retries, alerts, and production workflows.
- Perform Spark, Scala/PySpark, and SQL performance tuning, including optimization of transformations, joins, partitioning, caching, file formats, and query execution.
- Design and implement enterprise data lake and data warehouse/lakehouse solutions using Azure and Databricks.
- Develop data pipelines using Databricks Serverless capabilities where appropriate.
- Monitor, debug, troubleshoot, and resolve issues in live/production Databricks jobs and data pipelines.
|
| Required Qualifications | - Must have atleast 8+ years of extensive Data engineering experience in Databricks, ETL/ELT using data pipelines, SQL/Procesurr experience
- Must have 8 years of hands on development proficiency in implementing Databricks solutions using scala and spark with data frames and notebooks/sql
- Must have 8+ years of hands on development and performance tuning/ enhancement experience in scala/pyspark and sql
- Must have strong experience with Databricks, including developing and optimizing spark jobs, data transformations, and data processing workflows
- Must have experience and a good understanding on the below topics
- Databricks delta lake storage
- Unity catalog
- Autoloader
- Lakehouse architecture
- Medallion architecture
- Databricks serverless options
- Dataset and dataframe
- Databricks clusters configuration and sizing
- Databricks job orchestration
- Spark structured streaming
- Must have experience with complex and large volume data streaming and data transformations using Azure Databricks
- Must have extensive experience in datawarehouse / data lakehouse implementation
- Must have experience with monitoring, debugging, and resolving issues in kive Databricks jobs
- Must have strong hands on expertise in troubleshooting devops pipelines, azure , and AKS services
- The candidate shd be able to provide data ingestion, streaming and transformations solutions and do development work
- Experience with apache big data implementation with programming proficiency like pyspark, R or Java prior to Databricks is advantageous
- Good to have experience with containerization technologies such as docker and Kubernetes
|
