Sr Azure Databricks Engineer with Python & Spark (Min 10+ yrs with Azure Data) - W2 Role

Role : Sr Azure Databricks Engineer with Python & Spark

Location : Raleigh, NC (Onsite from Day 1)

Job Type : W2 Contract

Note : Only Visa Independent candidates are required

Experience Level : Senior (Min 10+ yrs with Azure Data)

Must have skills (Mandatory) :

Apache Spark

Python

Azure Data Factory

Azure SQL Microsoft

Azure Synapse Analytics

ETL / ELT Solutions

Nice to have skills :

Apache Kafka

CI / CD

Experience & Qualifications :

Expert level understanding of Azure Data Factory, Azure Synapse, Azure SQL, Azure Data Lake, and Azure App Service

Working knowledge of Python

Expert level skills writing and optimizing complex SQL

Experience with complex data modeling, ETL design, and using large databases in a business environment

Experience with building data pipelines and applications to stream and process datasets at low latencies

Fluent with Big Data technologies like Spark, Kafka and Hive

Designing and building data pipelines using API ingestion and Streaming ingestion methods

Knowledge of Dev-Ops processes (including CI / CD) and Infrastructure as code

Experience in developing NoSQL solutions using Azure Cosmos DB

Thorough understanding of Azure and AWS Cloud Infrastructure offerings

Key Responsibilities :

Design and implement scalable and secure data processing pipelines using Azure Data Factory, Azure Databricks, and other Azure services

Manage and optimize data storage using Azure Data Lake Storage, Azure SQL Data Warehouse, and Azure Cosmos DB

Monitor and troubleshoot data-related issues within the Azure environment to maintain high availability and performance

Automate data pipelines and workflows to streamline data ingestion, processing, and distribution tasks

Utilize Azure's analytics services, such as Azure Synapse Analytics, to provide insights and support data-driven decision-making

Provide guidance and support for data governance, including metadata management, data lineage, and data cataloging