Azure Data Engineer ADF

Azure Data Engineer ADF and

PySpark and python need

Competencies : 10 years experience required

Digital : Microsoft Azure

Agile Way of Working

Digital : Python

Role Description :

Take ownership of stories and drive them to completion through all phases of the entire 84.51 SDLC.

This includes external facing and internal applications as well as process improvement activities such as

Develop distributed data processing data pipeline solutions

Orchestrate multi-step data transformation pipelines

Perform unit integration and regression testing on packaged code

Build transformation logic and code in an Object Oriented Programming style

Enhance CICD pipelines in the path to production

Create data quality checks for ingested and post processed data

Ensure data observability via alerting and monitoring of automated pipeline solutions

Maintain and enhance existing applications

Build cloud resources via infrastructure as code

Provide mentoring to junior team members

Participate in retrospective reviews

Participate in the estimation process for new work and releases

Bring new perspectives to problems

Be driven to improve yourself and the way things are done

Essential Skills :

Bachelors degree typically in Computer Science Management Information Systems Mathematics Business Analytics or another technically strong program.

2 years of proven professional data development experience

2 years developing with SQL

Proficient with software engineering best practices

Proficient with data pipeline orchestration

Proficient with automated testing (PyTest etc)

Proficient with VCS (Git GitHub)

Proficient using Python frameworks

Proficient with Object Oriented Programming principles

Proficient with Java and Sprint Framework

Experience with distributed data processing (PySpark and / or Snowpark)

Experience with Data Observability

Experience with Data Quality checks and processes

Experience with Cloud Technologies Services (Azure preferred GCP AWS)

Experience with CICD

Experience with dependency management (Conda venv etc)

Experience with debugging enterprise applications

Understanding of performance tuning enterprise processes

Understanding of Infrastructure as Code

Understanding SOLID principles

Understanding of Agile Principles (Scrum)

Key Skills

Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala

Employment Type : Full Time

Experience : years

Vacancy : 1