Java Tech lead ( Nifi, Airflow)

ROLE : Java Tech lead ( Nifi, Airflow)

Description :

• Strong NiFi Expertise : Deep understanding of Apache NiFi architecture, processors, controllers, and data flow management capabilities.
• Programming Languages : Proficiency in Java, Spring boot & API Microservices.
• Database and SQL : Familiarity with databases, SQL, and data manipulation techniques.
• Big Data Technologies : Kafka ( Event driven & streaming architecture)
• Apachae Airflow - Workflow orchestration"
• Design and Implementation :

o Creating and implementing data flow solutions using NiFi, including defining data sources, destinations, and transformation processes.

o Managing data ingestion from various sources (e.g., files, databases, APIs) and transforming data into desired formats for downstream consumption.

o Configuring NiFi to route data to appropriate destinations and enriching data with additional information (e.g., timestamps, location).

o Ensuring data flows are scalable and performant to handle varying data volumes and velocity.

o Maintaining and improving NiFi clusters, including configuration, optimization, and troubleshooting.

o Developing and integrating microservices to support data conditioning, format validation, and transformation processes.

o Identifying and resolving issues related to data flows, NiFi configurations, and data integration processes

o Possessing deep knowledge of Airflow's architecture, including schedulers, executors (Celery, Kubernetes), and plugin development.

o Designing and developing complex, modular, and reusable DAGs (Directed Acyclic Graphs) to automate data pipelines.

o Identifying and addressing performance bottlenecks in Airflow environments and implementing best practices for orchestration and scheduling