Sr. Data Engineer

Title : Sr. Data Engineer

Location: Oregon

Duration: 12 Months+

Roles & Responsibilities

• Perform data analysis according to business needs
• Translate functional business requirements into high-level and low-level technical designs
• Design and implement distributed data processing pipelines using Apache Spark, Apache Hive, Python, and other tools and languages prevalent in a modern analytics platform
• Create and schedule workflows using Apache Airflow or similar job orchestration tooling
• Build utilities, functions, and frameworks to better enable high-volume data processing
• Define and build data acquisitions and consumption strategies
• Build and incorporate automated unit tests, participate in integration testing efforts
• Work with teams to resolve operational & performance issues
• Work with architecture/engineering leads and other teams to ensure quality solutions are implemented, and engineering best practices are defined and followed.

Tech Stack

1. Apache Spark
2. Apache Spark Streaming using Apache Kafka
3. Apache Hive
4. Apache Airflow
5. Python
6. AWS EMR and S3
7. Snowflake
8. SQL
9. Other Tools & Technologies :: PyCharm, Jenkin, Github.
10. Apache Nifi (Optional)
11. Scala (Optional)