Big Data Engineer

Software Guidance & Assistance, Inc., (SGA), is searching for a Big Data Engineer for a Contract assignment with one of our premier Regulatory clients in Rockville, MD or Tysons, VA.

This role is hybrid ( days / week Onsite in either Rockville or Tysons office)

Responsibilities :

• Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (, Hadoop, Spark, Python, Scala).
• Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable.
• Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Optimize and enhance existing data pipelines for performance, scalability, and reliability.
• Develop automated testing frameworks and implement continuous testing for data quality assurance.
• Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
• Work with data scientists and analysts to support data-driven decision-making across the organization.
• Ability to write and maintain automated unit, integration, and end-to-end tests
• Monitor and troubleshoot data pipelines in production environments to identify and resolve issues.

Requirements :

Essential Technical Skills :

Big Data technologies

◦ Data skew and strategies to mitigate it.

◦ Working with massive data volumes in PetaBytes.

◦ Troublehsooting job failures due to resource limitations, bad data, scalability challenged.

AI Skills

SQL Skills (Window Functions, Joins, Complex Queries)

Apache Spark (Development, Internals & Tuning)

Cloud Technologies

Programming - Python or Scala

Preferred Skills :