Talent.com
SGA
Big Data EngineerSGA • Rockville, MD
Big Data Engineer

Big Data Engineer

SGA • Rockville, MD
30+ days ago
Job type
  • Full-time
Job description

Software Guidance & Assistance, Inc., (SGA), is searching for a Big Data Engineer for a Contract assignment with one of our premier Regulatory clients in Rockville, MD or Tysons, VA.

This role is hybrid ( days/week Onsite in either Rockville or Tysons office)

Responsibilities :
  • Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (, Hadoop, Spark, Python, Scala).
  • Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable.
  • Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
  • Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
  • Optimize and enhance existing data pipelines for performance, scalability, and reliability.
  • Develop automated testing frameworks and implement continuous testing for data quality assurance.
  • Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipelines.
  • Work with data scientists and analysts to support data-driven decision-making across the organization.
  • Ability to write and maintain automated unit, integration, and end-to-end tests
  • Monitor and troubleshoot data pipelines in production environments to identify and resolve issues.
Requirements :
  • Bachelor's degree in Computer Science, Information Systems or related discipline with at least five () years of related experience, or equivalent training and/or work experience; Master's degree and past Financial Services industry experience preferred.
  • Demonstrated technical expertise in Object Oriented and database technologies/concepts which resulted in deployment of enterprise quality solutions.
  • Past experience with developing enterprise quality solutions in an iterative or Agile environment.
  • Extensive knowledge of industry leading software engineering approaches including Test Automation, Build Automation and Configuration Management frameworks.
  • Strong written and verbal technical communication skills.
  • Demonstrated ability to develop effective working relationships that improved the quality of work products.
  • Should be well organized, thorough, and able to handle competing priorities.
  • Ability to maintain focus and develop proficiency in new skills rapidly.
  • Ability to work in a fast paced environment.
  • Experience with object oriented programming languages such as Java, Scala or Python.
Essential Technical Skills:

Big Data technologies
• Experience with Big data technologies such as Hadoop, Spark, Hive & Trino
• Evaluate understanding of common issues like:
◦ Data skew and strategies to mitigate it.
◦ Working with massive data volumes in PetaBytes.
◦ Troublehsooting job failures due to resource limitations, bad data, scalability challenged.
• Look for real-world debugging and mitigation stories.

AI Skills
• Prompt Engineering: Proficiency in crafting effective prompts for AI coding assistants and analysis tools
• AI Workflow Design: Experience redesigning development processes to leverage AI capabilities
• Data Analysis: Ability to interpret AI-generated insights and translate them into actionable team improvements
• Change Management: Experience leading teams through AI adoption and workflow transformation

SQL Skills (Window Functions, Joins, Complex Queries)
• Assess comfort with SQL window functions, multi-table joins, aggregations.
• Provide examples or ask them to write/optimize SQL queries on the spot.
• Probe how they handle edge cases like NULLs, duplicates, ordering, etc.

Apache Spark (Development, Internals & Tuning)
• Test their understanding of Spark's core architecture — executors, tasks, stages, DAG.
• Focus on Spark performance tuning techniques: partitioning, caching, broadcast joins, etc.
• Ask scenario-based questions on troubleshooting slow running/stuck jobs or resource issues in Spark.
• Explore their experience optimizing Spark jobs for large-scale datasets.

Cloud Technologies
• Check exposure to AWS services like S, EMR, Glue, Lambda, Athena, etc.
• Ask how they've used S with Spark (, dealing with file formats, consistency issues).
• EKS, Serverless knowledge, etc.

Programming - Python or Scala
• Assess ability to write clean, modular, and performant code.
• Look for experience in functional programming concepts (, immutability, higher-order functions).
• Ask about real-world use cases where they wrote scalable data processing code.
• Evaluate understanding of collections, concurrency, and memory management.

Preferred Skills:
• Experience with managing production data pipelines/ETL systems
• Experience with CI/CD
• Experience writing test cases
• AWS certifications

Create a job alert for this search

Big Data Engineer • Rockville, MD

Similar jobs

Immediate interview for Azure Data Engineer 100% Remote EST Hours

Saxon GlobalWashington, DC, United States
Remote
Full-time

Azure Data Engineer PositionHello, I am Mohammed Dastagir with Saxon Global Inc wanted to let you know about the job opportunity for Azure Data Engineer position if interested please share your upd... Show more

 • Promoted

Spark Data Engineer (Remote)

Zachary PiperWashington, DC, United States
Remote
Full-time

Zachary Piper Solutions is seeking an Spark Data Engineer (DoD Secret) to support a variety of federal customer.This role focuses on big data engineering, optimization, and performance tuning using... Show more

 • Promoted

AI Data Scientist

Glint Tech Solutions LLCColumbia, MD, USA
Full-time
Quick Apply

Glint Tech Solutions is seeking an experienced AI Data Scientist to lead enterprise-wide Machine Learning and Artificial Intelligence initiatives for a healthcare-focused client.This is a contract-... Show more

Senior Data Engineer (Remote)

ExperianWashington, DC, United States
Remote
Full-time

Senior Data Engineer (Remote) Experian is a global data and technology company, powering opportunities for people and businesses around the world.We help to redefine lending practices, uncover and ... Show more

 • Promoted

Senior Data Engineer, Snowflake Expert (Hybrid/Remote)

Blue YonderWashington, DC, United States
Remote
Full-time

A leading supply chain technology firm is seeking a Sr Staff Data Engineer with expertise in Snowflake to join their team.This role involves building data ingestion pipelines and designing scalable... Show more

 • Promoted

Secret-Cleared Data Engineer

PTF Consulting, LLCWashington D.C., District of Columbia, United States
Full-time +1
Quick Apply

Direct Hire |DC Metro Area | Active Secret Clearance Required.PTF Consulting is supporting direct-hire opportunities for.Junior and Senior Data Engineers.In these roles, you will help organize... Show more

Data Scientist

TradeJobsWorkForce22245 Arlington, VA, US
Full-time

Data Scientist Job Duties: Formulates and leads guided, multifaceted analytic studies agains... Show more

 • Promoted

Powerbi Developer

TEKsystemsColumbia, MD, US
Full-time

Our healthcare client is looking for a Data Analyst to join there Data Group.For this role, 80% of the time will be dedicated to PowerBI experience.The need for data modeling within Power BI, not j... Show more

 • Promoted

Senior Data Center Design Engineer - Remote Fed Projects

Iron MountainWashington, DC, United States
Remote
Full-time

A global leader in storage solutions is looking for a Senior Project Engineer for Data Center Design.This client-facing role involves managing the design process for federal and special projects va... Show more

 • Promoted

Sr. Data Analyst (Federal Healthcare SME)

Glint Tech Solutions LLCColumbia, Maryland, USA
Full-time
Quick Apply

Job Title: Senior Data Analyst – Healthcare / CMS Stars.Location: Primarily Remote (2–3 onsite visits/month in Columbia, MD).Our client is seeking a Senior Data Analyst to support enterprise analyt... Show more

Lead Software Engineer | Remote

Stellent IT LLCColumbia, MD, United States
Remote
Full-time

Hi, My name is Avnish Rao and I am a Staffing Specialist at Stellent IT LLC.I am reaching out to you on an exciting job opportunity with one of our clients.Lead Software Engineer Location:Hybrid; 1... Show more

 • Promoted

Enterprise Data Architect (Hybrid OR Remote- Jersey City, Raleigh, or Hartford)

Arch Capital Group Ltd.Washington, DC, United States
Remote
Full-time

Enterprise Data Architect (Hybrid OR Remote- Jersey City, Raleigh, or Hartford)1 day ago - Be among the first 25 applicants.Base pay range$122,000 / yr - $149,000 / yrWith a company culture rooted ... Show more

 • Promoted

Distinguished Engineer (Remote Eligible)

Capital oneColumbia, MD, United States
Remote
Full-time +1

Director, Technical Program Management - AI / MLAre you interested in leading technical AI and ML programs that deliver on critical business goals and build large scale products & platforms?Abo... Show more

 • Promoted

Remote Data Analyst -- TS Cleared for Big Data & Visualization

Elder Research Inc.Arlington, VA, United States
Remote
Full-time

A leading data analytics firm is seeking an experienced Data Analyst to support U.In this role, you will handle data manipulation, infrastructure for data ingestion, and create visualizations.Candi... Show more

 • Promoted

Staff ML Engineer, Fraud Data - Remote

PLAIDWashington, DC, United States
Remote
Full-time

A leading fintech company is seeking a Staff Machine Learning Engineer for its Fraud Data team.This role involves designing scalable ML infrastructure and leading the evolution of model deployment ... Show more

 • Promoted

Chief Data Scientist

ConfidentialWashington, District of Columbia, US
Full-time

The role of Chief Data Scientist requires an AI leader with experience in building production-grade language model platforms within federal spaces.Is this your next job Read the full description be... Show more

 • Promoted

Data Engineer (Remote)

Authority BrandsColumbia, MD, United States
Remote
Full-time

Columbia, MD is a leading provider of home services, building brands that support the success of our franchisees, as well as bettering the lives of the homeowners we serve and the people we employ.... Show more

 • Promoted

Remote Federal Data Engineer - NIST Compliant Analytics

PeratonBethesda, MD, United States
Remote
Full-time

A technology services company is looking for a skilled IT Data Engineer to design and maintain secure data solutions compliant with federal standards.The role requires extensive experience in data ... Show more

 • Promoted

Bioinformatics Data Engineer Remote with Impact

Ariadne Genomics IncBethesda, MD, United States
Remote
Full-time

An innovative firm is seeking a Data Engineer to join their team in Bethesda, MD, with the flexibility of remote work.This full-time position involves supporting critical data operations for the Na... Show more

 • Promoted

Manager, Data Engineering

hackeroneWashington, DC, US
Full-time

HackerOne is a global leader in Continuous Threat Exposure Management (CTEM).The HackerOne Platform unites agentic AI solutions with the ingenuity of the world's largest community of security resea... Show more