Data Engineer - HadoopGTN Technical Staffing • New York, NY, United States
Data Engineer - Hadoop
GTN Technical Staffing • New York, NY, United States
[job_card.variable_days_ago]
[job_preview.job_type]
- [job_card.permanent]
[job_card.job_description]Strong hands-on experience with Hadoop administration , ideally within Cloudera environments. Proficiency with Python , particularly for automation and data workflows. Experience with Apache Spark (supporting jobs, tuning performance, understanding resource usage). Solid understanding of Linux / Unix systems administration , shell scripting, permissions, networking basics, and OS-level troubleshooting. Experience supporting distributed compute environments or large-scale data platforms. Familiarity with DevOps collaboration (patching, upgrades, deployments, incident response, etc.). Excellent communication skills with the ability to work directly with data scientists and technical end users. Ability to coordinate with multiple technical teams (DevOps, Infrastructure, ML Ops). Strong troubleshooting and problem-solving capabilities. Ability to manage multiple priorities in a fast-moving environment. Experience with ML Ops environments or supporting machine learning workflows. Experience with cluster performance optimization and capacity planning. Background in distributed systems or data engineering.
Data Engineer – Hadoop Administrator
HIGHLIGHTS
Location : Chicago, IL / New York, NY / Phoenix, AZ (Hybrid)
Position Type : Direct Hire
Compensation : BOE
Overview
We are seeking a Data Engineer to support Newton , our Data Science R&D compute cluster. This role functions as a Hadoop Administrator embedded within the ML Ops organization, providing hands-on operational support for the platform while partnering directly with data scientists, DevOps, and infrastructure teams. This individual will ensure the health, stability, performance, and usability of the Newton cluster, acting as the primary point of contact for platform support, troubleshooting, and environment optimization.
This is a highly collaborative and technical role with room for long-term career progression.
Key Responsibilities
- Serve as the primary administrator for the Newton Hadoop / Cloudera cluster.
- Provide direct support to data scientists experiencing issues with jobs, workloads, dependencies, cluster resources, or environment performance.
- Troubleshoot complex Hadoop, Spark, Python, and OS-level issues; drive root cause analysis and implement permanent fixes.
- Coordinate closely with DevOps to ensure patching, upgrades, infrastructure changes, and system reliability activities are completed on schedule.
- Monitor cluster performance, capacity, and resource utilization; tune and optimize for efficiency and cost.
- Manage Hadoop and Cloudera configurations, services, security, policies, and operational health.
- Implement automation and scripting to improve operational workflows and reduce manual intervention.
- Validate vendor patches, updates, and upgrades and coordinate deployments with DevOps and infrastructure teams.
- Maintain documentation, operational runbooks, troubleshooting guides, and environment standards.
- Serve as a liaison between Data Science, ML Ops, Infrastructure, and DevOps teams to ensure seamless platform operations.
- Support the organization’s commitment to protecting the integrity, availability, and confidentiality of systems and data.
Required Technical Skills
Required Soft Skills & Competencies
Preferred Skills (Nice to Have)
[job_alerts.create_a_job]
Data Engineer • New York, NY, United States
[internal_linking.related_jobs]
Trillium is a leading proprietary trading firm active in US Equities, US Options, Canadian Equities, and OTC Equities.With trading technology built, tested, and optimized in-house, our engineers an...[show_more]
At January, we're fixing what's broken in credit.Our data-driven platform rebuilds trust, delivers results, and helps millions move toward brighter financial futures while bringing humanity to cons...[show_more]
Anthropic’s mission is to create reliable, interpretable, and steerable AI systems.We want AI to be safe and beneficial for our users and for society as a whole.
Our team is a quickly growing group ...[show_more]
At Ladders, we’re redefining how professionals find better, more rewarding careers.With over a million job listings and a mission to make the job search smarter, faster, and more effective, we leve...[show_more]
Water Street, 7th Floor, New York, NY 10004 .New Yorkers by uniting communities through care.We believe that Health care is a right, not a privilege.
If you have compassion and a collaborative sp...[show_more]
A company is looking for a Databricks Data Engineer to design and operate scalable data workflows on the Databricks platform.
Key Responsibilities Build, optimize, and maintain batch and streaming...[show_more]
Hinge is the dating app designed to be deleted.In today's digital world, finding genuine relationships is tougher than ever.
At Hinge, we’re on a mission to inspire intimate connection to create a l...[show_more]
Founded in 2012, IEX launched a new kind of securities exchange in 2016 that combines a transparent business model and unique architecture designed to protect investors.
Today, IEX applies its propr...[show_more]
AI research organization with two mutually reinforcing goals.This means to establish the mathematical principles of what it means to reason, to learn, to make decisions, to understand, and to expla...[show_more]
The Cloud Data Engineer is part of a Data team that is responsible for supporting, modernizing, and transforming our data and reporting capabilities across our products by implementing a new modern...[show_more]
Labs is an applied research and AI security company trusted by AI unicorns, Fortune 10 companies, and U.We combine proprietary technology, deep expertise, and multilingual threat intelligence to de...[show_more]
The Daily Beast delivers award-winning original reporting and sharp opinions in politics, pop culture, and world news.We reach more than 20 million readers per month and are based in New York as an...[show_more]
Coast is re-imagining the trillion-dollar U.B2B card payments infrastructure, with a focus on the country’s 500,000 commercial fleets, 40 million commercial vehicles, and many million commercial dr...[show_more]
UJA-Federation of NY is seeking a technical, data-savvy.Senior Data Engineer, Digital Optimization.This is a unique opportunity to shape the future of donor intelligence and use advanced technology...[show_more]
Healthcare should work for patients, but it doesn’t.In their time of need, they call down outdated insurance directories.
Then wait weeks for the privilege of a visit.Then wait in a room solely desi...[show_more]
PLUM is a fintech company empowering financial institutions to grow their business through a cutting-edge suite of AI-driven software, purpose-built for lenders and their partners across the financ...[show_more]
Our roster has an opening with your name on it.At FanDuel, our data platform underpins critical business operations that need to run with maximum efficiency and scalability.As our business continue...[show_more]
Over the next few years, every financial institution will have teams of AI analysts working alongside their sharpest minds.
At Endex, we're on a mission to bridge the present to the inevitable by bu...[show_more]
Data Engineer
Open Roles • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Senior Data Engineer
January • New York, New York, United States
[job_card.full_time] +1
[last_updated.last_updated_30] • [promoted]
Analytics Data Engineer
Anthropic • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
AI Data Engineer
Ladders • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
MetroPlus Health Plan • New York, NY, United States
[job_card.full_time] +1
[last_updated.last_updated_30] • [promoted]
Databricks Data Engineer
VirtualVocations • Jersey City, New Jersey, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer III
Match Group • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Iex Group • New York, NY, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer, Platform
Basis Research Institute • New York, NY, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Cloud Data Engineer
Govx • New York, New York, United States
[filters.remote]
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]
Data Engineer
10a Labs • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
The Daily Beast • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Coast • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Senior Data Engineer, Digital Optimization
Uja Careers • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Senior Data Engineer, Data Infrastructure
Zocdoc • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Senior Data Engineer
Plum Inc • New York, New York, United States
[filters.remote]
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Fanduel • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Endex • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]