Data EngineerQloo • New York City, New York, USA
Data Engineer
Qloo • New York City, New York, USA
[job_card.variable_days_ago]
[job_preview.job_type]
- [job_card.full_time]
[job_card.job_description]B achelors degree in Computer Science Software Engineering or a related field or equivalent practical experience. Experience with Python and distributed data processing using Spark (PySpark) on EMR or a similar environment. Hands-on experience with core AWS data services ideally including : S3 (data lake partitioning lifecycle management) AWS Glue (jobs crawlers catalogs) EMR or other managed Spark platforms Athena / Hive and SQL for querying large datasets Relational databases such as RDS (PostgreSQL / MySQL or similar) Experience building and operating workflows in Airflow (MWAA experience is a plus). Strong SQL skills and familiarity with data modeling concepts for analytics and APIs. Solid understanding of data quality practices (testing validation frameworks monitoring / observability). Comfortable working in a collaborative environment managing multiple projects and owning systems end-to-end. Competitive salary and benefits package including health insurance retirement plan and paid time off. The opportunity to shape a modern cloud-based data platform that powers real products and ML experiences. A collaborative low-ego work environment where your ideas are valued and your contributions are visible. Flexible work arrangements (remote and hybrid options) and a healthy respect for work-life balance.
About Us
At Qloo we harness large-scale behavioral and catalog data to power recommendations and insights across entertainment dining travel retail and more. Our platform is built on a modern AWS data stack and supports analytics APIs and machine-learning models used by leading brands. We are looking for an experienced Data Engineer to help evolve and scale this platform.
Role Overview
As a Data Engineer at Qloo you will design build and operate the pipelines that move data from external vendors internal systems and public sources into our S3-based data lake and downstream services. Youll work across AWS Glue EMR (Spark) Athena / Hive and Airflow (MWAA) to ensure that our data is accurate well-modeled and efficiently accessible for analytics indexing and machine-learning workloads.
You should be comfortable owning end-to-end data flows from ingestion and transformation to quality checks monitoring and performance tuning.
Responsibilities
- Design develop and maintain batch data pipelines using Python Spark (EMR) and AWS Glue loading data from S3 RDS and external sources into Hive / Athena tables.
- Model datasets in our S3 / Hive data lake to support analytics (Hex) API use cases Elasticsearch indexes and ML models.
- Implement and operate workflows in Airflow (MWAA) including dependency management scheduling retries and alerting via Slack.
- Build robust data quality and validation checks (schema validation freshness / volume checks anomaly detection) and ensure issues are surfaced quickly with monitoring and alerts.
- Optimize jobs for cost and performance (partitioning file formats join strategies proper use of EMR / Glue resources).
- Collaborate closely with data scientists ML engineers and application engineers to understand data requirements and design schemas and pipelines that serve multiple use cases.
- Contribute to internal tooling and shared libraries that make working with our data platform faster safer and more consistent.
- Document pipelines datasets and best practices so the broader team can easily understand and work with our data.
Qualifications
We Offer
We may use artificial intelligence (AI) tools to support parts of the hiring process such as reviewing applications analyzing resumes or assessing responses. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed please contact us.
Required Experience :
IC
Key Skills
Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala
Employment Type : Full-Time
Department / Functional Area : Engineering
Experience : years
Vacancy : 1
[job_alerts.create_a_job]
Data Engineer • New York City, New York, USA
[internal_linking.similar_jobs]
Trillium is a leading proprietary trading firm active in US Equities, US Options, Canadian Equities, and OTC Equities.With trading technology built, tested, and optimized in-house, our engineers an...[show_more]
Fira Health builds AI-native infrastructure to automate the administrative backbone of healthcare.We’re starting with home health, helping agencies reduce overhead, accelerate payments, and focus m...[show_more]
A company is looking for a Data Engineer with 2-5 years of experience in a GCP-based big data environment.Key Responsibilities Build, maintain, and support data pipelines in Google Cloud Platform...[show_more]
Aircall is a place where voices are valued.Backed by over $220 million of investment since 2015, we create technology that fuels accessible, transparent and collaborative communication to empower o...[show_more]
Hinge is the dating app designed to be deleted.In today's digital world, finding genuine relationships is tougher than ever.
At Hinge, we’re on a mission to inspire intimate connection to create a l...[show_more]
Founded in 2012, IEX launched a new kind of securities exchange in 2016 that combines a transparent business model and unique architecture designed to protect investors.
Today, IEX applies its propr...[show_more]
We're looking for a motivated Data Engineer to help build and scale our cloud-native data pipelines using Snowflake and dbt.
This role is ideal for someone who enjoys solving data challenges, improv...[show_more]
AI research organization with two mutually reinforcing goals.This means to establish the mathematical principles of what it means to reason, to learn, to make decisions, to understand, and to expla...[show_more]
Prizeout is a fast-growing fintech transforming how people interact with their money by turning everyday transactions into rewarding, value-driven moments.
We sit at the intersection of payments, re...[show_more]
Kasheesh is the first and only product that lets consumers split their payments across multiple credit, debit, and gift cards.
Kasheesh is on an exciting mission to drive unparalleled flexibility fo...[show_more]
At Medal, we’re redefining how people capture and share gameplay experiences.Every day, our platform ingests tens of thousands of hours of gameplay video—raw, unfiltered, and packed with insights.W...[show_more]
Labs is an applied research and AI security company trusted by AI unicorns, Fortune 10 companies, and U.We combine proprietary technology, deep expertise, and multilingual threat intelligence to de...[show_more]
Design, develop, and maintain ETL pipelines using AWS Glue, Glue Studio, and Glue Catalog.Ingest, transform, and load large datasets from structured and unstructured sources into AWS data lakes / war...[show_more]
The Daily Beast delivers award-winning original reporting and sharp opinions in politics, pop culture, and world news.We reach more than 20 million readers per month and are based in New York as an...[show_more]
Coast is re-imagining the trillion-dollar U.B2B card payments infrastructure, with a focus on the country’s 500,000 commercial fleets, 40 million commercial vehicles, and many million commercial dr...[show_more]
We are Rokt, a hyper-growth ecommerce leader.Rokt is the global leader in ecommerce, unlocking real-time relevance in the moment that matters most.
Rokt’s AI Brain and ecommerce Network powers billi...[show_more]
Over the next few years, every financial institution will have teams of AI analysts working alongside their sharpest minds.
At Endex, we're on a mission to bridge the present to the inevitable by bu...[show_more]
Judi Health is an enterprise health technology company providing a comprehensive suite of solutions for employers and health plans, including : .
PBM) solutions to self-insured employers,.Enterprise H...[show_more]
Data Engineer
Open Roles • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Fira Health • New York, New York, United States
[filters.remote]
[job_card.full_time]
[last_updated.last_updated_variable_hours] • [promoted] • [new]
Data Engineer
VirtualVocations • Bronx, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer - NYC
Aircall • New York, NY, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer III
Match Group • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Iex Group • New York, NY, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Bluematrix • New York, NY, United States
[filters.remote]
[job_card.full_time]
[last_updated.last_updated_variable_hours] • [promoted] • [new]
Data Engineer, Platform
Basis Research Institute • New York, NY, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Prizeout • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Kasheesh • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer Clips / ML Data
Medal • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
10a Labs • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Qode • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_1_day] • [promoted]
Data Engineer
The Daily Beast • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Coast • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Rokt • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer
Endex • New York, New York, United States
[job_card.full_time]
[last_updated.last_updated_30] • [promoted]
Data Engineer II
Capital Rx • New York, NY, United States
[job_card.full_time]
[last_updated.last_updated_variable_days] • [promoted]