Big data engineer Jobs in North Charleston, SC

KORE1, a nationwide provider of staffing and recruiting solutions, has an immediate opening for a Data Engineer that is based in Charleston, SC. Remote /hybrid flexibility. Position Overview We are seeking a highly capable Data Engineer to design and build the data infrastructure that powers investment operations, reporting, and business intelligence capabilities. This is a foundational role within a firm actively modernizing its data architecture by replacing fragmented automations with scalable, production-grade pipelines and establishing a centralized, reliable source of truth across multiple systems. The Data Engineer will own the end-to-end data stack, including ingestion, transformation, modeling, data quality, and governance. This role requires deep technical expertise along with the ability to navigate a complex, multi-system financial data environment. This position will play a central role in supporting the organization's evolving AI and analytics initiatives. The ideal candidate will help structure, govern, and optimize warehouse data to support future AI-enabled workflows and advanced reporting capabilities. The Data Engineer will work closely with cross-functional teams including Operations, IT, business stakeholders, and external technology partners to ensure the data platform evolves in alignment with operational and strategic needs. Key Responsibilities Architecture & Infrastructure * Evaluate and recommend scalable data architecture solutions, including warehouse and lakehouse strategies. * Build and maintain a centralized cloud data warehouse environment. * Design and maintain scalable ingestion pipelines across multiple operational and financial systems. * Develop data structures and governance models that support analytics, reporting, and future AI-driven workflows. * Ensure the data platform is designed with long-term scalability, security, and accessibility in mind. Data Modeling & Transformation * Design and implement canonical data models across financial and operational datasets. * Write and optimize complex SQL queries, transformations, and performance-tuned data workflows. * Replace manual or lightweight automations with production-grade ETL/ELT pipelines using Python and cloud-native tooling. Integrations & APIs * Build and maintain REST API integrations across third-party platforms and internal systems. * Develop Python-based ingestion and transformation processes. * Support orchestration and storage within a cloud environment using modern data engineering best practices. Data Quality, Governance & Security * Implement validation frameworks, deduplication logic, monitoring, and exception handling processes. * Configure role-based access controls and maintain auditability across the data environment. * Document and maintain end-to-end data lineage and governance standards. * Support compliance and security requirements for sensitive financial and operational data. Reporting & Documentation * Deliver reporting-ready datasets supporting portfolio analytics, operational reporting, and business intelligence initiatives. * Support BI tool integrations and semantic layer development for technical and non-technical users. * Create and maintain documentation for architecture decisions, pipelines, and data dictionaries. Cross-Functional Collaboration * Partner closely with Operations, IT, and leadership teams to align data infrastructure with business goals. * Participate in ongoing technology and systems planning initiatives. * Act as a key liaison between technical and business stakeholders to ensure data solutions meet organizational needs. Technical Requirements * Advanced Python development for data engineering and automation * Advanced SQL skills including performance optimization and complex transformations * Strong experience with Snowflake or similar cloud data warehouse platforms * Experience designing scalable data models and ETL/ELT pipelines * REST API integration experience including authentication, pagination, and error handling * Cloud platform experience, preferably AWS * Strong understanding of data quality, governance, and access controls * Ability to work cross-functionally in fast-paced environments Preferred Skills * Experience supporting AI or LLM-enabled workflows * Financial services, private equity, asset management, or investment operations experience * Familiarity with CRM, fund administration, or portfolio management platforms * Experience with BI tools such as Tableau, Looker, or Sigma Required Qualifications * Bachelor's degree in Computer Science, Engineering, Mathematics, or a related technical field * 5+ years of experience in data engineering * Strong production experience with Python and SQL * Experience building scalable cloud-based data platforms * Strong documentation and communication skills Preferred * Experience working with financial datasets and investment operations * Exposure to AI-driven analytics or agent-based workflows * Experience implementing enterprise reporting and business intelligence solutions Core Competencies * Ownership and accountability * Strong analytical and systems thinking * Cross-functional collaboration * Clear communication with technical and non-technical stakeholders * Adaptability in fast-paced environments * High standards for governance, accuracy, and compliance Location & Work Environment This position is preferably based in the Southeast United States, though remote candidates may be considered for the right fit. Compensation & Benefits * Competitive compensation package * Performance-based incentives * Comprehensive health and retirement benefits * High visibility with leadership and strategic initiatives * Opportunity to help build foundational infrastructure within a growing organization Compensation depends on experience but is typically 145-160k ABOUT KORE1 Specializing in professional and technical recruiting, KORE1 is committed to supporting top IT, Engineering, Creative, Scientific, Accounting and Finance professionals in their career paths. We build deep relationships with leading companies, connecting them to exceptional talent every day. With extensive industry expertise and unmatched opportunities, our goal is to provide a unique experience for our contractors and consultants as they prepare for their next role. We are passionate about matching the right people with the right companies. Kore1 provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability or genetics. In addition to federal law requirements, Kore1 complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training. Kore1 expressly prohibits any form of workplace harassment based on race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status. Improper interference with the ability of Kore1's employees to perform their job duties may result in discipline up to and including discharge.