Lead Data Architect (Sunnyvale)

Fractal is looking for a proactive and driven AWS Lead Data Architect / Engineer to join our cloud and data tech team. In this role, you will work on designing the system architecture and solution, ensuring the platform is scalable while performant, and creating automated data pipelines.

Responsibilities :

• Design & Architecture of Scalable Data Platforms
• Design, develop, and maintain large-scale data processing architectures on the Databricks Lakehouse Platform to support business needs
• Architect multi-layer data models including Bronze (raw), Silver (cleansed), and Gold (curated) layers for various domains (e.g., Retail Execution, Digital Commerce, Logistics, Category Management).
• Leverage Delta Lake, Unity Catalog, and advanced features of Databricks for governed data sharing, versioning, and reproducibility.
• Client & Business Stakeholder Engagement
• Partner with business stakeholders to translate functional requirements into scalable technical solutions.
• Conduct architecture workshops and solutioning sessions with enterprise IT and business teams to define data-driven use cases
• Data Pipeline Development & Collaboration
• Collaborate with data engineers and data scientists to develop end-to-end pipelines using Python, PySpark, SQL
• Enable data ingestion from diverse sources such as ERP (SAP), POS data, Syndicated Data, CRM, e-commerce platforms, and third-party datasets.
• Performance, Scalability, and Reliability
• Optimize Spark jobs for performance tuning, cost efficiency, and scalability by configuring appropriate cluster sizing, caching, and query optimization techniques.
• Implement monitoring and alerting using Databricks Observability, Ganglia, Cloud-native tools
• Security, Compliance & Governance
• Design secure architectures using Unity Catalog, role-based access control (RBAC), encryption, token-based access, and data lineage tools to meet compliance policies.
• Establish data governance practices including Data Fitness Index, Quality Scores, SLA Monitoring, and Metadata Cataloging.
• Adoption of AI Copilots & Agentic Development
• Utilize GitHub Copilot, Databricks Assistant, and other AI code agents for
• Writing PySpark, SQL, and Python code snippets for data engineering and ML tasks.
• Generating documentation and test cases to accelerate pipeline development.
• Interactive debugging and iterative code optimization within notebooks.
• Advocate for agentic AI workflows that use specialized agents for
• Data profiling and schema inference.
• Automated testing and validation.
• Innovation and Continuous Learning
• Stay abreast of emerging trends in Lakehouse architectures, Generative AI, and cloud-native tooling.
• Evaluate and pilot new features from Databricks releases and partner integrations for modern data stack improvements.

Requirements :