Azure Databricks Architect - San Diego, CA

Overview

An Azure Databricks Architect is a technical role focused on designing and implementing scalable data solutions using Azure Databricks, acting as a subject matter expert in the platform to create efficient data pipelines, manage data lakes, and ensure data quality while leveraging the full capabilities of the Azure cloud ecosystem; essentially, they are responsible for architecting robust data solutions on Azure Databricks, considering data ingestion, processing, storage, and analysis needs within the context of an organization's business requirements.

Responsibilities

• Solution Design : Develop and implement data architecture using Azure Databricks, including data ingestion pipelines, data transformation logic, and data warehousing strategies. Build and optimize ETL pipelines, particularly in Databricks using PySpark, Scala, or Spark SQL. Create and manage Databricks notebooks for data exploration, pipeline development, and data visualization. Familiarity with scheduling and automating Databricks jobs.
• Big Data Processing : Design and optimize ETL / ELT pipelines using Apache Spark, Python, Scala, or SQL. Experience building ETL / ELT pipelines, data cleansing, and data transformation processes.
• Data Technologies : Proficiency in setting up, configuring, and managing Azure Databricks clusters, as well as optimizing performance and costs. Understanding of how to manage, process, and store big data in Azure Data Lake or other storage solutions.
• Technical Leadership : Guide development teams, data engineers, and business analysts on best practices for data modeling, data processing, and optimization within the Databricks platform.
• Data Integration : Design and build data integration solutions to connect diverse data sources, including on-premises systems, cloud-based applications, and APIs.
• Performance Optimization : Monitor and optimize data processing performance on Databricks clusters to ensure efficient data processing at scale.
• Security and Governance : Implement data security measures and access controls to protect sensitive data within the Azure Databricks environment. Knowledge of networking concepts within Azure, such as virtual networks, firewalls, private endpoints, and network security groups. Ensure compliance with security and governance best practices, including RBAC, encryption, and data masking.
• Collaboration : Work closely with data engineers, data scientists, business intelligence analysts, and business stakeholders to understand requirements and translate them into technical solutions.
• Compliance : Adhere to company policy and procedure, complete annual compliance training, and comply with business continuity and vendor management policy. These are crucial to the success of the organization and have regulatory compliance and / or safety and soundness for the sustainability of the credit union. Comply with all Anti Money Laundering laws such as the Bank Secrecy Act, OFAC and USA PATRIOT Act and all related policies and procedures including accurate Member / Customer Identification Program, member due diligence, accurate OFAC match processing, and reporting unusual activities and suspected fraud.
• SUPERVISORY RESPONSIBILITIES : Effectively lead data teams, manage complex data projects, and ensure proper implementation of architectural solutions.

Qualifications

EDUCATION and / or EXPERIENCE : Bachelor’s degree and 8+ years related experience required, or equivalent combination of education and experience. Credit union or financial institution experience highly preferred.

OTHER QUALIFICATIONS

BluPeak Credit Union is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex including sexual orientation and gender identity, national origin, disability, protected Veteran Status, or any other characteristic protected by applicable federal, state, or local law.

Employment may be contingent upon BluPeak Credit Union’s receipt of an acceptable and job-related background check, drug screen and credit check, as applicable and permissible by law. BluPeak Credit Union is committed to working and providing reasonable accommodation to applicants with physical, mental disabilities and sincerely held religious beliefs. For more information, contact Human Resources.

PM17

Min

USD $130,000.00 / Yr.

Max

USD $150,000.00 / Yr.

#J-18808-Ljbffr