Allied Reliability is seeking a visionary Principal Data Engineer to lead the delivery of innovative data analytics solutions, with a focus on technical excellence, governance, and operational performance in a high-volume data environment. This role involves strategic and technical leadership in building Azure-based data lake solutions, driving maturity in data management, and engaging with cross-functional teams to leverage data for organizational success.
*Candidates must be local to Houston, TX and authorized to work in the US without sponsorship*
|
| Key Responsibilities: |
| Architect, design, and implement scalable Azure-based data lake solutions. |
| Establish roadmaps for capability-building across technologies such as Qlik Replicate, Databricks Delta Live Tables, and Unity Catalog. |
| Provide technical leadership, ensuring adherence to best practices and quality standards. |
| Lead the development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming. |
| Optimize Databricks resources for high-volume data processing without exceeding budget. |
| Oversee the use of Databricks Unity Catalog for data governance, lineage, and metadata management. |
| Develop and enforce robust data governance policies, ensuring compliance, security, and data quality. |
| Monitor data pipeline performance and proactively resolve production issues. |
| Collaborate with cross-functional teams to align project objectives with business goals. |
| Translate technical challenges into actionable insights for executive teams. |
| Mentor and develop a high-performing team skilled in Azure, Databricks, and data streaming technologies. |
Facilitate knowledge transfer sessions to empower internal teams with cutting-edge methodologies.
Education and Experience: |
| Bachelor’s or Master’s degree in IT or a related discipline. | | Proven track record in building data lake architectures and optimizing performance in big data environments. | | | Certification in Azure Data Engineering or Databricks (preferred).
Technical Skills | Strong knowledge of Azure technologies, including ADF, ADLS, Databricks, Stream Analytics, Delta Live Tables, and Synapse Analytics.
Advanced proficiency in SQL and programming languages such as Python, Scala, or Spark-SQL.
Experience in designing and developing scalable ETL pipelines and optimizing data workflows.
Deep understanding of data migration from on-premises to cloud, cloud to cloud.
Familiarity with big data tools like Hadoop, Kafka, and Spark.
Experience with visualization tools such as PowerBI or QlikSense.
Knowledge of AI/ML, NLP, and GenAI concepts.
Familiarity with AGILE and KANBAN methodologies.
#LI-TS1 | | | | | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| . |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
