Data engineer [h1.location_city]

The Senior Data Engineer role will be an expert in developing and building data management processes from a variety of data sources. This involves an advanced ability to actively conduct data modeling, data mapping, data pipeline construction and data integration from disparate sources and targets while utilizing the university enterprise framework of data lakes and stores. The Senior Data Engineer will also participate in requirements gathering, documenting, collecting data definitions, source to target cataloging, and defining and implementing data transformation, quality checking, cleansing, and standardization processes. Additionally, this role is responsible for testing the resulting processes for accuracy and performance.

Although this is an individual contributor role, this is a highly collaborative agile team environment requiring personal flexibility in performing a variety of roles such as analyst, designer, developer and tester,; all to take the necessary steps to ensure our partners' needs are met to the maximum extent possible in an accurate and timely manner.

Essential Functions :

• Create and maintain data pipelines, job, etc. for extracting, transforming, and loading data from diverse sources (e.g., databases, APIs, flat files).
• Extract, Transform, Load (ETL) : ETL / IPAAS processes are the backbone of data pipelines. Extract data from various sources (e.g., APIs, databases), transform it (cleaning, aggregating, enriching), and load it into storage systems (data warehouses, databases). Ensure data flows smoothly, adhering to best practices and industry standards.
• Monitoring : Evaluation of code and performance to see if there are opportunities for improvement.
• Automation : Writing code (Python, SQL, or other languages) to automate data management tasks ensures consistent and reliable data flow. Schedule jobs, handle error handling, and monitor pipeline performance.
• Data Quality : Implementing data quality checks ensures that the data is accurate, complete, and consistent. Define rules (e.g., missing values, outliers) and monitor adherence.
• Deliver support / troubleshooting for operational data integration issues. Debug and provide solutions for performance items related to data flows
• Test conduct unit, integration, and end-to-end tests for data quality. Data Engineers prepare test data, set up test environments, and validate all components of ETL (Extract, Transform, Load) pipelines and data flows to maintain system integrity.
• Document Create or maintain comprehensive documentation is required for data processes, pipelines, architecture, and metadata. Within the prevailing standard that team members and stakeholders can understand the structure and logic of data systems and supports scalability and troubleshooting
• Mapping conduct mapping of data sources to target systems. Describe how data fields correspond between systems, supporting migration, integration, and transformation efforts
• Business Requirements work closely with stakeholders to translate business needs into technical requirements. This includes understanding domain-specific needs, designing corresponding data solutions, and ensuring pipelines and storage meet objectives
• Modeling Collaborate with data architects and stakeholders to create / build robust data schemas, and structures with business requirements
• As business needs require, actively being a resource for team members to answer questions and provide solutions for data integration outside their primary responsibility
• Demonstrate active membership in internal / external organizations like the Association of Jesuit Colleges and Universities (AJCU)
• Provide effective communication on matters relating to the job description
• Be an active contributor to onboarding and ongoing training of team members
• Remains current in the field of expertise to ensure assigned applications remain at the industry forefront.

Qualifications :

Knowledge, Skills, and Abilities :