Data & Analytics Engineer (16)

TIP Data & Analytics is dedicated to driving meaningful impact through data. We achieve this by building resilient, reliable, and scalable data platforms and pipelines that deliver engaging, informative, and actionable data products to internal and external stakeholders. As a Data Engineer in our agile, dynamic department, you will play a crucial role leveraging a modern, fully Azure-based cloud infrastructure. 

You will support our mission by designing and implementing robust data architectures within a medallion (bronze, silver, gold) architecture framework. Your primary responsibility will be developing metadata-driven, test-driven ETL/ELT pipelines using Azure Databricks and Apache Airflow, ensuring efficient extraction, transformation, and loading of data into Delta Lake storage on Azure Data Lake Storage Gen2. 

Your activities will consist of: 

• Designing, developing, and maintaining scalable, metadata-driven data pipelines using Azure Databricks and Airflow for workflow orchestration. 

• Contributing to our lakehouse platform engineering framework, including enhancing reusable, pre-tested code components. 

• Implementing Infrastructure as Code (IaC) solutions using Terraform within Azure DevOps deployment pipelines to automate and manage cloud resources effectively. 

• Developing and managing Delta Lake storage solutions within Azure Data Lake Storage Gen2, optimizing for performance, reliability, and scalability. 

• Applying test-driven development (TDD) practices to ensure pipeline reliability, data integrity, and facilitate continuous integration and deployment. 

• Collaborating closely with internal stakeholders to design source system architectures that enhance data quality and availability. 

• Developing monitoring, alerting, and observability solutions to maintain high availability and performance of data platforms. 

• Actively participating in integrating data from MS Dynamics-based ERP systems. 

TIP Data & Analytics is dedicated to driving meaningful impact through data. We achieve this by building resilient, reliable, and scalable data platforms and pipelines that deliver engaging, informative, and actionable data products to internal and external stakeholders. As a Data Engineer in our agile, dynamic department, you will play a crucial role leveraging a modern, fully Azure-based cloud infrastructure. 

You will support our mission by designing and implementing robust data architectures within a medallion (bronze, silver, gold) architecture framework. Your primary responsibility will be developing metadata-driven, test-driven ETL/ELT pipelines using Azure Databricks and Apache Airflow, ensuring efficient extraction, transformation, and loading of data into Delta Lake storage on Azure Data Lake Storage Gen2. 

Your activities will consist of: 

• Designing, developing, and maintaining scalable, metadata-driven data pipelines using Azure Databricks and Airflow for workflow orchestration. 

• Contributing to our lakehouse platform engineering framework, including enhancing reusable, pre-tested code components. 

• Implementing Infrastructure as Code (IaC) solutions using Terraform within Azure DevOps deployment pipelines to automate and manage cloud resources effectively. 

• Developing and managing Delta Lake storage solutions within Azure Data Lake Storage Gen2, optimizing for performance, reliability, and scalability. 

• Applying test-driven development (TDD) practices to ensure pipeline reliability, data integrity, and facilitate continuous integration and deployment. 

• Collaborating closely with internal stakeholders to design source system architectures that enhance data quality and availability. 

• Developing monitoring, alerting, and observability solutions to maintain high availability and performance of data platforms. 

• Actively participating in integrating data from MS Dynamics-based ERP systems.