Principal Data Engineer - Canada Life : Job Details

Location: London, Potters Bar, Bristol or Isle of Man (Hybrid working options available)

The Principal Engineer will provide expert-level technical leadership in Azure cloud and Databricks-based solutions. The role requires deep expertise in Azure cloud services, Databricks, data engineering, and DevOps, leading a cross-functional team to build, deploy, and support high-performance data-driven solutions.

The role involves:

• Designing and implementing scalable data solutions on Azure and Databricks.
• Developing and optimizing data pipelines and ETL processes using Azure Data Factory, Databricks, and Synapse Analytics.
• Building and managing CI/CD pipelines for data applications with Azure DevOps.
• Ensuring compliance with security best practices and governance frameworks in cloud and data solutions.
• Driving cloud modernization initiatives, including migrating on-prem data workloads to Azure.
• Leading engineering best practices to ensure data quality, performance, and reliability.

Duties/Responsibilities

• Develop and optimize data pipelines and ETL workflows in Databricks (PySpark, Delta Lake).
• Implement data lake architectures using Azure Data Lake, Delta Lake, and Synapse Analytics.
• Build machine learning and real-time analytics solutions leveraging Databricks MLflow and Azure AI services
• Develop CI/CD pipelines for data applications using Azure DevOps, Terraform, and Bicep.
• Automate infrastructure provisioning and infrastructure-as-code (IaC) using Terraform or Bicep.
• Implement monitoring & logging solutions with Azure Monitor, Datadog, or Prometheus.
• Enforce security, access control, and compliance in Azure and Databricks.
• Apply cost optimization strategies for efficient cloud resource usage.
• Ensure adherence to data governance and lineage using Purview and Databricks Unity Catalog.
• Lead code reviews, knowledge sharing, and mentoring within the engineering team.
• Ensure their assigned team, work within organisational guardrails and meet engineering standards and best practice. They are collectively responsible for the quality of applications in their eco-system.
• Experiment with new Azure and Databricks features to improve solution architecture.
• Principal engineers help the team develop and implement DevOps practices, balancing build and run responsibilities, removing technical debt and making their applications more automated and reliable over time through removal of toil.

Skills, Knowledge and Experience

Principal Engineers are expected to have expertise in at least three of the following areas of Engineering practices

Core skills

• Automation including Testing
• Possess strong teamwork, communication, and problem-solving skills to ensure they can collaborate effectively with cross-functional teams and bring out better results.
• Security Best Practices: Awareness of security principles and best practices to ensure secure software delivery and infrastructure management.
• Continuous Learning: Keeping up-to-date with the rapidly evolving technologies and trends in the industry.
• Monitoring and Troubleshooting: Ability to set up monitoring and logging solutions and troubleshoot issues effectively.
• Strong experience working within an agile development methodology
• Excellent time and self-management through effective planning & prioritisation of tasks.
• Proven and demonstrable engineering capability
• Ability to influence others and strong communication skills

Data Engineer (New Technology / Microsoft)

• Expertise in Databricks (Spark, PySpark, Delta Lake, MLflow, Unity Catalog).
• Strong proficiency in Azure Cloud (Azure Data Factory, Synapse, Data Lake, Functions).
• Experience with CI/CD pipelines (Azure DevOps, GitHub Actions, Terraform, Bicep).
• Hands-on experience with SQL and NoSQL databases (Azure SQL, CosmosDB).
• Scripting & Programming skills (Python, Scala, PowerShell).
• Deep understanding of DevOps & automation (Terraform, ARM, YAML pipelines).
• Strong understanding of cloud security, compliance, and governance.