Lead Data Engineer - SC Cleared - Outside IR35 - 6 Months - SR2 : Job Details

Lead Data Engineer - SC Cleared - 6 Months - Hybrid - Outside IR35

We are seeking someone with a strong technical background in building Java-based microservices, AWS Glue, PySpark, SQL, and Athena to join our team.In this role, you will design, build, and optimise data pipelines and platforms in a cloud environment.

Due to the nature of the role, active SC clearance is required.

Key Responsibilities:

• Data Pipeline Development: Design, develop, and maintain scalable data pipelines using AWS Glue, PySpark, and Java-based microservices to process large volumes of structured and unstructured data.
• Microservices Development: Build and optimize Java-based microservices to support data integration, transformation, and orchestration for real-time and batch data processing.
• AWS Services Management: Leverage AWS services such as S3, Glue, Athena, Lambda, and Step Functions to build and optimize cloud-native data solutions.
• SQL and Data Queries: Write complex SQL queries to analyse data and support business intelligence tools, ensuring efficient querying and data retrieval using AWS Athena and other querying tools.
• Team Leadership: Lead and mentor a team of data engineers, providing technical guidance, setting development standards, and ensuring high performance across the team.
• Client Stakeholder Engagement: Collaborate with key client stakeholders to understand business requirements, translate them into technical solutions, and ensure the successful delivery of data projects.
• Data Architecture: Work with data architects to define data models, design data storage solutions, and optimize data integration processes to ensure data quality, performance, and scalability.
• Data Quality and Governance: Implement data quality controls, monitoring, and governance practices to ensure the integrity and security of data across the platform.
• Performance Tuning: Identify bottlenecks in data processes and optimize performance by fine-tuning Glue jobs, PySpark code, SQL queries, and microservices.

Required Skills and Experience:

• Experience: 6+ years of hands-on experience in data engineering, with a focus on building cloud-based data platforms.
• Java Microservices: Strong expertise in developing and deploying Java-based microservices in a cloud environment.
• AWS Glue & PySpark: Proven experience with AWS Glue and PySpark to build, schedule, and run ETL/ELT processes at scale.
• SQL Expertise: Advanced proficiency in SQL for querying and optimizing data in cloud-based environments like AWS Athena or Redshift.
• Cloud Technologies: Deep understanding of AWS services (S3, Lambda, Glue, Athena, Step Functions) and experience with cloud data infrastructure.
• Team Leadership: Experience leading a team of data engineers, providing mentorship, technical oversight, and fostering collaboration.
• Client Interaction: Strong communication skills with experience working directly with client stakeholders to gather requirements and deliver solutions that align with business objectives.

Preferred Skills:

• Big Data Technologies: Familiarity with big data ecosystems and tools such as Hadoop, Spark, Kafka, or similar.
• Automation & CI/CD: Experience with CI/CD pipelines, containerization (Docker), and orchestration tools (Kubernetes).
• Data Governance: Knowledge of data governance frameworks, data security, and compliance practices in cloud environments.
• Certifications: AWS certifications such as AWS Certified Data Analytics or AWS Certified Solutions Architect are a plus.

Lead Data Engineer - SC Cleared - 6 Months - Hybrid - Outside IR35