Lead Data Engineer

We offer a variety of competitive benefits, some of which vary depending on the role you're recruited to. Some of what you can expect in this role includes:

A competitive rate of pay and pension contribution ( £85,000 -£105,000)

Generous discretionary bonus schemes, incentives and competitions

An annual leave entitlement that increases with length of service

Access to an online GP 24/7, 365 days a year for you and your immediate family.

Employee wellbeing support through our Employee Assistance Programme

Enhanced Maternity & Paternity Pay

Long Service Recognition

Access to a pay day savings scheme, financial coach and up to 40% of your earned wage ahead of payday, through Wagestream.

This Lead Data Engineer will be the senior technical authority for our data platform, setting the architectural direction and engineering standards for the team. This hands-on role will focus on leading the technical delivery of our migration from SQL Server to an AWS lakehouse, and designing the real-time streaming architecture that underpins automation, business intelligence, and analytics, while mentoring engineers across the team.

Own the technical architecture, standards and design patterns for the data platform, covering batch and streaming pipelines built on AWS Glue, AWS Lambda, AWS S3 and Redshift.

Lead the technical delivery of the Data Warehouse migration from SQL Server to AWS, including target modelling, cutover approach, reconciliation strategy and technical risk management.

Design and build the real-time data architecture using Amazon MSK / Apache Kafka and PySpark Structured Streaming (evaluating Apache Flink where it is the better fit), including event schema governance, exactly-once semantics and stream-to-lakehouse integration.

Define and embed data quality, validation and observability frameworks, and hold the team to them through design and code review.

Govern the lakehouse and data warehouse design, ensuring consistency, scalability and sensible cost characteristics across the platform.

Own platform infrastructure-as-code (Terraform) and CI/CD standards, so environments and pipelines are reproducible and reviewable.

Monitor and optimise AWS platform spend, making the cost and performance trade-offs explicit when proposing architecture.

Act as the technical escalation point for complex data issues and performance bottlenecks, leading root-cause analysis and driving permanent fixes.

Collaborate with data scientists, analysts and other engineering leads to support the deployment of machine learning and advanced analytical solutions.

Mentor and technically coach data engineers through pairing, review and design sessions, and contribute to technical assessment during hiring.

Represent data engineering in cross-functional architecture, planning and prioritisation discussions, translating technical trade-offs for senior technical and non-technical stakeholders.

Embed data security and privacy by design, working with Information Security and our Data Protection team on how customer data is modelled, accessed and retained.