Forward Deployed Engineer - AVP

On-site in London·Full-time·Added 5 days ago (last checked 26 Oct)

Citi

177 open roles

Apply

Our summary

Work mode: On-site, London
Contract: Full-time

Skills

Requirements

Education & certifications

Relevant software engineering or systems analysis experience or a specialized focus in Python and Generative AI application development).
Solid Python programming proficiency, with practical experience integrating REST/gRPC APIs, Docker containers, and Git workflows.
Strong working knowledge of AI evaluation frameworks, benchmarking methodologies (BLEU, ROUGE, semantic similarity, LLM-as-a-judge), and automated testing suites.
Experience in managing and implementing automated test pipelines in CI/CD environments (Lightspeed, Tekton, Jenkins).
Ability to work effectively under pressure, managing tight sprint deadlines and unexpected changes in model outputs or LOB requirements.
Good written and verbal communication skills with a methodical, detail-oriented approach to software quality and data integrity.
Bachelor's/University degree or equivalent experience in Computer Science, Data Science, or related quantitative discipline.

The role

What you'll do

Conducts tasks related to the construction, calibration, and execution of automated evaluation test suites using Agentflow Studio, measuring agent task completion, tool call correctness, latency, and hallucination metrics.
Curates and maintains domain-specific golden datasets, synthetic benchmarks, and regression testbeds in collaboration with LOB domain experts and senior engineers.
Optimizes prompt templates, system instructions, few-shot examples, and structured JSON output parsers (Pydantic/JSON Schema) to maximize deterministic agent behavior.
Authors and maintains automated test execution and deployment pipelines within Lightspeed CI/CD, enforcing continuous evaluation gates prior to staging and production rollout.
Utilizes in-depth specialty knowledge of Python applications development to analyze complex agent errors, evaluate system performance, and recommend advanced coding and prompt solutions.
Recommends and implements security measures and Lakera Guardrail policies in post-implementation analysis of agent usage to ensure safe functionality.
Monitors and controls evaluation test cycles, providing operational support, performance metrics, and prompt diagnostic reporting to business stakeholders and squad leads.
Has the ability to operate with a limited level of direct supervision, exercising independence of judgement and autonomy while acting as an evaluation SME to pod team members.
Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices, and escalating, managing and reporting control issues with transparency.

About Citi

Industry: Banking
Founded: 1812
Employees: 200,000–250,000
Headquarters: New York City, USA

In their own words

About the company & team

Build the next generation of agentic AI.

The Applied AI and Agents Factory brings together ambitious engineers to turn advanced AI capabilities into production-ready solutions. Working directly with business and technology teams, you'll tackle complex engineering challenges, build intelligent agents and take ideas from initial concept through to live deployment.

This is an opportunity to work at the cutting edge of AI engineering, developing systems that go beyond conversational tools to reason, plan and execute multi-step workflows. You'll build reliable agent workflows, integrate AI with enterprise systems and ensure solutions perform securely and consistently in production.

With a strong focus on hands-on engineering, collaboration and rapid delivery, the Factory gives you the opportunity to solve challenging technical problems, develop your expertise and contribute reusable tools and engineering practices that support AI adoption across Citi.

If you want to do more than experiment with AI - and build, deploy and scale systems that deliver real business value - this is where you can make your mark.

About the Role:

The Forward Deployed Engineer is an intermediate-level software engineering position responsible for constructing automated evaluation harnesses, curating regression testbeds, optimizing prompts and output schemas, and automating testing pipelines within embedded LOB delivery pods. The overall objective of this role is to ensure the reliability, accuracy, and safety of production AI agents by building rigorous evaluation suites in Agentflow Studio and driving automated CI/CD test execution.

All open roles at Citi