Bản tiếng Việt: Forward Deployed Engineer: vai trò đưa AI agent vào production. Part 6 of the series. Previous: Evaluating Agents. Next: MCP.

The first time I did this job, it did not have a name. A client had bought a document-processing pilot from us, and after two months of weekly calls the accuracy on their real documents was nowhere near what the demo showed. So I went and sat in their office for a week. By Wednesday I understood the problem: a third of their "PDFs" were photographs of paper taken with a phone, the folder structure encoded information nobody had written down, and the people who would use the system did not open email, they opened a shared drive. None of that was in the brief. All of it decided whether the project would work.

That week was Forward Deployed Engineering. The role now has a name, a Wikipedia page, a hiring wave at every AI lab, and a growing pile of commentary. This post is what I think it is from the inside, what it is not, and how to do it well when you do not have a frontier lab's budget.

What the role is, and where it came from#

A Forward Deployed Engineer is a software engineer who works inside a customer's environment to make a product deliver value there, and who carries what they learn back into the product. The two halves matter equally. Without the first half it is a product job. Without the second it is consulting.

Palantir invented the practice; the earliest published use of the term that I can find is a 2010 TechCrunch report on the company, and its early deployments were on secure, often air-gapped government networks where the only way to ship was to be physically present. Palantir still calls the role Forward Deployed Software Engineer. For a decade it was a Palantir curiosity. Then large language models arrived with a specific property: they are general-purpose, and the value they create is entirely specific to the data, workflow and systems of one organisation. The gap between "the model can do it" and "it works here" became the whole business, and FDE is the job that closes the gap.

By 2025 OpenAI had built an FDE organisation, Anthropic was hiring for it, and by 2026 the title was showing up at Databricks, Google Cloud, Stripe and a long tail of startups. One analysis I read put the twelve-month rise in search interest for the term at over 1,300% through August 2026, and Business Insider reported a large increase in advertised FDE roles on Indeed between April 2025 and April 2026. The compensation data that exists is US-only and self-reported (Levels.fyi has Palantir's median around $211K total, and published base bands at the labs run from roughly $185K to $320K), so treat it as a signal that companies value the role, not as a benchmark for anywhere else.

Why AI made it necessary#

Three things changed with LLMs that made the FDE the bottleneck role instead of a nice-to-have.

The integration surface exploded. A classic SaaS product has an API and a UI. An agent product touches the customer's documents, their database, their ticketing system, their email, their permissions model, and it has to behave correctly across all of them (the harness from part 2, built against systems you do not control). Nobody at the vendor knows what those systems look like. Someone has to go and find out.

Evaluation needs domain knowledge. You cannot know whether the extraction agent is right about a Vietnamese construction contract without someone who reads Vietnamese construction contracts (part 5's eval sets are built from the customer's failures, and the customer's failures are only visible from inside). The FDE is the person who turns "it feels wrong" from an operations clerk into a task in the eval suite.

The product learns from deployments. Every feature in a mature agent product started as something an FDE built for one customer: a tool, a skill file, a guardrail, a compaction rule. The labs are explicit about this; the FDE is their product-discovery function. In our smaller company the same thing is true: our internal harness and eval library are the generalised residue of customer engagements.

There is a less flattering reason too. Labs sell usage-based contracts, and usage follows adoption, and adoption stalls at the last mile. An FDE is the cheapest way to unstick a seven-figure contract. That is not cynicism, it is the economics that pays for the role, and it is why FDEs get production access and roadmap influence that consultants never do.

The loop#

The job has a shape, and once you see it you can tell real FDE work from rebadged professional services.

Sit with the work. Not with the sponsor, with the people who will use the system. Watch what they do for a day. Find the shared drive, the spreadsheet with the real process, the step everyone does differently. In my week on site this was the whole value; the engineering afterwards was simple once I knew what the problem actually was.

Build in place. In their environment, against their data, with their constraints. This is where the series' machinery comes in: the harness configured for their systems, the tool policy matched to their risk appetite, the sandbox that satisfies their security team. The deliverable is not a slide. It is something running.

Ship to production. Real users, real data, a number that moves. The definition of done I use is borrowed and I think correct: the customer's team relies on it in production, and the product has learned something. Contrast that with a product engineer (done when merged), a solutions architect (done when the deal closes and the design is handed over) and a consultant (done when the deliverable is accepted and the hours are billed).

Generalise back. Take what was built for one customer and decide what becomes product: the tool, the eval tasks, the skill, the integration. This is the step that most often does not happen, because the FDE has already been pulled to the next customer and nobody owns it. If your organisation does not have an explicit mechanism for this step, you do not have FDEs, you have a consulting arm with a nicer title.

A good week#

What the week looks like when it is going well, from a recent engagement rather than an idealised one.

Monday is access. Half the day is spent with the customer's IT getting a read-only database user, a service account scoped to one mailbox, and a VM inside their network, which is exactly the sandbox discipline from part 4 applied to someone else's infrastructure. The other half is sitting with two operations staff while they process the morning's documents, with a notebook.

Tuesday and Wednesday are building. The harness is cloned from our template: the loop, the policy layer, the trace, the eval runner. The work is the customer-specific layer: a CLAUDE.md-style map of their systems, four tools against their data (two read, one reversible write, one that needs approval), a skill file for their document types. By Wednesday evening the agent processes a document end to end on their VM.

Thursday is evals. Thirty of last month's documents, with the operations staff's answers as ground truth, become the first task set. The agent gets 22 of 30. Each of the eight failures becomes a labelled task. Two of them are a new document type nobody had mentioned; one is a tool returning dates in the wrong format. All eight get fixed or filed before the week ends.

Friday is people. A demo for the sponsor with the real numbers, including the failures. Training for the two staff who will be the champions, with the rule that they can mark any output "wrong" with one click and that feedback goes straight into the eval set. A written handover: what runs where, who to call, what the agent is not allowed to do.

Notice the ratio. Perhaps a third of that week is engineering as a product team would recognise it. The rest is access, observation, evaluation and communication, which is why strong product engineers sometimes struggle in the role and strong operations-minded engineers thrive.

What makes someone good at it#

The engineers who do this well share a few traits that do not show up on a standard interview loop.

They can ship end to end alone: backend, a little frontend, infrastructure, data. There is no team to hand off to on site.

They are pragmatic about other people's systems. The customer's database schema is not going to be redesigned. The job is to work with what exists, document the sharp edges, and make the harness absorb them.

They are curious about the domain. The best FDE I have worked with spent her first two days at a logistics client learning how customs declarations actually move, and the agent she built reflected that in ways no spec could have captured.

They communicate with non-engineers without condescension or jargon, and they can say "the model got 22 of 30 and here is what the 8 were" to a sponsor without flinching.

They have judgement about generalisation: the instinct to notice that the thing they just built for this customer is the third time they have built it, and to turn it into product.

If you are hiring for the role, the interview that works for us is a half day with a messy, real, anonymised dataset and the instruction "find out what this is and tell us what you would build". It tests all five traits at once and it is very hard to fake.

The fair critiques#

The role is fashionable, and fashionable roles attract criticism, some of which is right.

Andrew Ng has argued that there will be far more AI engineering jobs than FDE jobs, and that the role is not a replacement for normal engineering. I agree; FDE is a bottleneck role, not a majority role, and a company with more FDEs than product engineers is a services company whether it admits it or not.

The cost is real. Dedicating a strong engineer to one customer for a quarter is expensive, and the model only works when the contract value or the product learning justifies it. Engagements shorter than about a quarter rarely do, because there is not enough time to learn the domain before you leave.

The tension between solving one customer's problem and building a standard product is permanent. Every FDE will be asked to build something that should never become product, and the organisation needs a way to say no that does not fall on the person in the customer's office.

And the role depends on access. If the customer will not give production access, the FDE is a very expensive person waiting for a ticket to be approved. Negotiate access before the engagement starts, not during it.

Doing this from a services company in Vietnam#

I lead an AI team at a software services company, not a lab, and most of what I read about FDEs assumes the lab's economics. Here is how we have adapted the model, for anyone in a similar position.

We run deployment squads instead of lone FDEs: one engineer in the FDE role, one domain analyst who speaks the customer's language (sometimes literally; our customers are in Vietnam and Australia), and a shared platform team behind them. The analyst does much of the "sit with the work"; the engineer does the building; the platform team owns the generalise-back step so it actually happens.

We scope pilots to eight to twelve weeks with production as the explicit target, and we define the success metric with the customer's operations lead, not their sponsor, in week one. A pilot that ends in a slide deck is a failed pilot, whatever the sponsor says.

We carry the series' machinery with us. The loop, the policy layer, the trace, the eval runner and the harness template are internal assets that get better with every engagement. The customer-specific layer is thin by design. This is the generalisation step made structural: it is not that someone remembers to extract the reusable parts, it is that the reusable parts were never customer-specific to begin with.

And we price the first weeks openly, as discovery. The week in the client's office that started this post was unbilled at the time, and it saved a project worth many times its cost. Now it is a line item, and customers who refuse it are customers whose projects we have learned not to take.

Where this leaves the series#

Six posts ago I said that the model is the most reliable part of an agent and the scaffolding around it is not. The harness, the context, the policy and the evals are that scaffolding, and the FDE is the person who assembles it in a place the vendor has never seen, with data the vendor has never read, for people the vendor has never met. If agents are going to do real work in real organisations, somebody has to go to the organisation. That is the whole job, and I think it is the most interesting one in the field right now.

The next six posts go one level out, to what connects agents to each other and to the rest of a company: MCP, A2A, graph engineering, two tools for running fleets of coding agents (Orca and Paseo), and n8n for business workflows.

References#

• Wikipedia, Forward deployed engineer (origin, criticism section)
• Valletta Software, What Is a Forward Deployed Engineer? The 2026 FDE Guide (demand data, compensation bands, hiring companies)
• OpenAI, Harness engineering (February 2026), on encoding what is learned into the repository
• Anthropic, Effective harnesses for long-running agents (November 2025)