How We Use RLHF on Manufacturing Data

Reinforcement learning from human feedback is usually described with a chat model in mind: people rank answers, a reward model learns the ranking, the policy is optimized against it. The loop we run on engineering drawings and quotes has the same shape and almost none of the same parts. The humans are senior estimators and quality engineers, not crowd raters. Much of the reward is a fact rather than a preference, checkable against a released BOM or a realized price. And the thing being optimized is, deliberately, not the model that reads the drawing.

This post is about how that loop is built: where the feedback comes from, what it rewards, what it changes, and why it never leaves the customer's tenant.

Where the feedback comes from

Three kinds of humans put signal into the system, and they do it in three different ways.

The first are domain experts building golden sets. A drawing goes to two quality engineers independently; each labels every characteristic on it, type, value, and location. On a dense casting sheet that is more than two hundred items. Where the two disagree, and on a real drawing they disagree on roughly one item in ten, the disagreement is settled by writing the rule down rather than by a tiebreak: whether a position tolerance with a three-datum frame is one feature or three, whether "4X" is a quantity or part of the feature's identity, which sheet owns a note that modifies another sheet. The labeling standard that comes out of this is the first thing the humans teach the system, and everything downstream is measured against it.

The second is the judge. Comparing a model's output to the golden set is not a string-matching problem; a string matcher pairs "R71.5" with "R7.5" and calls it a hit. So a judge model, trained on the labeling standard, decides whether two feature descriptions are the same feature. Every verdict is cached and carries a one-line reason. That makes the reward auditable: when a score moves, we can read why.

The third, and the one that runs continuously in production, is the estimator. Every package the system processes comes back with an exception list: the ambiguities it found, each with a proposed resolution and the evidence for it. The estimator accepts, edits, or overrides each one, and then approves or changes the BOM and cost lines before anything goes out. Each of those actions is logged as a preference: what the system proposed, what the human chose, and on which sheet. That log is the RLHF dataset in the strict sense, and it grows with every quote.

What the reward actually is

The chat-model recipe uses one reward model because there is no ground truth for "helpful." In quoting there often is, so the reward is assembled from sources of different kinds, and the expensive one is used last.

Per-line verifiable reward. When a customer's released BOM exists, every extracted line is checked against it, part number, quantity, material, exact match. This is the densest and cheapest signal in the system and it needs no human at all.

Delayed verifiable reward. Whether the should-cost landed within tolerance of what the part actually sold for. Sparse, arriving months later, noisy, and the one the business cares most about.

Judged reward. For outputs with no released row to check, the judge's verdict against the golden set.

Preference reward. The estimator's accept and override log, used for the decisions that are genuinely judgment. Whether a proposed resolution to a revision conflict was right. Whether an exception was worth raising. Which of two write-ups of the same ambiguity was more useful.

Ordering them this way means human hours are spent only where nothing cheaper can produce the signal, which is what makes the loop affordable at the scale of a quoting department rather than a labeling farm.

What gets optimized

The policy under optimization is not the reader. The frontier vision models that transcribe a drawing are already close to the ceiling on that task, and our own measurements show the gains coming from elsewhere. So the reader is held fixed and the reward is pointed at the parts of the system where a better policy changes the outcome:

The detector, which decides where on the sheet the regions are. Its reward is per-line extraction accuracy downstream, and it is the single largest lever in the pipeline.

The exception policy, which decides what to flag and at what confidence. Its reward comes straight from the estimator's log: exceptions that were accepted score, exceptions that were dismissed cost, and ambiguities that were missed and later caught cost more.

Model routing, which picks which model reads which region at what cost. Its reward is accuracy per dollar, measured on the same golden sets.

Precedent retrieval, which decides which past quote, which clarification thread, and which rate record to surface for a new package. Its reward is whether the estimator used what was surfaced.

Each of these is a small policy with a dense, mostly verifiable reward, which is exactly the setting where reinforcement learning earns its cost. Training a large reader against a sparse preference signal is the setting where it doesn't, and we don't.

Why it stays in the tenant

In a consumer product every rater is drawn from the same population and the policy is shared. In a Tier-1 supplier's quoting department the raters are that supplier's estimators, their preferences encode that supplier's margins and risk appetite, and the data is under an agreement that forbids it from improving anyone else's product.

So the loop is split along a line that also happens to be the right engineering line. Perception is shared: a thread callout is a thread callout at every supplier, and the verifiable rewards for reading it pool across customers without carrying anything customer-specific. Judgment is not shared: how aggressively to flag, what margin precedent applies, whose clarification history matters. The exception policy and the retrieval policy are trained per tenant, on that tenant's log, and nowhere else.

This is the reason the system gets better on a specific supplier's drawings rather than on the industry's average drawing. It is also the reason an estimator's override is worth logging at all: it is the only record of that supplier's judgment that exists, and it belongs to them.

Guarding the reward

Every reward can be gamed and the manufacturing versions are specific enough to name, so the guards are built before the first policy update rather than after the first incident.

Rewarded for fewer wrong lines, a system learns to produce fewer lines; recall and precision are scored separately. Rewarded for confirmed exceptions, it learns to flag everything; every exception raised carries a cost, so flagging has a price. Rewarded for should-cost accuracy against realized price, it learns to anchor on the last quote for the same part; held-out programs the system has never seen are scored separately, so anchoring shows up as a drop where it can be seen.

None of that is novel. All of it is what separates a feedback loop that improves a quoting system from one that quietly teaches it to under-report.

What this means for the supplier

Your estimators' overrides are the training data. They stay in your tenant, they improve the system on your drawings and your programs, and they do not show up in anyone else's product. The exceptions you dismiss teach it to stop asking; the ones you confirm teach it what matters; the precedent you use teaches it what to surface next time.

The practical consequence is that the system's value curve is set by how much of your quoting runs through it, not by the vendor's release schedule. That is the thing to ask any vendor about, including us: what exactly your team's feedback changes, and where it goes.