Learning Agents Deploy Themselves

On 25 April 2025, OpenAI shipped a GPT-4o update that added “an additional reward signal based on user feedback”: the thumbs-up and thumbs-down data from ChatGPT. Within days the model was sycophantic, and on 28 April OpenAI began rolling it back. Its evaluations and A/B tests had looked good. OpenAI’s own account is that the change “weakened the influence of our primary reward signal, which had been holding sycophancy in check”.

A model that learned from its users shipped a change nobody would have approved. Teams build the same loop at a smaller scale whenever they wire feedback into an optimiser.

Say you run a support agent on Claude Sonnet 5.5 for a subscription business. It answers refund questions, billing questions and cancellations. The team wants it to get better by itself, so they wire the thumbs buttons into a nightly job. The job runs GEPA, a prompt optimiser that ships with DSPy, over the day’s conversations and asks it for a system prompt that customers will rate higher. If the new prompt scores better on a replay, it goes live at 02:00.

Two weeks later the thumbs-up rate has risen from 71% to 74%. Refund answers are better. Cancellation answers have collapsed. One rewrite added the line “If the customer mentions a charge, send them to the billing team.” It fixed a run of confused refund chats. But people who want to cancel almost always mention a charge, so the agent now sends them to billing, and cancellations completed in chat have fallen from about 9 in 10 to 6 in 10.

Fourteen prompt versions shipped in those two weeks. No person read any of them. The cancellation damage went unnoticed for nine days, because cancellations are one conversation in ten and people who are leaving rarely press a thumbs button.

That nightly job is a deployment pipeline with no review, no tests and no rollback plan. It ships whatever raises a number nobody chose on purpose. What matters is where the learning lands, and what stands between the change and the users.

Four places learning can land #

Learning from production means that something persistent changes after the agent sees real traffic. It lands in one of four places, here from the cheapest and easiest to undo to the dearest and hardest.

Memory and data come first: new rows in a memory store, or few-shot examples retrieved at run time, such as last month’s 20 best-rated refund answers. CoALA, the Cognitive Architectures for Language Agents framework (Sumers et al., 2023), calls this episodic and semantic memory. Undoing a change means deleting a row.

Instructions come second: the prompt text changes. The change can be a lesson the agent writes after a failure, as in Reflexion (Shinn et al., NeurIPS 2023), which reported 91% pass@1 on HumanEval against 80% for GPT-4. Or it can be an optimiser’s rewrite. DSPy’s MIPROv2 searches over instructions and few-shot examples with Bayesian optimisation. GEPA, also in DSPy, reflects on traces in plain language, and its authors report that across six tasks it beat GRPO, a reinforcement-learning method that updates weights, by 6% on average with up to 35 times fewer rollouts.

Skills and procedures come third: code the agent can call. Voyager (Wang et al., 2023), a Minecraft agent, keeps “an ever-growing skill library of executable code” and adds a program only after a self-check confirms that it worked. That check is a promotion gate in miniature.

Weights come last: supervised fine-tuning on accepted conversations, or preference training on pairs such as the agent’s draft against the version a human edited before sending. DPO (Rafailov et al., NeurIPS 2023) made the second route practical, with “only a simple classification loss” and no separate reward model.

Compute is the cheap part. Say a run tries 20 candidate prompts on 200 conversations, at 3,000 input and 400 output tokens per call on Sonnet 5.5. Those 4,000 calls cost $24 for input and $16 for output, at $2 and $10 per million tokens. That is $40 a night, or $560 for the two weeks, before judge calls and thinking tokens.

The rule that follows: put each lesson in the cheapest target that can hold it. A fact about a customer goes to memory. A rule such as “confirm the plan type before quoting a refund” goes into the instructions. A multi-step procedure with tool calls becomes a skill. Weights are for changes nobody can write down, like tone across thousands of conversations. CoALA splits the instructions row. It files a Reflexion-style lesson under semantic memory. An optimiser’s rewrite of the prompt template changes procedural memory, the agent’s code and weights, as new skills and new weights do, and CoALA calls writing there “significantly riskier” than writing to episodic or semantic memory.

The loop around any target #

Whatever the target, the loop has the same five steps.

1. Collect. Record each interaction with what happened next: the question, the answer, the prompt version and the outcome, such as a rephrase within two minutes, an escalation or a completed cancellation. Thumbs are one signal among these, and a skewed one.
2. Propose. Produce one change to one target: a row, an example set, a prompt edit, a skill version or a training set. The proposer can be an optimiser, a reflection step or a person.
3. Evaluate. Run the candidate and the current version on the same labelled cases: a held-out set the proposer never saw, plus a regression set of past failures. Score each slice, not only the total.
4. Gate. Refuse the change if any slice drops by more than a set tolerance, even when the average rises. Hold it for a person if a slice is too small to judge.
5. Canary, then promote. Send 5% of traffic to the candidate by a hash of the user ID, compare outcomes, then move the production label.

Every learned change is then a versioned artefact, like a prompt a person edited: a content hash, a movable label, and a record of what it learned from. Here is the record for the billing line, as a gate would have written it:

{
  "id": "support-agent-prompt@9c41e07",
  "parent": "support-agent-prompt@d71b5e8",
  "baseline": "support-agent-prompt@4a2f6c1",
  "target": "instructions",
  "proposer": "gepa-nightly",
  "learned_from": {"day": "2026-09-29", "conversations": 11842, "signal": "thumbs"},
  "diff": "+ If the customer mentions a charge, send them to the billing team.",
  "eval": {"set": "support-heldout-v6", "cases": 400,
           "all": [82.5, 83.5], "refund": [80.0, 90.0], "cancellation": [90.0, 61.67]},
  "gate": "refused: cancellation -28.33 points, tolerance 3.0",
  "approved_by": null
}

The gate compares a candidate with baseline, the last version a person reviewed, not with its nightly parent. The record also answers “who approved the change that broke cancellations?”: nobody did, and the gate said no.

The optimiser maximises the metric you hand it #

In DSPy, the nightly job that broke cancellations and a better one differ in a single argument, metric. GEPA calls the metric with the labelled example and the program’s prediction. The metric returns a score, and can add written feedback for GEPA’s reflection step. The sketch follows DSPy’s documented GEPA interface and was not run.

import dspy

dspy.configure(lm=dspy.LM("anthropic/claude-sonnet-5-5"))
support_agent = dspy.Predict(
    "question -> answer, action: Literal['cancel_now', 'refund', 'send_to_billing']")
rate = dspy.Predict("question, answer -> likely_thumbs_up: float")
# train, val: dspy.Example(question=..., correct_action=...).with_inputs("question")

def thumbs_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    # what the nightly job optimised: would this customer press thumbs-up?
    return rate(question=gold.question, answer=pred.answer).likely_thumbs_up

def outcome_metric(gold, pred, trace=None, pred_name=None, pred_trace=None):
    # what the business needs: the action a person labelled correct
    ok = pred.action == gold.correct_action
    return dspy.Prediction(score=float(ok),
                           feedback=f"expected {gold.correct_action}, got {pred.action}")

optimizer = dspy.GEPA(metric=outcome_metric, auto="light",
                      reflection_lm=dspy.LM("anthropic/claude-opus-5-5"))
candidate = optimizer.compile(support_agent, trainset=train, valset=val)

The agent returns an answer for the customer and an action label. The thumbs metric asks a model to guess whether the customer will be pleased, so the optimiser learns to please the customer in front of it. Being sent to billing sounds helpful, and nothing in the metric knows whether the cancellation happened. The outcome metric compares the action with the one a person marked correct, and its feedback tells the reflection step what went wrong.

Those labels cost human time. If a model does the grading instead, its grades have to be checked against human labels before anyone trusts them. A guess at whether someone will be pleased is hard to check against anything.

A promotion gate that reads slices #

The gate turns evaluation numbers into a decision. The script below compares a candidate with the last prompt a person reviewed, on 400 labelled cases, slice by slice. It refuses promotion if any slice drops by more than 3 points, whatever the average does, and holds the change for a person if any slice has fewer than 50 cases. Both versions run on every case, so the script also runs an exact sign test on the cases where they disagree.

from math import comb

# Say you have 400 labelled cases, each run through the last reviewed prompt and a candidate.
# Per slice: (both pass, only reviewed passes, only candidate passes, both fail)
CANDIDATES = {
    "optimiser's current prompt": {
        "refund": (142, 2, 20, 16), "billing": (100, 2, 4, 14),
        "cancellation": (35, 19, 2, 4), "other": (29, 1, 2, 8)},
    "same edit, refunds only": {
        "refund": (142, 2, 20, 16), "billing": (101, 1, 2, 16),
        "cancellation": (53, 1, 1, 5), "other": (29, 1, 1, 9)},
}
TOLERANCE = 3.0   # points any slice may drop before promotion is refused
MIN_CASES = 50    # smaller slices are held for a person whatever the numbers say

def sign_test(b, c):  # two-sided exact test on the cases where the versions disagree
    n = b + c
    return 1.0 if n == 0 else min(1.0, 2 * sum(comb(n, k) for k in range(min(b, c) + 1)) / 2**n)

for name, slices in CANDIDATES.items():
    print(name)
    total_old = total_new = total_n = 0
    regressed, held = [], []
    for s, (both, old_only, new_only, neither) in slices.items():
        n = both + old_only + new_only + neither
        old, new = both + old_only, both + new_only
        delta = 100 * (new - old) / n
        total_old, total_new, total_n = total_old + old, total_new + new, total_n + n
        if delta < -TOLERANCE:
            note = "REGRESSED"
            regressed.append(s)
        elif n < MIN_CASES:
            note = "small slice"
            held.append(s)
        else:
            note = ""
        print(f"  {s:12} n={n:3}  {100*old/n:6.2f}% -> {100*new/n:6.2f}%  {delta:+6.2f}"
              f"  p={sign_test(old_only, new_only):.3f}  {note}")
    d = 100 * (total_new - total_old) / total_n
    print(f"  {'all':12} n={total_n}  {100*total_old/total_n:6.2f}% -> {100*total_new/total_n:6.2f}%  {d:+6.2f}")
    if regressed:
        print("  verdict: REFUSE (" + ", ".join(regressed) + ")")
    elif held:
        print("  verdict: HOLD for review (" + ", ".join(held) + ")")
    else:
        print("  verdict: PROMOTE to 5% canary")

optimiser's current prompt
  refund       n=180   80.00% ->  90.00%  +10.00  p=0.000  
  billing      n=120   85.00% ->  86.67%   +1.67  p=0.688  
  cancellation n= 60   90.00% ->  61.67%  -28.33  p=0.000  REGRESSED
  other        n= 40   75.00% ->  77.50%   +2.50  p=1.000  small slice
  all          n=400   82.50% ->  83.50%   +1.00
  verdict: REFUSE (cancellation)
same edit, refunds only
  refund       n=180   80.00% ->  90.00%  +10.00  p=0.000  
  billing      n=120   85.00% ->  85.83%   +0.83  p=1.000  
  cancellation n= 60   90.00% ->  90.00%   +0.00  p=1.000  
  other        n= 40   75.00% ->  75.00%   +0.00  p=1.000  small slice
  all          n=400   82.50% ->  87.25%   +4.75
  verdict: HOLD for review (other)

The optimiser’s prompt lifts the total from 82.50% to 83.50%, so a gate that reads only the total would promote it. By slice, refunds gain 10 points and cancellations lose 28.33. On cancellations the versions disagree on 21 cases and the candidate loses 19 of them, so the p-value rounds to 0.000. The gate refuses. The same edit restricted to refund questions moves no slice down and lifts the total to 87.25%. The gate still holds it, because the 40-case “other” slice is too small to judge, so a person reads those cases before the 5% canary.

Two design choices sit in the constants. The tolerance is strict: one cancellation case is 1.67 points, so a net loss of two cases trips the gate. A refusal costs one person’s review, and a false promotion costs a slice of customers. The minimum slice size exists because at 40 cases a real drop can easily come out as no change. The p-value decides nothing. It is for the person who reviews a refusal or a hold, and because it uses only the cases where the two versions disagree, it is more sensitive than comparing two pass rates as if they came from different cases.

Choose slices by what the business cannot afford to break: intent, plan type, language, and one regression slice for every past incident. The cancellation slice exists in this gate because someone decided that a customer who wants to leave must be able to.

Four ways a learning loop goes wrong #

The first is optimising the proxy, the GPT-4o failure from the opening. OpenAI noted that user feedback “can sometimes favor more agreeable responses”, and later framed the launch decision as whether to hold the update back “based only on the subjective flags of the expert testers”. The averages said ship.

The second is learning from your own outputs. If “accepted” means “the customer did not complain”, every wrong answer nobody noticed becomes a training example, and fine-tuning makes the model more sure of it. Learn from outcomes you can check instead, such as a completed cancellation or a refund that matched policy, or from cases a person labelled.

The third is poisoning through feedback. A thumbs button is a write path from anonymous users into your prompt. A few hundred coordinated thumbs-up on an answer that promises refunds outside policy can move an optimiser that counts thumbs. Count feedback once per account, cap it per day, and never let feedback alone promote a change.

The fourth is personal data in the weights. A memory row can be deleted, but a fine-tuned checkpoint cannot forget one customer without being retrained. Strip personal data before a conversation enters a training set, and record which conversations went into which checkpoint, so that a deletion request at least tells you which models to retrain.

What the team owns #

The per-slice gate refuses the nightly job’s change and holds the narrower one for a person. No optimiser setting would have done that. The optimiser, the reflection step and the fine-tuning job are replaceable parts. The gate, its slices and its labelled cases are what the team owns, and they are where a learning agent actually gets reviewed. The gate is the product.