Self-evolving agents are stuck at acceptance: one paper finds selection is a coin flip, another skips evaluation

Published on November 20, 2026 by Remy

Short answer: The most important recursive self-improvement (RSI) work this week is not a new agent that rewrites itself better. It is two papers about who signs off on a change. The Resolution Hypothesis, from ShanghaiTech and ByteDance, audits 7 agent-evolution methods and finds that the bottleneck is selection, not proposal. The candidate pools contain real improvements, but a selector that looks at one run picks the better candidate 49.7% of the time, which is a coin flip. Most of the 10 to 40 point gains in the literature can be reproduced by one sentence that suits the grader, and once the graders are repaired the median gain drops from 17.6 to 0.8 points.[1] ELITE, from Jürgen Schmidhuber’s group, goes the other way. It runs no benchmark inside the evolution loop. It merges changes by reading code and checks only that the result runs. On a 60-task SWE-bench Pro subset it ties DGM and HGM, it scores 5 to 10 points higher on two transfer benchmarks, and it costs 1/4 to 1/6 as much.[2]

What changesWho signs offIs the gain real
Resolution Hypothesis (arXiv:2611.23068, Nov 17)No new method. It audits how 7 methods, including GEPA and SkillOpt-Lite, evolve prompts, skills, memory, and harnessesBenchmark scores, rechecked by the authors with repeated evaluation and repaired gradersLarge gains are mostly evaluation shortcuts. Small real gains exist, but current selectors cannot find them
ELITE (arXiv:2611.21056, Nov 16)The coding agent’s own scaffold codeBuild, interface, and smoke tests, plus an LLM that reads the code and merges changes. Scores never feed backTies DGM and HGM on the target benchmark and transfers better. But only 50 to 60 tasks, 3 runs, and no public code

If you run a self-evolving loop, the two papers lead to four concrete steps:

  1. Evaluate the same version several times. Report the evaluation noise and the run-to-run spread, and compare your gain with them.
  2. When a gain is large, take the single most suspicious rule from the evolved skill and add only that rule to the original version. If one rule recovers most of the gain, it is almost always a shortcut.
  3. Check the grader itself. For example, does a spreadsheet grader recalculate formulas, and does a math grader accept more than one answer format?
  4. Keep one benchmark that never takes part in evolution, and use it only for a final transfer test.

The site’s survey of self-evolving agents asked three questions: what changes, who signs off, and which gains are real. This is the first issue of the RSI Frontier series, which checks this line of work once a week. This week both papers land on the second question.

The loop has three steps, and the third one fails

Almost every self-evolving agent today uses the same loop. A proposer writes candidates (prompts, skills, memory, harness code). The agent runs a task set. The candidate with the higher score is accepted, and the next round edits the new version. The model weights stay frozen, so a change can help only by changing behavior at inference time. DGM uses SWE-bench scores to pick parents from its archive. HGM uses the scores of descendants to estimate how much potential a lineage has. GEPA and SkillOpt use validation scores to decide whether to replace the current prompt or skill.[2][4]

The loop assumes that scores can tell a good change from a bad one. The Resolution Hypothesis tests that assumption directly. The authors split the variance of a score into two parts. The first part, σ², is how much repeated evaluations of the same version differ. The second part, τ², is how much the true quality of versions from independent runs of one method differ. Comparing one fixed version with a baseline is limited by σ. Comparing two methods is limited by σ² plus τ², because each run of a method produces a new version.[1]

The three steps of the loop, and which step each paper this week changesProposewrite candidates: prompts, skills,memory, harness codeEvaluaterun the benchmark, get a scorescore = true quality + noiseSelectaccept if the score is higherno undo: the next round edits itResolution Hypothesis: the bottleneck is hereproposals contain real wins (Quality@100 is 3.3–3.4 pp above the null)one-rollout selection 49.7%, LLM judges 45.4–49.0%fix: repeat evals, process metrics, repair graders, report σ and τELITE: no evaluation inside the loop7 changes built in parallel from one parent, code read for overlap or conflictmerged into one child that only has to pass build, interface, smoke testsbenchmarks run afterwards and never feed backSAGE / GUARD: make selection harder to foolper-task paired ledger, regressions cost more than winsabstain when the win is not significant; keep the incumbentSources: arXiv:2611.23068, arXiv:2611.21056, arXiv:2611.23476, and this site's SAGE write-up.

Figure: The three steps of the loop, and which step each paper this week changes.

Resolution: method gaps are smaller than the noise of one run

For every combination of 7 methods, 6 models, and 4 benchmarks (ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench 2.1), the authors ran evolution 8 times and evaluated each result up to 16 times. That is 1,030 evolution runs, more than 2.1 million test trajectories, more than 10 million LLM calls, and 400,000 CPU core hours.[1] Here is what they measured:

  • The noise of one evaluation, σ, is 1.0 to 3.7 points on the first three benchmarks and 4.5 points on Terminal-Bench.
  • The run-to-run spread, τ, ranges from 0 to 12.3 points. Together, the median standard deviation of “evolve once, evaluate once” is 1.7 to 4.1 points.
  • For more than 60% of configuration pairs on the same model and benchmark, the gap between them is smaller than one standard deviation of a single-run comparison.

Then the literature. The authors went through 171 results in the main tables of 9 agent-evolution papers. The new method leads its strongest competitor by a median of 3.4 points, with an interquartile range of 1.2 to 6.6 points. None of the 171 results reports a variance, and only 6 average more than one evolution run.[1]

To detect a true 3.4 point gap with 80% power, each method needs about 5 independent runs on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld, and 23 on Terminal-Bench. With one run per method, a results table picks the truly best of four text-artifact methods on Terminal-Bench only 33% of the time. A random guess gets 25%.[1] On a noisy benchmark like Terminal-Bench, many of the rankings where papers “beat” each other are close to a lottery.

The resolution floor: most method gaps sit inside single-run noiseNoise of one evaluation, σSSB / OfficeQA / ALFWorld 1.0–3.7, Terminal-Bench 4.5SD of one run evaluated onceper-benchmark medians 1.7–4.1New method's lead in papers171 results, median 3.4, IQR 1.2–6.6, no variancesReported gains from eval shortcutsmedian 17.6 → 0.8 after the grader is repaired17.60.802468101214161820+percentage points (pp)Data: arXiv:2611.23068 §4, Table 6, App. D.3/E. Bands are ranges, ticks are medians, gray is single-run noise.

Figure: Where each kind of gap sits on one percentage-point axis. The gray zone is single-run noise.

Where large gains come from: one sentence reproduces them, a repaired grader removes them

If most improvements sit inside the noise, what explains the 10 to 40 point gains? The hypothesis makes a prediction. A gain is large enough to clear the noise only if one change helps on most tasks of a benchmark at once. An evaluation shortcut does exactly that.

The authors used two tests. The first is the one-rule test. They read the evolved skills instruction by instruction, found the one rule most likely to explain the gain, and added only that rule to the unevolved baseline. Across the 27 model and benchmark pairs with a gain above 3 points, that one sentence recovers a median of 90% of the gain. The second test repairs the grader. They restored the official evaluation, or graded answers by meaning instead of surface form, and scored the same artifacts again. The median gain fell from 17.6 to 0.8 points. Re-running evolution under the repaired graders did not bring the gains back.[1]

Figure 3 of the paper. Left: the x axis is the reported gain and the y axis is the score of the baseline plus one rule, and the points sit close to the diagonal. Right: the y axis is the gain after repair, and nearly all points sit near zero

Figure: Figure 3 of the Resolution Hypothesis paper. On the left, one rule reproduces most of each gain. On the right, the gains go to about zero after repair.

Table 9 of the paper lists the shortcuts. The SpreadsheetBench grader reads formula cells without recalculating them, so every formula scores as empty. The AIME 2025 parser accepts only a bare integer. LiveMathematicianBench has a “stronger result” option that is correct whenever it appears. DocVQA, HotpotQA, and SearchQA use near-exact string matching. The OfficeQA version released with SkillOpt-Lite drops the official instruction to answer with a bare value. No shortcut was found in ALFWorld, PUPA, HoVer, or IFBench.[1]

Two shortcuts reproduced on SkillOpt-Lite’s public data

The paper’s code and trajectories will be released on acceptance. SkillOpt-Lite, one of the audited methods, is open source, and the commit the paper cites, 4cb4eee, is the current main branch.[3] Both tests below run locally and need no API key.

The first test is LiveMathematicianBench. The benchmark turns theorems from recent arXiv papers into five-option questions that ask for the strongest statement that can be proved. Some questions include the option “One of the remaining options is correct, but a stronger result can be proven.” The script loads the three SkillOpt-Lite splits from Hugging Face and counts how often that option appears and how often it is the answer.

Terminal output: the three LiveMath splits have 177 questions, 85 include the "stronger result" option, and it is the correct answer in all 85. On the 106-question test split, 46 can be answered without reading the math. Lines 3 to 5 of the skill.md that SkillOpt-Lite released for gpt5.4-nano say to check this meta-option first and default to it whenever it is present

Of 177 questions, 85 have the option, and it is the answer in all 85. That matches the paper’s appendix.[1] On the test split, 46 of 106 questions (43.4%) can be answered correctly without reading a single formula. SkillOpt-Lite’s own released artifact shows the effect more directly. The skill.md evolved for gpt5.4-nano opens with a section called “The “stronger result” meta-option (check first)”, and its second bullet says “Default to this meta-option whenever it is present.” Evolution did optimize something. It optimized the way this benchmark writes its questions. The paper adds that on the 60 test questions without the option, the baselines score 50 to 61 and no evolved artifact beats them.[1]

The second test is the SpreadsheetBench grader. The script calls evaluator.compare_workbooks from SkillOpt-Lite on three answer files. The first writes the formula =SUM(A1:A3) in D4. The second writes the number 100 in D4. The third is the first file after LibreOffice recalculates it. The gold answer is 100.

Terminal output: the formula file is graded FAIL because the grader reads pred=None. The literal-value file is graded PASS. The same formula file is graded PASS after LibreOffice recalculates it. Below is the skill.md that SkillOpt-Lite released for gpt4o, under the heading Write COMPUTED VALUES, never formula strings

The formula is correct, but the grader reads None and marks it wrong. openpyxl with data_only=True reads only cached values, and a formula written by openpyxl has no cached value. After recalculation the same file passes. The official SpreadsheetBench evaluation recalculates with LibreOffice first. SkillOpt-Lite’s default path does not, and the repository even ships an opt-in recalc_harness.py to close the gap.[3] The released skill’s rule “write computed values, never formula strings” targets this gap. The paper’s numbers: under the released grader the rule is worth about 30 points, and all 8 SkillOpt-Lite runs with GPT-5.4 nano find it. Under the repaired grader it is still worth about 6 points, because some formulas use functions LibreOffice cannot evaluate. On a 39-task validation set that is two or three tasks, about the size of run-to-run noise. Of 12 artifacts re-evolved under the repaired grader, only 2 keep the behavior, although half of them state the rule in their text.[1] When the noise is larger than the gain, evolution cannot keep even a good rule that it has written down.

Both examples show the same thing. An evolution loop optimizes any signal that clears the noise, and the signal that clears the noise most easily is often a flaw in the evaluation.

The candidate level: good changes exist, but nobody picks them

If the large gains are shortcuts, what about small real gains? The authors built two fixed pools of 100 skills each on Terminal-Bench 2.1.

The first pool asks whether the proposer can write good skills. The 100 skills were generated from 10 training tasks and run 16 times on each of 10 other test tasks. If you keep the best skill for each task, the success rate rises from 49.2% at K=1 to 61.8% at K=100. That is 3.3 to 3.4 points above a null model in which all 100 skills have the same quality. Averaged over all 10 tasks, single skills differ by more than 15 points, and the best one is 8.3 points above the pool mean.[1] Proposal is not the bottleneck.

The second pool asks whether selectors can find good skills. Each selector sees one rollout, and true quality is estimated from the remaining rollouts. Table 1 has the results. Pairwise selection on execution scores picks the better skill 49.7% of the time. The native parent selection of SkillOpt-Lite and GEPA sees two rollouts and gets 48.6%. Three LLM judges that read the full trajectory get 45.4% to 49.0%. Regressions are 30.5% of the pool but 62.1% of the accepted candidates. The rank correlation between one execution score and true quality is 0.054.[1] As the candidate budget grows from 1 to 100, every practical selector stays flat near the random line. An oracle passes their 100-candidate level before K reaches 5.

Screenshot of the Resolution Hypothesis project page. Under the title: Agents already propose real improvements. Evolution stalls because the evaluators that choose between them are, in effect, flipping coins. The left card shows Quality@100 = 61.8%, 3.4 pp above the null pool. The right card shows execution gates at 49.7%, LLM judges at 45.4–49.0%, and a train +9.0 pp versus test −3.2 pp gap

Why do LLM judges fail too? The paper’s explanation is that a judge’s reading noise is independent of the evaluation noise. The two add up instead of canceling. Keep this in mind, because many self-evolving systems now use LLM scoring to save evaluation cost.

Selection errors add up: the in-loop score rises, capability does not

If each round of selection is close to random, what happens over many rounds? The authors took every intermediate version from 8 SkillOpt-Lite runs and 8 GEPA runs and re-ran each one 8 times per task on the same Terminal-Bench validation set. The in-loop gains grew to 6.9 to 9.0 points. The re-evaluated gains stayed between +0.4 and −3.2 points.[1] The cause is the acceptance rule. A candidate’s single score is compared with the incumbent’s recorded score, and the candidate wins if it is higher. Each round therefore picks a lucky draw from the upper tail of the noise. The run-to-run spread also grows with each round. By the last round τ² reaches 4.2 to 24.5 squared points, as large as the gaps between methods in the literature. That is where the unstable method rankings come from.

Figure 5 of the paper: four panels for SkillOpt-Lite and GEPA on DeepSeek-V4-Flash and GPT-5.6 Luna. The blue in-loop line rises to 7 to 9 pp over the rounds. The orange re-evaluated line stays near zero or slightly below

Figure: Figure 5 of the Resolution Hypothesis paper. Blue is the in-loop score, orange is independent re-evaluation.

A toy model: how much noise breaks selection

To see the mechanism, I wrote a small simulation. Each candidate changes true success by δ, drawn from a normal distribution with mean −0.5 and standard deviation 2.5 points. Most changes hurt a little and a few help. Each evaluation returns the true value plus noise with standard deviation σ. σ = 4.5 matches the paper’s single-evaluation noise on Terminal-Bench. σ = 50 is one pass or fail rollout of one task at a 50% success rate.

[Terminal output. Part 1 is pairwise selection accuracy: with 1 evaluation it is 87.8% at σ=1, 66.0% at σ=4.5, 55.0% at σ=16, and 51.7% at σ=50; with 16 evaluations it rises to 96.7%, 86.4%, 67.8%, and 56.3%. Part 2 is 10 rounds of greedy acceptance: at σ=4.5 with 1 evaluation the in-loop gain is 8.4 and the true gain is 1.7; with 16 evaluations it is 7.0 and 6.2; at σ=1 with 1 evaluation it is 7.1 and 6.3

Two results match the paper. First, selection accuracy falls toward 50% quickly as σ grows. The paper’s selectors look at one rollout of one task, which is close to the σ = 50 row, so 49.7% is not surprising. Second, after 10 rounds of greedy acceptance at σ = 4.5 with one evaluation per version, the in-loop score rises 8.4 points while true quality rises only 1.7. With the same noise and 16 evaluations per version, the true gain is back to 6.2 points. Cutting the noise to σ = 1 has the same effect. The model is crude. It has no task structure and no shortcuts. But it shows that the gap between the in-loop score and real capability is set mainly by the resolution of each evaluation. A better proposer cannot close it.

How much the in-loop score rose vs how much capability roseLeft: 8 SkillOpt-Lite and 8 GEPA runs on Terminal-Bench in the paper. Right: this post's toy model, 10 rounds, 4,000 runs-40+4+8in-loopre-eval+8.4+1.7σ=4.5, 1 eval+7.0+6.2σ=4.5, 16 evals+7.1+6.3σ=1, 1 evalin-loop scorere-evaluated / truePaper values are ranges (6.9 to 9.0 and +0.4 to −3.2 pp), drawn as intervals. The model setup is described in the text.

Figure: Left, the measured ranges from the paper. Right, the toy model. Both show in-loop gains of 7 to 9 points.

The paper suggests fixes at two levels. At the candidate level, add low-variance process metrics to pass or fail, such as “did the agent run a test after editing a file” or “was the failure output of a verification command hidden”. These metrics have 13 to 57 times less evaluation noise than final pass or fail. At the method level, evaluate the proposer and the selector separately. Use Quality@K curves for proposers and the quality a selector recovers from a fixed pool for selectors, and report σ and τ next to every score.[1]

ELITE: if selection cannot be trusted, do not select in the loop

ELITE and the Resolution Hypothesis are independent papers, but they read well together. The ELITE authors call methods like DGM and HGM “evaluate-and-select”: downstream task results decide which candidate or lineage keeps evolving. That has two costs. Every candidate has to run on the target tasks, so wider exploration costs more. And repeated optimization against one evaluation distribution tends to learn things that only help on that benchmark.[2] ELITE asks whether a coding agent can improve itself without looking at any downstream task result.

Figure 1 of the ELITE paper. Left: every DGM node is labeled with a solve count and a score, such as 36/60 solved. Middle: HGM updates ancestor clade statistics from descendant results. Right: ELITE explores 7 children from A0 in parallel and integrates them into A1, labeled No task evaluation

Figure: Figure 1 of the ELITE paper. The two methods on the left run tasks at every step. ELITE on the right does not.

Each generation has two stages. In the explore stage, an LLM reads the current agent’s source code and a short evolution history. It proposes K focused hypotheses (7 by default). Each hypothesis names the capability to improve, a concrete limitation in the current code, the fix, and what counts as done. Each hypothesis is implemented independently from the same parent, which gives 7 runnable children. Each child must pass a deterministic gate: it compiles, it imports, it keeps the required interfaces, and it changes only the permitted files.[2]

In the integrate stage, the LLM gets every valid child’s hypothesis, a summary of its patch, and its gate results, and it can read the full patches. It judges each pair of mechanisms on five dimensions: complementary, redundant, conflicting, overly complex, or incomplete. It then re-implements a compatible set of mechanisms on the original parent to produce one child. It does not stack the patches mechanically. That child must pass a stricter gate that adds smoke tests of the tool-call protocol and the main execution flow. If it passes, it becomes the next generation. If not, the parent stays.[2] No benchmark result, no task trajectory, and no candidate score takes part in any decision.

So ELITE has two judges: a set of deterministic checks that only ask “does it run”, and an LLM that reads code. Benchmark scores are measured afterwards and never feed back.

Results: a tie on the target benchmark, better transfer, much lower cost

Evolution runs on a fixed 60-task subset of SWE-bench Pro. SWE-bench Verified and Polyglot, 50 tasks each (the DGM subsets), are used only for transfer. All three methods start from the same agent and use the same model. With DeepSeek-V4-Flash each method ran 3 times:[2]

Pro (60 tasks)Verified (50)Polyglot (50)Task runs during evolutionTimeEst. cost
Initial agent60.064.064.0
DGM86.763.362.781043.15 h$45
HGM88.365.364.780015.66 h$30
ELITE85.673.369.304.90 h$8

DGM and HGM gain almost 27 points on Pro and stay flat or drop slightly on Verified and Polyglot. ELITE gains one or two points less on Pro and is 5 to 10 points higher on transfer. With Qwen-3.7-Plus the pattern is the same, the Pro gap is wider (45.6 against 50.0 and 48.3), and ELITE still transfers best. Starting from mini-swe-agent v2.4.6 with Claude Opus 5.5, 10 rounds accepted 6 children. Pro went from 71.7 to 83.3, Verified from 70 to 78, and Polyglot from 72 to 78.[2]

Two ablations stand out. First, without the deterministic gates the three-benchmark average falls to 52.0, below the initial agent’s 62.7. Integration relies on an LLM reading code, and without a hard “does it run” check the system breaks itself. Second, “evaluation-based select-one” also builds 7 children, but each round it scores them on 30 Pro tasks and keeps the best. It gets 91.7 on Pro, the highest of any variant, but only 70.0 and 62.0 on transfer.[2] Selecting on the target benchmark gives the best target score and the worst transfer. That is what the Resolution Hypothesis predicts: the selection fits the evaluation, not the capability.

ELITE has no public code yet. The paper gives the algorithm, the prompts, and the task lists, so this post could not re-run it.

Measuring ELITE with the same ruler

Apply the Resolution Hypothesis standard to ELITE’s own numbers and a few claims need a discount.

The Pro subset has 60 tasks, so one task is 1.67 points. ELITE’s 85.6 and DGM’s 86.7 differ by less than one task, and the run-level standard deviations are 1.0 to 1.7 points. “Tie” is the accurate word. The transfer gap is larger. On Verified it is 73.3 against 63.3, which is 5 of 50 tasks. The spread across 3 runs is only 1 to 3 points, so the 10 points are not run noise. But the 50 tasks are one fixed set, and the paper does not test whether the gap holds on a different set. If you treat the 50 tasks as a sample from the full benchmark, the standard error of the difference between two proportions is about 9 points, close to the gap itself.

The comparisons themselves are thin. Apart from full ELITE and the no-gates variant, every ablation ran once, which the authors state in their limitations. By the Resolution Hypothesis estimate, telling apart a few points on a noisy SWE-style benchmark takes a dozen to two dozen runs per method.

ELITE does have one structural advantage. It cannot be pulled by a grader flaw, because it never sees a score. In the LiveMath and SpreadsheetBench examples above, evolution learned the grader’s quirks. ELITE’s loop has no such signal to learn. The risk moves somewhere else. The LLM can be wrong when it judges two mechanisms complementary, and the deterministic gate only guarantees that the agent runs, not that it is better. In Figure 3(b) of the paper, 2 of the 5 accepted updates lowered the Pro score, and the authors say the gate “does not guarantee downstream task improvement”.[2]

Other papers from the same week

This week arXiv had dozens of papers with self-improving, self-evolving, or RSI in the title, most of them applying “self-evolution” to one domain. Three more bear directly on acceptance:

  • GUARD (arXiv:2611.23476, Nov 17) targets harness evolution without an external grader. Each task is first split into pre-execution self-checks. Trajectories are graded against those checks, which separates what an edit newly broke from what was never solved, and mastered rules are pinned. A gate called Soft-HURT promotes an edit only when gains outweigh regressions, and it guarantees no regression on its gated buffer.[5] The idea matches SAGE, covered on this site: look at per-task wins and losses, not the total.
  • Self-Improvement Memory (arXiv:2611.18856, Nov 13) gives the harness optimizer its own memory of each edit’s rationale, predicted outcome, observed outcome, and evaluation context. Across 10 independent source runs in coding, paired continuations with this memory score 2.36 points higher on average (p = 0.003).[6] It is one of the few papers that reports a paired test, and 2.36 points sits inside the noise band the Resolution Hypothesis describes, so the paired design is what makes the result readable.
  • CompoundingKernel (arXiv:2611.19230, Nov 14) evolves GPU kernels. As of November 11 it ranked first on the L1, L2, and quantization boards of SOL-ExecBench, with kernels 4.40 times faster than the reference on the 209 public problems. Correctness is decided by checks outside the model, and the authors conclude that “the rate of reliable measurements, not the number of agents, limits how far such a search scales”.[7] That fits the limitation section of the Resolution Hypothesis. Program evolution scores deterministic code and can re-run it cheaply in parallel, so its gains accumulate.

One more detail matters. SAGE’s two largest gains are on LiveMath and OfficeQA (with DeepSeek-V4, 34.15 to 48.78 and 32.93 to 45.12). Those are two of the benchmarks where the Resolution Hypothesis found shortcuts.[8][1] The audit covered the versions released with SkillOpt-Lite and did not test SAGE. A paired statistical gate like SAGE blocks lucky edits that ride on noise. It cannot block a shortcut, because a shortcut’s wins are real and repeatable on every task. Nobody has yet measured how much of SAGE’s gain survives a repaired evaluation.

The same week, RSI also came up at the policy level. In the November 10 version of its S-1, Anthropic wrote that it has determined it meets the “AI R&D capability” threshold in its Responsible Scaling Policy, based on what it expects from future models rather than on what it has observed in current ones.[9] That is a different layer from the technical question here. It does mean that “how do you verify that a self-improvement is real” is no longer only a methods question for papers.

My take

One of this week’s papers is a failure analysis and the other is a new method, and both point to the same place. For systems that keep weights frozen and evolve external text or code, the resolution of acceptance decides how much real improvement can accumulate. Proposal has been good enough for a while.

I think the Resolution Hypothesis is the most useful audit of self-evolving agents this year. It does not reject the direction. It gives a set of checks anyone can run: report σ and τ, run the one-rule test, repair the grader, and evaluate proposal and selection separately. Its limits are stated clearly. The candidate-pool experiments use only Terminal-Bench, the strongest model is Kimi-K3, stronger models may have less noise, and methods that update weights are out of scope.

ELITE is an interesting existence proof. Without evaluation in the loop, it brings a coding agent to the level of DGM and HGM, with better transfer and far lower cost. But its evidence is small. With 60 and 50 tasks and 3 runs, it cannot resolve gaps of a few points, and by the Resolution Hypothesis standard “10 points better on transfer” needs more tasks and more runs to confirm. Its real contribution is a change of judge, from a score that can be fooled to an LLM that reads code plus deterministic checks. Whether that judge has its own systematic bias is the next thing someone should measure.

Next week this series will check whether the Resolution Hypothesis data and code are out, whether ELITE is open-sourced, and whether anyone has re-run SAGE, GEPA, or similar methods under repaired evaluations.

Sources

  1. Zeng et al., The Resolution Hypothesis: Agent Evolution’s Key Is Identifying Improvements, arXiv:2611.23068, 2026-11-17: https://arxiv.org/abs/2611.23068. Project page: https://lineshogan.github.io/resolution-hypothesis/ ↩
  2. Ren et al., Evaluation-Free Recursive Self-Improvement for Coding Agents (ELITE), arXiv:2611.21056, 2026-11-16: https://arxiv.org/abs/2611.21056 ↩
  3. EvolvingLMMs-Lab/SkillOpt-Lite, commit 4cb4eee (2026-07-22): https://github.com/EvolvingLMMs-Lab/SkillOpt-Lite. Data splits: https://huggingface.co/datasets/yshenaw/SkillOpt_Lite_Benchmarks ↩
  4. Zhang et al., Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents, arXiv:2505.22954: https://arxiv.org/abs/2505.22954 ↩
  5. Zhang et al., GUARD: Stabilizing Recursive Self-Improvement with Regret Guidance, arXiv:2611.23476, 2026-11-17: https://arxiv.org/abs/2611.23476 ↩
  6. Yu et al., Self-Improvement Memory Enables Continual Recursive Self-Improvement, arXiv:2611.18856, 2026-11-13: https://arxiv.org/abs/2611.18856 ↩
  7. Qu et al., CompoundingKernel: Self-Improving from Zero to First, arXiv:2611.19230, 2026-11-14: https://arxiv.org/abs/2611.19230 ↩
  8. Wang et al., SAGE: A Statistical Acceptance Gate for Self-Evolving Agents, arXiv:2609.36043: https://arxiv.org/abs/2609.36043. On this site: SAGE: A Statistical Acceptance Gate for Self-Evolving Skills ↩
  9. Gizmodo, Anthropic Says Its Recursive Self-Improvement Alarm Has Been Triggered, 2026-11-16: https://gizmodo.com/anthropic-says-its-recursive-self-improvement-alarm-has-been-triggered-2000840253 ↩

The test scripts in this post (livemath_shortcut.py, ssb_grader_demo.py, resolution_sim.py) use only SkillOpt-Lite’s public data and code and call no model API.