Generation 1Return: 176.45

Generation 10Return: 780.55

CarRacing, search 3, episode seed 1000. Same search, same starting seed. Episode return rises from 176.45 to 780.55. The task’s reporting target remains unmet.

Overview

Recursive self-improvement is a topic of great discussion. While experts cannot agree on exactly what RSI means, they can agree that it will be exist as some sort of system that improves over time. Our belief is that we can design a meta-algorithm that evolves programs over a period of time that then get better and better. Then, we set the target of what that meta-algorithm optimizes to the meta-algorithm itself. By demonstrating this, we can show that such a system exists that can not only find policies that improve over time, but also that even the ability to find those policies can improve, thus creating 'recursion' to the self-improvement (hence, RSI).

EliteTable is one such attempt at designing the first layer of meta-algorithm that could then go to create the next-generation of itself. We applied it to 6 classic control reinforcement tasks and found that generally, it improves over time while maintaining reasonable performance in held-out runs.

However, EliteTable is not flawless. It struggles on certain problems for reasons currently unknown. More research must be conducted so that we better understand exactly the mechanisms that add to its performance and what classes of problems it fails to improve against such that we can design a better and more generalized model.

How EliteTable works

As is so with most other self-improving systems, EliteTable follows a training loop that is pretty much identical to those found in evolutionary computing:

  1. 1. ProposeGenerate 50 candidates using new ideas, edits, and remixes.
  2. 2. EvaluateRun each candidate on ten fixed episodes and average its return.
  3. 3. RetainKeep the ten highest-scoring programs from the candidates and existing table.
  4. 4. RepeatUse the retained programs to guide the next generation.

Where EliteTable differs from other algorithms is in its design.

EliteTable keeps a fixed size leaderboard that subsequent generations and attempts at better solutions to a problem must compete against. Once a new solution makes it to the leaderboard and kicks out the loser, it stays there until something better beats it, at which point it drops down on the list accordingly.

To enable innovation and exploration of concepts, EliteTable features 3 operators:

  1. New - proposes a brand new solution.
  2. Edit - makes a single edit to one of the leaders on the leaderboard
  3. Remix - creates one new solution by mixing together the concepts from three current leaders.

For each generation, the distribution of operators after the initial seed generation is 20% new, 40% edit, and 40% remix.

What changes is the policy source code. The language model and search procedure stay fixed. Separate held-out evaluations happen after search finishes and never influence which programs are retained.

How it Performed

MountainCar

EliteTable generally did well on the 6 problems it had to solve. Below are videos of the first generation's winner versus the last generation's winner in a search against the MountainCar task.

As you can see, the final generation gets to the flag in fewer timesteps than the first generation winner, demonstrating that EliteTable has provided improving policies over time.

Generation 1171 steps

Generation 10102 steps

MountainCar, search 6, episode seed 1000. These are new replays of the archived policies; both returns match the recorded episode. Each video plays at its original speed and ends when its episode finishes.

Across 100 held-out episodes, this search’s mean return improves from −128.70 to −97.69, passing the −110 reporting target. Higher return is better; MountainCar charges one point for each step taken.

CarRacing

On some of the environments and tasks, EliteTable was able to solve it completely (by reaching the goal, surviving for long enough, finishing a task in a certain amount of time.) However, in the CarRacing environment, EliteTable was not able to reach a solved state in it's 500 proposals or 10 generations.

EliteTable was still able to provide a highly improved policy which demonstrates clear driving ability. In the following study, you can see exactly how the two policies differ between first generation and last:

Inside the CarRacing controllers

Each diagram follows one camera frame through to an action. The process repeats at every step.

Generation 1Brightness heuristic
Generation 1: grayscale frame → average lower-half columns → brightness-weighted offset → steering. Border brightness below 100 selects gas 0.5; otherwise gas 0.1. No braking or control memory.
Steers by brightness, with two gas levels and no memory between frames.
Generation 10Road geometry and control memory
Generation 10: detect road → estimate offset and heading → update speed estimate → set target speed and steering → gas at or below target, brake above target → remember action. Steering uses road geometry and previous steering; larger heading angles lower the speed target.
Steers by road geometry and uses previous actions to estimate speed and adjust gas and braking.

Does the improvement generalize?

Search rewards come from ten fixed episodes. To test whether the selected programs work beyond those episodes, we evaluated each distinct generation winner on 100 separate held-out episode seeds.

Search median Held-out median Reporting target
CartPole
CartPole: search and held-out median returns remain at 500 across all ten generations, above the 475 reporting target.
MountainCar
MountainCar: held-out median return improves from −119.18 to −107.07, crossing the −110 target. Search median improves from −120.30 to −114.90.
Acrobot
Acrobot: held-out median return declines from −94.83 to −101.53 while search median improves from −82.00 to −74.55. The reporting target is −100.
LunarLander
LunarLander: held-out median return improves from −39.04 to 214.04, crossing the 200 target. Search median improves from −32.79 to 258.04.
BipedalWalker
BipedalWalker: held-out median return improves from −39.30 to −7.65, remaining below the 300 reporting target. Search and held-out medians closely track each other.
CarRacing
CarRacing: held-out median return improves from 206.66 to 691.81; search median improves from 226.05 to 775.30. Both remain below the 900 reporting target.
Ten searches per task. Shading shows the middle half of held-out results; faint orange lines show individual searches. Each task has its own vertical scale. MountainCar generation 6 uses nine complete held-out evaluations; its missing result is excluded.

All ten searches improved between their first and final held-out evaluations in MountainCar, LunarLander, BipedalWalker, and CarRacing. CartPole’s selected winners were already at the ceiling in generation one. In Acrobot, six of ten searches declined, and the median paired change was negative.

Held-out performance can fall between generations even as the best recorded search score stays level or rises. Retaining strong programs protects their recorded scores; it does not guarantee better performance on new episodes.

Progress—and remaining limits

Improvement also appears in the newly generated candidates. Their median search scores rise from the first to the final generation in all six tasks. CarRacing shows the pattern clearly: a stronger group emerges while a low-scoring group persists.

CarRacing candidate-score distributions from generations 1 through 10, stacked from bottom to top. Later generations develop a stronger high-return group, while a substantial low-return group remains.
Newly evaluated CarRacing candidates, pooled across ten searches. Each score averages ten search episodes. The distributions describe the candidate population, rather than only its retained winners.

BipedalWalker and CarRacing improved without reaching their reporting targets. The tested budget does not establish whether more generations would solve them. These tasks are also well known, so the language model may have encountered relevant solutions during training.

The next steps are budget-matched comparisons with other search methods, experiments that isolate the contribution of editing, remixing, retention, and repair, and evaluation across new episode panels and language models. The present study documents improvement under one tested configuration; it does not establish superiority over other methods.

The paper provides the full results and limitations. The code and recorded evidence support inspection of the search procedure and reconstruction of its core results.