Evolve numerical methods with Agent Elo¶
Run two independent agents on a small numerical-integration problem, compare their evidence, and let successful agents produce descendants. This walkthrough uses two generations to show how a child inherits a solution and then develops it further.
The task is to integrate exp(-x^2) from zero to one. Agents must execute Python,
compare their result with a reference computed using math.erf, and save their
implementation and validation results. The task itself specifies the evaluation
criteria seen by both competitors and the judge.
For competition rules, persistence details, and configuration options, see the Agent Elo guide.
1. Set up the example¶
Clone URSA and start from the repository root. You will need
uv, Python 3.11 or newer, and credentials for the
model used in run_elo.py. The supplied runner uses OpenAI's gpt-5
for both competitors and the judge.
The local pyproject.toml installs URSA from this checkout in editable mode. Model and credential configuration is described in Models and inference providers.
2. Inspect the population¶
Open agent_elo.yaml. It defines two ExecutionAgent members,
two generations per invocation, and at most one replacement per generation.
The seed controls pairing and selection tie-breaking, not the model responses.
name: numerical_integration_elo
group: default
description: >
A small evolutionary environment where competing agents develop and
improve numerical solutions to a simple scientific-computing problem.
initial_rating: 1500
k_factor: 32
deaths_per_round: 1
# Reproducible random pairing and Elo tie-breaking.
seed: 12345
# One env.invoke() runs both generations.
generations: 2
# Set this to a saved environment_state.json to resume a previous run.
restart_from_json: null
# Specify evaluation criteria in the task supplied to invoke().
# judge_prompt is optional additional guidance.
members:
- name: researcher_1
role: >
Independent computational scientist developing and validating
numerical methods.
agent: ExecutionAgent
- name: researcher_2
role: >
Independent computational scientist developing and validating
numerical methods.
agent: ExecutionAgent
Keep member names unique and the population even. For this first run, leave the two-member population and generation count unchanged.
3. Run the numerical task¶
From this example directory, run:
The runner loads the adjacent YAML, supplies the task, prints each generation's
matches and standings, and reports the restart snapshot path. Its TASK string
defines correctness, executed validation, reproducibility, and improvement as
the evaluation criteria. Change those criteria when adapting the example to a
different problem.
from pathlib import Path
from langchain.chat_models import init_chat_model
from ursa.environments import AgentEloEnvironment
TASK = """
Numerically evaluate
integral_0^1 exp(-x^2) dx
using Python.
This is an execution task: create and run code in your workspace.
Requirements:
- implement a numerical integration method;
- compare the numerical result with the reference value computed from
math.erf;
- report the absolute error;
- perform at least one convergence or accuracy check;
- save the executable code and useful numerical results in your workspace.
Keep the calculation small and reproducible.
Evaluation criteria, in priority order:
1. Correctness and numerical accuracy.
2. Evidence from executed validation and convergence checks.
3. Reproducibility of the implementation and results.
4. Meaningful improvement over existing work, when applicable.
For partial work, assess the available evidence using these same criteria.
Novelty alone is not an improvement.
Your final response should summarize the method, numerical result, error,
validation performed, files produced, and any remaining weakness.
"""
def print_generation(result):
print(f"\nGeneration {result['generation']}")
print("-" * 50)
for match in result["matches"]:
print(
f"{match['player_a']} vs {match['player_b']}: "
f"winner={match['winner']}"
)
print(f" {match['reasoning']}")
if result["eliminated"]:
print("Eliminated:", ", ".join(result["eliminated"]))
if result["children"]:
print("Children:", ", ".join(result["children"]))
print("Standings:")
for row in result["standings"]:
print(
f" {row['rank']}. {row['name']} "
f"Elo={row['rating']:.2f} "
f"generation={row['generation']} "
f"parent={row['parent']}"
)
def main():
llm = init_chat_model(
model="openai:gpt-5",
timeout=None,
max_retries=2,
)
env = AgentEloEnvironment.from_yaml(
"agent_elo.yaml",
llm=llm,
workspace=Path("./workspace_agent_elo"),
persist_members=True,
)
try:
result = env.invoke({"task": TASK})
for generation in result["generations"]:
print_generation(generation)
print("\nFinal standings")
print("-" * 50)
for row in result["standings"]:
print(f"{row['rank']}. {row['name']} Elo={row['rating']:.2f}")
print(
"\nRestart state:",
result["environment_state"],
)
finally:
for member in env.members.values():
close = getattr(member, "close", None)
if callable(close):
close()
if __name__ == "__main__":
main()
The runner creates workspace_agent_elo/ under this directory. Each competitor
works in a separate subdirectory. After a decisive first match, the selected
survivor produces a child; in the second generation, that child starts from its
parent's saved work. A draw produces no replacement.
4. Inspect the results¶
Look for the following in the printed output:
- The winner and reasoning for each match, or a draw.
- Names of eliminated agents and their replacements.
- Updated ratings and the parent of each descendant.
- The path to
environment_state.json.
The reference integral is approximately 0.746824132812427. Inspect the agents'
saved scripts and convergence checks to understand how closely their methods
approach it. Filenames, ratings, winners, and numerical accuracy depend on the
generated work; this example does not prescribe an exact transcript.
Compare the first and second generation's artifacts. Did the survivor strengthen its validation? Did a descendant implement a useful improvement? A high inherited rating alone does not demonstrate new work, and children born at the end of the last generation have not yet competed.
5. Continue or adapt the experiment¶
To continue from a completed run, set restart_from_json in agent_elo.yaml to
./workspace_agent_elo/environment_state.json and rerun the script. Keep the
environment name, group, workspace, and member persistence directories intact.
The runner supplies the task again, and generations: 2 requests two additional
generations. See restart behavior
for which settings come from the snapshot.
For a new experiment, use a new environment name in the YAML and a new workspace path in the runner. You can then vary the agents' roles, add an even number of competitors, or change the task and evaluation criteria.
Optionally set member_timeout_seconds in the YAML. Timed-out agents remain
eligible for judging through saved progress reports or brief workspace
inspection. Start without a timeout so you can observe the complete workflow.
Cleanup¶
The script closes its active members when it exits. To discard this example's
data, remove workspace_agent_elo/ and the named agent directories beginning
with numerical_integration_elo_ under
~/.cache/ursa/default/agents/, including any descendants. Keep both locations
if you intend to restart; the JSON snapshot alone is not a full backup.
The example's .venv/ can also be removed when you no longer need its local
dependencies. Avoid removing unrelated agents from the shared URSA cache.