Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .autoloop/programs/tsb-perf-evolve/code/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This directory holds the **fixed inputs** for the program: the benchmark scripts

## Files

- `config.yaml` — tunables read by the AlphaEvolve playbook (`exploitation_ratio`, `num_islands`, `population_size`, `archive_size`, dataset size).
- `config.yaml` — tunables read by the OpenEvolve playbook (`exploitation_ratio`, `num_islands`, `population_size`, `archive_size`, dataset size).
- `benchmark.ts` — tsb-side benchmark. Builds a Series of `dataset_size` random floats with ~5% NaN, calls `sortValues` in a tight loop, prints `{"function": "Series.sortValues", "mean_ms": …, "iterations": …, "total_ms": …}`.
- `benchmark.py` — pandas-side benchmark. Builds an equivalent `pd.Series`, calls `.sort_values()` in the same loop structure, prints the same JSON shape.

Expand Down
4 changes: 2 additions & 2 deletions .autoloop/programs/tsb-perf-evolve/code/config.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# AlphaEvolve tunables — read by strategy/alphaevolve.md every iteration.
# OpenEvolve tunables — read by strategy/openevolve.md every iteration.

# Operator weights. Must sum to 1.0. Defaults bias toward exploitation.
exploitation_ratio: 0.50
Expand All @@ -7,7 +7,7 @@ crossover_ratio: 0.15
migration_ratio: 0.05

# Island count. Should match the number of islands enumerated in
# strategy/alphaevolve.md's "Pick parent(s)" section.
# strategy/openevolve.md's "Pick parent(s)" section.
num_islands: 5

# MAP-Elites population caps.
Expand Down
6 changes: 3 additions & 3 deletions .autoloop/programs/tsb-perf-evolve/program.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Concretely, we minimize the **ratio**

`fitness < 1.0` means tsb is faster than pandas; lower is better. We will keep iterating as long as fitness keeps improving.

This is a **performance-evolution program** — there is one self-contained artifact (`Series.sortValues`), one scalar fitness (the ratio), and many plausible algorithmic families to try (comparison sort, typed-array indirect sort, dtype-dispatched non-comparison sort, batched/SoA, etc.). It is the canonical case for the AlphaEvolve strategy.
This is a **performance-evolution program** — there is one self-contained artifact (`Series.sortValues`), one scalar fitness (the ratio), and many plausible algorithmic families to try (comparison sort, typed-array indirect sort, dtype-dispatched non-comparison sort, batched/SoA, etc.). It is the canonical case for the OpenEvolve strategy.

### Validity invariants

Expand Down Expand Up @@ -43,10 +43,10 @@ Do NOT modify:

## Evolution Strategy

This program uses the **AlphaEvolve** strategy. On every iteration, read `strategy/alphaevolve.md` and follow it literally — it supersedes the generic analyze/accept/reject steps in the default autoloop loop.
This program uses the **OpenEvolve** strategy (modeled on [openevolve](https://github.com/algorithmicsuperintelligence/openevolve)). On every iteration, read `strategy/openevolve.md` and follow it literally — it supersedes the generic analyze/accept/reject steps in the default autoloop loop.

Support files:
- `strategy/alphaevolve.md` — the runtime playbook (operators, parent selection, population rules).
- `strategy/openevolve.md` — the runtime playbook (operators, parent selection, population rules).
- `strategy/prompts/mutation.md` — framing for exploitation and exploration operators.
- `strategy/prompts/crossover.md` — framing for crossover and migration operators.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# AlphaEvolve Strategy — tsb-perf-evolve
# OpenEvolve Strategy — tsb-perf-evolve

This file is the **runtime playbook** for this program. The autoloop agent reads it at the start of every iteration and follows it literally. It supersedes the generic "Analyze and Propose" / "Accept or Reject" steps in the default autoloop iteration loop — all other steps (state read, branch management, state file updates) still apply.

Expand Down Expand Up @@ -90,7 +90,7 @@ Append a new entry to the `## 🧬 Population` subsection in the state file usin

### Step 8. Fold through to the default loop

Continue with the normal autoloop Step 5 (Accept or Reject → commit / discard, update state file's Machine State, Iteration History, Lessons Learned, etc.) as defined in the workflow. The only additional requirements from AlphaEvolve are:
Continue with the normal autoloop Step 5 (Accept or Reject → commit / discard, update state file's Machine State, Iteration History, Lessons Learned, etc.) as defined in the workflow. The only additional requirements from OpenEvolve are:

- The Iteration History entry must include `operator`, `parent_id(s)`, `island`, and `fitness` fields (in addition to the normal status/change/metric/notes).
- Lessons Learned additions should be phrased as *transferable heuristics* about the problem space, not as reports of what this iteration did. (E.g. "Indirect sort over `Uint32Array` indices beats object-pair sort above n≈10k" — not "Iteration 17 tried indirect sort.")
Expand All @@ -113,7 +113,7 @@ The population lives in the state file `tsb-perf-evolve.md` on the `memory/autol
```markdown
## 🧬 Population

> 🤖 *Managed by the AlphaEvolve strategy. One entry per candidate that has been evaluated (accepted or rejected). Newest first.*
> 🤖 *Managed by the OpenEvolve strategy. One entry per candidate that has been evaluated (accepted or rejected). Newest first.*

### Candidate <id> · island <n> · fitness <score> · gen <iter>

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Crossover & Migration prompt — tsb-perf-evolve

You are about to apply a **two-parent operator** — either crossover (combine ideas from parents on different islands) or migration (graft a technique that works on one island into a solution on another). This file frames how to reason about that change. Use it together with `strategy/alphaevolve.md`.
You are about to apply a **two-parent operator** — either crossover (combine ideas from parents on different islands) or migration (graft a technique that works on one island into a solution on another). This file frames how to reason about that change. Use it together with `strategy/openevolve.md`.

## What these operators are for

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Mutation prompt — tsb-perf-evolve

You are about to apply a **single-parent operator** — either exploitation (refine an elite) or exploration (try something new in an under-represented island). This file frames how to reason about that change. Use it together with `strategy/alphaevolve.md`.
You are about to apply a **single-parent operator** — either exploitation (refine an elite) or exploration (try something new in an under-represented island). This file frames how to reason about that change. Use it together with `strategy/openevolve.md`.

## What this operator is for

Expand Down
Original file line number Diff line number Diff line change
@@ -1,19 +1,21 @@
# Adopting the AlphaEvolve strategy for a new program
# Adopting the OpenEvolve strategy for a new program

This file is a **creator-time guide**it is read by the maintainer (or a "create program" agent) **once**, when authoring a new program that wants to use AlphaEvolve. It is **not** copied into the program's `strategy/` directory and is **not** read by the iteration agent at runtime.
> **Inspiration.** This strategy is modeled on [OpenEvolve](https://github.com/algorithmicsuperintelligence/openevolve)an open-source implementation of the evolutionary-code-search approach popularized by DeepMind's AlphaEvolve paper. We've adapted the core ideas (MAP-Elites niching, island model, four operators — exploitation / exploration / crossover / migration) into a playbook the autoloop agent follows at iteration time. Consult the OpenEvolve repo for background on the underlying algorithm and worked examples.

If you are an iteration agent and have somehow ended up here: stop, go back to `strategy/alphaevolve.md` in the program directory, and follow that.
This file is a **creator-time guide** — it is read by the maintainer (or a "create program" agent) **once**, when authoring a new program that wants to use OpenEvolve. It is **not** copied into the program's `strategy/` directory and is **not** read by the iteration agent at runtime.

## When to pick AlphaEvolve
If you are an iteration agent and have somehow ended up here: stop, go back to `strategy/openevolve.md` in the program directory, and follow that.

AlphaEvolve is the right strategy when **all** of the following are true:
## When to pick OpenEvolve

OpenEvolve is the right strategy when **all** of the following are true:

- The target is a **self-contained artifact** — a single function, a single file, a config blob — that can be replaced atomically each iteration.
- Fitness is a **scalar metric** the evaluator can produce in a few seconds to a few minutes (lower or higher is better — pick one).
- There are **multiple plausible algorithmic families**, not just one obvious approach with knobs to tune. AlphaEvolve's island model is wasted if everything collapses to one family.
- Iterations are **independent** — a candidate's fitness does not depend on the previous candidate's state. (If you need to *accumulate* changes, use the default loop, not AlphaEvolve.)
- There are **multiple plausible algorithmic families**, not just one obvious approach with knobs to tune. OpenEvolve's island model is wasted if everything collapses to one family.
- Iterations are **independent** — a candidate's fitness does not depend on the previous candidate's state. (If you need to *accumulate* changes, use the default loop, not OpenEvolve.)

If the program is "add another test", "port another feature", or any kind of coverage / accumulation task — **do not use AlphaEvolve**. Use the default loop.
If the program is "add another test", "port another feature", or any kind of coverage / accumulation task — **do not use OpenEvolve**. Use the default loop.

## Steps to adopt

Expand All @@ -22,15 +24,15 @@ If the program is "add another test", "port another feature", or any kind of cov

```bash
mkdir -p .autoloop/programs/<program-name>/strategy/prompts
cp .autoloop/strategies/alphaevolve/strategy.md \
.autoloop/programs/<program-name>/strategy/alphaevolve.md
cp .autoloop/strategies/alphaevolve/prompts/mutation.md \
cp .autoloop/strategies/openevolve/strategy.md \
.autoloop/programs/<program-name>/strategy/openevolve.md
cp .autoloop/strategies/openevolve/prompts/mutation.md \
.autoloop/programs/<program-name>/strategy/prompts/mutation.md
cp .autoloop/strategies/alphaevolve/prompts/crossover.md \
cp .autoloop/strategies/openevolve/prompts/crossover.md \
.autoloop/programs/<program-name>/strategy/prompts/crossover.md
```

3. Resolve every `<CUSTOMIZE: …>` marker in `strategy/alphaevolve.md` and the two prompt files. See the marker-by-marker guidance below.
3. Resolve every `<CUSTOMIZE: …>` marker in `strategy/openevolve.md` and the two prompt files. See the marker-by-marker guidance below.
4. Add the `## Evolution Strategy` pointer block to `program.md` (template below).
5. Sanity-check: `grep -R "<CUSTOMIZE" .autoloop/programs/<program-name>/strategy/` should return **nothing**.

Expand All @@ -41,10 +43,10 @@ Replace (or add) `program.md`'s `## Evolution Strategy` section with exactly thi
```markdown
## Evolution Strategy

This program uses the **AlphaEvolve** strategy. On every iteration, read `strategy/alphaevolve.md` and follow it literally — it supersedes the generic analyze/accept/reject steps in the default autoloop loop.
This program uses the **OpenEvolve** strategy. On every iteration, read `strategy/openevolve.md` and follow it literally — it supersedes the generic analyze/accept/reject steps in the default autoloop loop.

Support files:
- `strategy/alphaevolve.md` — the runtime playbook (operators, parent selection, population rules).
- `strategy/openevolve.md` — the runtime playbook (operators, parent selection, population rules).
- `strategy/prompts/mutation.md` — framing for exploitation and exploration operators.
- `strategy/prompts/crossover.md` — framing for crossover and migration operators.

Expand All @@ -55,7 +57,7 @@ Population state lives in the state file on the `memory/autoloop` branch under t

### `strategy.md` markers

- **`# AlphaEvolve Strategy — <CUSTOMIZE: program-name>`** — the program name as it appears in the file path.
- **`# OpenEvolve Strategy — <CUSTOMIZE: program-name>`** — the program name as it appears in the file path.
- **`## Problem framing`** — 2–4 sentences. State the artifact, the fitness function, and the validity invariants. The agent reads this every iteration; make it dense.
- **Operator weight table** — only change defaults if you have a strong prior. The defaults bias toward exploitation, which is right for most perf problems.
- **Islands** — the most important thing to get right. Pick 3–6 **algorithmic families** that span the design space. Examples:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Crossover & Migration prompt — <CUSTOMIZE: program-name>

You are about to apply a **two-parent operator** — either crossover (combine ideas from parents on different islands) or migration (graft a technique that works on one island into a solution on another). This file frames how to reason about that change. Use it together with `strategy/alphaevolve.md`.
You are about to apply a **two-parent operator** — either crossover (combine ideas from parents on different islands) or migration (graft a technique that works on one island into a solution on another). This file frames how to reason about that change. Use it together with `strategy/openevolve.md`.

## What these operators are for

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Mutation prompt — <CUSTOMIZE: program-name>

You are about to apply a **single-parent operator** — either exploitation (refine an elite) or exploration (try something new in an under-represented island). This file frames how to reason about that change. Use it together with `strategy/alphaevolve.md`.
You are about to apply a **single-parent operator** — either exploitation (refine an elite) or exploration (try something new in an under-represented island). This file frames how to reason about that change. Use it together with `strategy/openevolve.md`.

## What this operator is for

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
# AlphaEvolve Strategy — <CUSTOMIZE: program-name>
# OpenEvolve Strategy — <CUSTOMIZE: program-name>

> **Inspiration.** This strategy is modeled on [OpenEvolve](https://github.com/algorithmicsuperintelligence/openevolve) — an open-source implementation of the evolutionary-code-search approach popularized by DeepMind's AlphaEvolve paper. We've adapted the core ideas (MAP-Elites niching, island model, four operators — exploitation / exploration / crossover / migration) into a playbook the autoloop agent follows at iteration time. Consult the OpenEvolve repo for background on the underlying algorithm and worked examples.

This file is the **runtime playbook** for this program. The autoloop agent reads it at the start of every iteration and follows it literally. It supersedes the generic "Analyze and Propose" / "Accept or Reject" steps in the default autoloop iteration loop — all other steps (state read, branch management, state file updates) still apply.

Expand Down Expand Up @@ -84,7 +86,7 @@ Append a new entry to the `## 🧬 Population` subsection in the state file usin

### Step 8. Fold through to the default loop

Continue with the normal autoloop Step 5 (Accept or Reject → commit / discard, update state file's Machine State, Iteration History, Lessons Learned, etc.) as defined in the workflow. The only additional requirements from AlphaEvolve are:
Continue with the normal autoloop Step 5 (Accept or Reject → commit / discard, update state file's Machine State, Iteration History, Lessons Learned, etc.) as defined in the workflow. The only additional requirements from OpenEvolve are:

- The Iteration History entry must include `operator`, `parent_id(s)`, `island`, and `fitness` fields (in addition to the normal status/change/metric/notes).
- Lessons Learned additions should be phrased as *transferable heuristics* about the problem space, not as reports of what this iteration did. (E.g. "Hex layouts dominate grid layouts above n=20" — not "Iteration 17 tried a hex layout.")
Expand All @@ -107,7 +109,7 @@ The population lives in the state file `{program-name}.md` on the `memory/autolo
```markdown
## 🧬 Population

> 🤖 *Managed by the AlphaEvolve strategy. One entry per candidate that has been evaluated (accepted or rejected). Newest first.*
> 🤖 *Managed by the OpenEvolve strategy. One entry per candidate that has been evaluated (accepted or rejected). Newest first.*

### Candidate <id> · island <n> · fitness <score> · gen <iter>

Expand Down
2 changes: 1 addition & 1 deletion .autoloop/strategies/test-driven/CUSTOMIZE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Test-Driven is the right strategy when **all** of the following are true:
- Iterations **accumulate**: each iteration pins one more behaviour (or fixes one more bug), and the work product grows monotonically. You're not searching for a single best artifact; you're building up a body of pinned behaviour.
- There exists a **source of truth** the agent can consult when ambiguity arises (a reference implementation, a spec document, an issue with a reproducer, etc.).

If the program is "make this faster" or "minimize this scalar", **do not use Test-Driven**. Use AlphaEvolve (`.autoloop/strategies/alphaevolve/`).
If the program is "make this faster" or "minimize this scalar", **do not use Test-Driven**. Use OpenEvolve (`.autoloop/strategies/openevolve/`).

If the program is genuinely "do whatever the agent thinks is best", neither strategy fits — use the default loop.

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/autoloop.md
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,7 @@ Each run executes **one iteration for the single selected program**:
Before executing the generic iteration loop below, check whether this program has opted into a specialized strategy:

1. Read `<program-dir>/program.md` and look for a `## Evolution Strategy` section.
2. If that section points to a strategy file — e.g., "This program uses the **AlphaEvolve** strategy. Read `strategy/alphaevolve.md` at the start of every iteration and follow it literally." — read the referenced file and follow it.
2. If that section points to a strategy file — e.g., "This program uses the **OpenEvolve** strategy. Read `strategy/openevolve.md` at the start of every iteration and follow it literally." — read the referenced file and follow it.
3. The strategy playbook **supersedes** the generic "Step 2: Analyze" through "Step 5: Accept or Reject" steps below. The other steps (state read, branch management, state file updates, CI gating) still apply.
4. If `## Evolution Strategy` is absent, contains only prose, or points to a file that does not exist, fall back to the default iteration flow below.

Expand Down
Loading