V2.2/V2.3/V2.3.1 — Indoors phase + deterministic findPath + town/castle classification#8
Closed
ArturSkowronski wants to merge 363 commits into
Closed
V2.2/V2.3/V2.3.1 — Indoors phase + deterministic findPath + town/castle classification#8ArturSkowronski wants to merge 363 commits into
ArturSkowronski wants to merge 363 commits into
Conversation
Update Licence (GPLv3)
Update README.md
Fixing errors with controller logic
…or and handle MemoryRomLoader
Nes.kt refactor
Update JetBrains Junie Workflow
Initialize JetBrains Junie 🚀
UI Fixes from KotlinConf
Kotlin 2.0 Migration
chore: update Compose 1.8.2 and Gradle wrapper versions
…et before Garland
FF1 agent V2 — first autonomous run (boot→overworld), OutOfBudget before Garland
…ecutor input/result, no truncation) + FF1-aware prompts
…; softer 'must call tool' phrasing
FF1 agent V2.1 — FF1-aware prompts + ~/.knes traces; agent reasons but stuck in Coneria castle
…n-castle root cause)
…g-of-war Addresses the (146, 152) blocked-by-terrain deadend evidenced in PR #97 by adding a deterministic 16x16 BFS pathfinder, a fog-of-war accumulator, and ASCII map rendering for the advisor — keeping the executor's overall control flow unchanged. Informed by Gemini Plays Pokemon research: textual tile grids match or beat raw screenshots for spatial reasoning, so V2.3 starts text-only and defers screenshot augmentation to V2.4.
…idence run + PR steps
…es NametableReader PPU nametable approach didn't work: FF1 uses scrolling + mid-frame writes, so NT0/NT1 dumps don't reflect the actual visible terrain at any meaningful sample point. Pivoted to FF1 ROM map data: pointer table at file offset 0x4010, RLE rows decoded into a 256x256 ByteArray. Tile bytes are ALREADY game-classified (grass/forest/mountain/water/town/castle/bridge), so OverworldTileClassifier is a hardcoded lookup. Removed: NametableReader, NametableReaderLiveTest, TileClassifierResearchTest. Added: OverworldMap, OverworldTileClassifier, OverworldMapTest (5 tests pass). Format reference: https://datacrystal.tcrf.net/wiki/Final_Fantasy/World_map_data
…optional ViewportMap
…ViewportSource)
…SCII map + fog stats
…dvisorAgent in Main
…ew bug at town entrance
V2.3 live run revealed party walks INTO Coneria town at overworld tile (145,152) but RAM says locationType=0 (not 0xD1) while localX/Y populate to non-zero. Our classifier treated this as Overworld and walkOverworldTo futilely retried world coords that don't change. Fix is two-fold: 1) RamObserver: phase = Indoors when partyCreated AND (locationType==0xD1 OR localX/Y != 0). Captures both castle interiors AND town outdoor maps. 2) ExitBuilding: terminate on (locationType==0 AND localX==0 AND localY==0) rather than just locationType==0 — towns already have locationType=0 entering, old condition returned ok=true immediately without moving. Now the executor will see Indoors → call exitBuilding → walk SOUTH → reach overworld → resume northward navigation toward Garland's bridge.
Author
|
Wrong repo target; reopening on ArturSkowronski/kNES |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three stacked feature increments to the FF1 Koog agent. The narrative below is the
full debugging journey — what we thought the bug was, what the live runs revealed,
how the design evolved.
exitBuildingskill (1 commit, was already on this branch)findPathBFS + ROM-decoded overworld map + ASCII map for advisor (12 commits)Specs:
docs/superpowers/specs/2026-05-02-ff1-koog-agent-v2-3-design.mdPlan:
docs/superpowers/plans/2026-05-02-ff1-koog-agent-v2-3.mdEvidence:
docs/superpowers/runs/2026-05-02-v2-3-deadend-escape/,docs/superpowers/runs/2026-05-02-v2-3-1-town-fix/The bug we set out to fix (V2.1 evidence)
Per PR #97 trace: the agent reached Overworld(146, 158) after party creation, walked
6 tiles north to (146, 152), then ran out of budget over 22 turns trying to walk
further north. The model self-diagnosed "stuck inside Coneria castle" — wrong: real
RAM said
locationType=0(overworld). The actual cause was terrain blocking: northof (146, 152) is mountain. The greedy
walkOverworldToskill had no obstacle awareness,just kept pressing UP forever.
V2.2 — wrong fix (already on this branch as commit
29aa653)Initial hypothesis: model was right that party was indoors. Added
Indoorsphasedetection (
locationType==0xD1) +exitBuildingskill (walks SOUTH out of castles).A re-run showed: the stuck position was Overworld, not Indoors. V2.2 was honest
infrastructure for future castle/town entries, but didn't fix V2.1's actual bug.
Kept on the branch as "valid for later".
V2.3 — research, then design pivot
Research input
Web research on Gemini Plays Pokemon revealed three findings that shaped V2.3:
pathfinderwas their main tool — sub-agent in blank context with structured map.We brainstormed four V2.3 paths (A: screenshots, B: hardcoded waypoints, C: obstacle-
aware walk, D: advisor-only screenshots) and converged on D'+C — advisor gets ASCII
map + fog-of-war, executor gets a deterministic pathfinder.
Plan revision: pathfinder becomes deterministic
User question "could we give the agent a deterministic tool for path detection in the
viewport?" reframed the design.
findPath(targetX, targetY)exposed as @tool, no LLMtokens consumed, BFS over 16x16 viewport. API stable across versions; only the
SearchSpaceenum widens (VIEWPORT today, FOG / FULL_MAP / A* / LLM-pathfinderlater).
First implementation attempt: read PPU nametable
Plan: read 16x16 tile IDs around party from PPU nametable, classify via empirical
tile-classifications/ff1-overworld.json(built by a research test that dumps hexgrids alongside PNG screenshots).
Implemented (T1-T5). Hit a wall at T6:
controller.setButtonsadvanceFramesdirectly; FF1 needs the InputQueue + tap pattern viaEmulatorToolset).overworld.
NT1 has overworld content, but it's the same content before and after walking 4
tiles north (worldY 0x9E -> 0x9A). Conclusion: PPU nametable is not a reliable
terrain source for FF1. Mid-frame writes overwrite NT after rendering; PPUSCROLL
shifts the visible window; we sample post-frame which catches scratch state.
Design pivot: parse FF1 ROM map directly
FF1 stores its 256x256 overworld as RLE-compressed pointer-table-indexed rows in PRG
bank 1 (file offset 0x4010). Tile bytes are already game-classified (grass/forest/
mountain/water/town/castle/...) per the wiki. Parser is ~50 LOC.
Replaced
NametableReaderwithOverworldMap.fromRom(File)+OverworldTileClassifier.The interface contract
ViewportSource(a fun-interface withreadViewport(xy)) letsOverworldMapand any future map source plug in interchangeably.5/5 OverworldMap tests pass against the real FF1 ROM. Coneria castle and town tiles
are findable in the bounding box around spawn.
Live run #1 (V2.3 raw)
Out-of-budget after 13 turns. But: party advanced spawn -> (145, 152) in 1 turn,
which is exactly the V2.1 deadend tile. Pathfinder + BFS worked.
Two issues surfaced:
maxIterations=10on Koog agent was too tight for V2.3's larger tool surface(findPath chained with walkOverworldTo). Bumped to 20.
findPathconsistently returned BLOCKED at (145, 152) even though theOverworldMap showed plenty of grass. Cause discovered in the trace's RAM dump:
worldX=145, worldY=152, localX=5, localY=28, locationType=0. Party walked INTOConeria town; the overworld coord stayed sticky while local-map navigation took
over. RamObserver only treated
locationType==0xD1as Indoors -> false-classifiedas Overworld ->
walkOverworldTofutilely retried world coords that didn't change.V2.3.1 — phase classifier fix
RamObserver.classify: phase = Indoors whenpartyCreatedAND(
locationType==0xD1ORlocalX/Y != 0). Captures both castle interiors AND townoutdoor maps.
ExitBuilding: terminate condition changed fromlocationType==0(towns alreadyhave that, would return ok=true immediately without moving) to
locationType==0 AND localX==0 AND localY==0.Live run #2 (V2.3.1)
Phase transitions now correct:
TitleOrMenu -> Overworld(146, 158) -> Indoors(localX=5, localY=29). Executor callsexitBuildingandbattleFightAllappropriately. Partyfights 5 random encounters inside Coneria, advances localY 29 -> 17 (12 tiles of
interior progress).
exitBuildingdoesn't fully exit because FF1 castle interiors have multi-segmentgeometry (stairs, doors, sub-maps) — a simple "walk SOUTH" loop bumps into walls and
gets reset by battles. This is the next layer of debt (V2.4: ROM-decode castle
interior maps the way V2.3 decoded the overworld), out of this PR's scope.
What this PR ships
Observation(phase, ram, viewportMap?)Test plan
Known limitations / V2.4 scope
exitBuildingwalks SOUTH but FF1 castles have stairs/doors that don't form a straight line. V2.4 should decode castle interior maps from
ROM (analogous to V2.3 overworld decode) and pathfind through them.
worldX/worldYto map-coord offset: minor calibration may be needed; pathfinder'sBFS starting at party-tile may classify the start tile as impassable due to a
1-tile offset between RAM coords and map indexing.