feat(spec5): town-overlay nav — mapflags-aware readPos + minimap + 30-iter action log#14
Closed
ArturSkowronski wants to merge 610 commits into
Closed
feat(spec5): town-overlay nav — mapflags-aware readPos + minimap + 30-iter action log#14ArturSkowronski wants to merge 610 commits into
ArturSkowronski wants to merge 610 commits into
Conversation
…ew bug at town entrance
V2.3 live run revealed party walks INTO Coneria town at overworld tile (145,152) but RAM says locationType=0 (not 0xD1) while localX/Y populate to non-zero. Our classifier treated this as Overworld and walkOverworldTo futilely retried world coords that don't change. Fix is two-fold: 1) RamObserver: phase = Indoors when partyCreated AND (locationType==0xD1 OR localX/Y != 0). Captures both castle interiors AND town outdoor maps. 2) ExitBuilding: terminate on (locationType==0 AND localX==0 AND localY==0) rather than just locationType==0 — towns already have locationType=0 entering, old condition returned ok=true immediately without moving. Now the executor will see Indoors → call exitBuilding → walk SOUTH → reach overworld → resume northward navigation toward Garland's bridge.
V2.2/V2.3/V2.3.1 — Indoors phase + deterministic findPath + town/castle classification
…agnostics + live E2E
… + dumps maps 0/1/8/12
…P or south-edge exit
…needs refinement (V2.4.1)
…ust neighbour V2.4 evidence run revealed false-positive exits flagged inside Coneria town: internal walls have impassable tiles directly south of passable floor, but the playable area continues further south. Tightening the check to require the ENTIRE column south to be impassable filters out internal walls — only outer south boundary of the playable area qualifies as an implicit exit.
…eals next bug (multi-map classifier gaps)
…ypes Histogram across mapIds 0..30 revealed that: - Each interior uses its own outside-of-map padding byte: towns mostly 0x39; castle/dungeon variants 0x3c, 0x3f, 0x47. - Walls span 0x30..0x37 (V2.4 only knew 0x30/0x32-0x35 — Coneria town subset). - 0x31 is consistently floor across maps (must NOT be classified as wall even though it falls in the wall range). - Anything else in 0x00..0x7F is reasonably treated as passable floor / decoration / NPC sprite — over-permissive is self-correcting via the ExitInterior idle-detection (non-moving step gets fog-marked blocked). This unblocks mapId=24 (sub-interior reached after walking south through Coneria town in V2.4.1 evidence run).
V2.4.2 evidence run revealed agent reaches Battle, wins, but gets stuck in PostBattle (screenState=0x63) — XP/gold results screen requires per-stage A taps to dismiss. The 10-tap final flush wasn't enough. Now battleFightAll explicitly loops single A taps with frame buffers between, checking screenState each iteration, until both BATTLE and POST_BATTLE are cleared (max 30 taps). Also widens canExecute to cover PostBattle phase so the executor can call battleFightAll from either state.
…tuck in PostBattle modal
… need same as overworld) V2.4.3 evidence run revealed exitInterior fails despite findPath returning a 17-step path: party never physically moves between iterations because 16 frames isn't enough to consume one indoor tile of motion. The idle-detection then mis-marked the target tile as blocked; after ~10 iterations all neighbours were blocked and BFS returned no-path. WalkOverworldTo bumped 16→24 in V2.3 for the same reason. Apply the same to ExitInterior.
…d; ExitInterior frames bug exposed
V2.4.4 evidence run showed party still not moving with 24 frames indoors. findPath returns valid path but exitInterior loop's button presses don't register party movement. FF1 interior walk timing differs from overworld; bump to 48 to provide margin. If still failing, the issue is structural (input mechanism, not timing) and warrants a different approach.
…ture; V2.4.6 next State snapshot for next-session pickup. Captures: - V2.4.x progress: 6 fixes shipped (south-edge, multi-map classifier, PostBattle dismiss, frames bumps). - What's working: ROM decoder, phase classifier with mapId, advisor ASCII for indoors, overworld navigation. - Current blocker: ExitInterior fails when party spawns near map edge — 16x16 viewport doesn't include the south-edge exit at localY~14-15. - V2.4.6 plan: BFS the full 64x64 InteriorMap instead of the 16x16 viewport. Less invasive than refactoring Pathfinder interface — add InteriorMap.readFullMapView() returning a 64x64 ViewportMap, pass to pathfinder; advisor ASCII rendering stays at 16x16.
Wire 5 interior-scan trace events through InteriorExplorer + InteriorScanner
via a `(String) -> Unit` traceSink param (typealias InteriorTraceSink, defined
independently in both runtime/ and skills/ packages to avoid coupling).
Events emitted:
- interior_scan_triggered (explorer, with frame_detector mode)
- interior_scan_candidates (explorer)
- interior_scan_confirmed (scanner)
- interior_scan_rejected (scanner; covers pass2-rejected + invalid-coords)
- interior_scan_error (scanner)
AgentSession.discoverWeaponShop wires `trace.record` as the sink for both
explorer + scanner. Default `traceSink = {}` keeps existing tests/call sites
unaffected. New test asserts the 3 Found-path events fire.
Spec 4 §7 (interior_explore_outcome already wired by Task 11).
…oser cap Three empirically-driven fixes after 2026-05-08 cold-start run revealed: 1. Opus_4 → Opus_4_5 (claude-opus-4-0 returns 500; koog 0.6.1 max is claude-opus-4-5-20251101). ModelRouter + test updated. 2. GeminiVisionConsult.scanInteriorCandidates: stub → real Gemini Pro call. With KNES_VISION=gemini-pro the runtime vision is Gemini, but only Pass 2 had real impl — Pass 1 was the empty-list stub from Task 2, causing 3-of-3 empty scans → pass1-degraded bailout in walkSteps=2. Per user: "pixele najlepiej w gemini". 3. pass1-degraded cap 3 → 10. Pixel-hash fallback triggers a scan per walk step (1-byte FNV-1a delta), so 3 consecutive empty Pass 1s accumulate before party can explore. 10 lets explorer cover meaningful ground before bailout. Test counts unchanged (333 total, 3 baseline failures, 7 skipped).
Run 5 (commit ce9a23b) succeeded structurally — Gemini Pass 1+2 confirmed EXIT_TILE in Coneria mapId=8 — but bailed at walk-stuck-after-4-steps. WalkInteriorVision STUCK is common in Coneria narrow corridors; cap=3 fires before party can re-route. Bumping consecutiveStuck cap 3→8 + explorer capSteps 50→100 gives explorer room to navigate past initial stuck states. Test updated.
Empirical run 6 showed explorer used only $0.06 of $3 budget before walk-stuck cap=8 fired at step 9. Most budget burned in main loop LLM (Sonnet nav + Opus advisor) after explorer bailed, not in explorer. Bumping all caps radically lets explorer use more of its budget: - consecutiveStuck: 8 → 30 (LLM nav often stucks; tolerate longer) - consecutiveScanEmpty: 10 → 30 (Pass 1 empty in transit corridors OK) - capSteps: 100 → 200 (full Coneria interior coverage budget) Tests updated.
Empirical runs 5+6+7 showed 8-30 consecutive STUCK from Sonnet 4.6 navigator in Coneria narrow corridors, regardless of explorer cap. Per user feedback: Opus advisor should drive interior navigation. Bumping default model to claude-opus-4-5-20251101 (koog enum doesn't expose 4.7 yet). Cost goes up ~5x per nav call, but explorer was using only $0.05 of $3 budget — there's headroom to spend on better nav.
Spec 4: FF1 interior self-discovery landmark scan
Closes Spec 4 empirical gap (run 9 "WrongClass" failures): party stood in town overlay (mapId=8) when BuyAtShop fired, hence taps opened game menu not BUY menu. New skill issues hardcoded N+W tap pattern from Coneria spawn (~smPlayer(12,35)) toward weapon shop building entrance, detects mapId change as boundary signal, then walks N to face the shopkeeper. Updates the cached NPC_SHOPKEEPER landmark with the discovered sub-shop mapId so BuyAtShop's preflight matches. Wired into runOutfitBootPhase between cachedShop hit and per-char buy loop, gated on currentMapId==8 (skipped on warm-start where party already inside). Coneria-specific POC; generic interior nav is Spec 6+ territory.
…p timing Run 10 v1 hit west wall at smPlayer(7,30) without finding door. v2: - Walk N until smPlayerY stops decreasing (= south wall of building) - Sweep horizontal offsets ±1..±5 along wall, tap N at each - Detect mapId change after each tap (wall-hit-aligned, sweep-N-at-offset) - After entry, walk N×6 to face shopkeeper - Tap timing: pressFrames 6→12, gapFrames 6→8 (FF1 needs full hold for 1-tile move)
Run 12: party reached mapId=24 sub-shop at (12,14) but BuyAtShop WrongClass'd because keeper was further N. First Down cursor tap in BuyAtShop walked party south out of shop, dropping mapId back to 8 mid-purchase. v3 walks N until smPlayerY stops decreasing (keeper blocks), so party is reliably facing the keeper.
Run 13: party reached mapId=24 sub-shop at (12,18) but blocked by wall before keeper, BuyAtShop tap-A loop never opened BUY menu (12× WrongClass). After EnterShop succeeds, capture screenshot and run Pass 1 (Gemini) to locate shopkeeper sprite. Walk party so it stands at (sx, sy+1) — directly south of keeper, facing N. Final N-tap ensures facing remains correct after horizontal moves. Then BuyAtShop fires with correct positioning.
Run 14: Gemini Pass 1 saw count=0 in mapId=24, confirming we entered the wrong building. Reorder sweepOffsets to prefer ±3, ±4 first so we hit a different door. Also dump post-enter screenshot to /tmp when shopkeeper isn't visible — gives manual inspection point.
Run 15: reordered sweep failed to enter (party stranded at 9,18 mapId=8). Restore original sweep order (preferring small offsets first) so we reproduce mapId=24 entry. Always dump post-enter screenshot to disk so we can manually inspect which building we landed in.
Run 16 dump revealed mapId=24 was Coneria CASTLE entrance hall, not weapon shop. Walking N×16 from spawn passes through plaza into castle gate. Limit maxNorthTaps to 7 (mid-plaza) + sweep offsets ±8 to find shop doors which are below castle gate.
Hardcoded sweep landed in castle entrance (run 16). Replace with Opus 4.5 advisor that sees screenshot + Coneria layout context (per Mike's RPG Center map) and recommends single-step actions. Pipeline: 1. Loop up to 20 iters while mapId=8 2. Each iter: capture screenshot → Opus advise → execute one tap 3. Loop terminates on mapId change OR Done/Fail advice 4. Update cached weapon-shop landmark with discovered mapId Cost: ~$0.05-0.15 per Opus call × 20 iters worst-case = up to $3 per run. Within $3 cost-cap budget if advisor finds shop quickly.
Run 20 revealed: KNES_VISION=gemini-pro routes outfitVision to GeminiVisionConsult, which has only a Fail-stub for adviseShopApproach. Add a separate outfitAdvisor: HaikuConsult? defaulting to a fresh AnthropicHaikuConsult instance — its real impl uses Opus 4.5 directly. runOutfitBootPhase prefers outfitAdvisor if set, falls back to outfitVision otherwise.
Run 21: Opus reasoning was solid but recommended Up 20x straight. Party walked from (12,35) to (12,18), hit wall, advisor didn't notice. Track prevSx/prevSy across iters; when position unchanged, append 'BLOCKED — try Left/Right' to advisor context. Bumps maxAdvisorIters to 25 to give more search time.
Run 22: advisor switched Left after blocked but entered castle again. Two improvements: 1. SYSTEM_ADVISOR prompt now includes empirically-observed coordinate layout: castle gate at smPlayer(10-12, ~17), weapon shop X~8-9, armor shop X~5-7. Explicit warning to AVOID castle gate. 2. After mapId change, run Pass 1 scan to verify shopkeeper visible. If not (wrong building), tap Down ×15 to exit, continue advisor loop. Give up after 3 wrong-building entries.
…andoff 23 paid runs, ~$63 empirical validation. Spec 4 vision pipeline validated and merged (PR #119). Spec 5 multi-mapId nav implemented in two variants (hardcoded sweep + Opus 4.5 advisor) but neither reliably navigates Coneria plaza to weapon shop door — both land in castle gate (mapId=24) instead. Three forward paths documented: ROM pathfinder, FF1 disasm vendoring, or manually-recorded gameplay path.
…-iter action log
V3 architecture (pre-session, was uncommitted):
- RAM-triggered boot phase (mapId=0 && char1_str>0) replaces FfPhase trigger
- Pre-boot deterministic pressStartUntilOverworld
- Vision advisor loop in runOutfitBootPhase (Gemini 2.5 Pro thinking mode)
- Castle short-circuit when curMapId in {8, 24}
- SUB-GOAL HIERARCHY in AdvisorAgent.systemPrompt (T not C; outfit→grind→bridge→shrine)
- SYSTEM_ADVISOR rewritten: locate-party first, ban training-data game geography
- Removed v1 EnterConeriaWeaponShop (hardcoded sweep)
- Coneria8DoorDumpTest diagnostic for ROM-decoded mapId=8 layout
Empirical fixes (this session, run #3-7 evidence):
- readPos branches on mapflags bit 0, not mapId — town overlay (mapId=0+mf=1)
reads smPlayerX/Y instead of frozen worldX/Y. Fixes "all 4 cardinals
blocked" false-Fail seen in run #1, #3.
- walk_settle 30→120 frames so town transition + sprite render stabilizes
before advisor takes over (run #2 Fail "party not visible" was mid-render).
- 3-regime advisor context: OVERWORLD / TOWN_OVERLAY / SUB_MAP, with regime-
appropriate position semantics ("worldX/Y" vs "smPlayer x,y").
- Mandatory double-check retry when advisor bails with party-visibility doubt
(defensive; not exercised post-settle bump).
- V5.38 minimap: visited-tile set + per-tile blocked-cardinals, rendered as
13×13 ASCII grid in advisor context. Run #5→#6 evidence: with no minimap
Gemini circled INN for 40 iters; with minimap reached weapon shop in 22.
- Action log 6→30 + counter-intuitive detour prompt ("walk away from goal
to break out of pocket"). Run #6→#7 evidence: with 6 entries Gemini
oscillated (13,8)↔(13,9)↔(14,8) for 38 iters; with 30 reached weapon shop
door tile and opened the BUY/SELL/EXIT menu in run #7 iter 22-23.
Diagnostic dump: each boot-advisor iter persists screenshot to
/tmp/spec5-boot-iter-NN-midM-mfF-posPX_PY-smSX_SY-wWX_WY.png so the position
regime + RAM-vs-engine state can be cross-verified post-mortem.
Open architectural debt (next session): FF1 NES shops are NPC dialog overlays
in the town overlay (mapflags=1, mapId=0), NOT sub-maps. The runOutfitBootPhase
inSubShop check (`curMapId != 0`) is structurally wrong — Gemini visually
reached the weapon shop dialog in run #7 but Done was rejected because mapId
stayed 0. Shop entry detection needs to switch to vision (BUY/SELL/EXIT menu
recognition) or RAM (screenState/menuCursor); BuyAtShop precondition needs to
allow mapId=0 + mapflags=1.
Author
|
Misrouted to upstream — opening in fork repo instead. |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two layers of work on FF1 Coneria weapon-shop entry (Spec 5 boot phase):
V3 architecture (pre-session, was uncommitted): RAM-triggered boot phase, vision advisor loop driven by Gemini 2.5 Pro, castle short-circuit, sub-goal hierarchy in AdvisorAgent prompt. Removes the v1 EnterConeriaWeaponShop hardcoded sweep.
Empirical fixes (run #1-7 this session):
mapflags bit 0, notmapId. Town overlay state (mapId=0 + mf.0=1) readssmPlayerX/Yinstead of frozenworldX/Y. Fixes "all 4 cardinals blocked" false-Fail seen in run Initial Support for Kotlin Compilation #1 and Fixing NES Emulator implementation and Compose Rendering introduced #3.OVERWORLD/TOWN_OVERLAY/SUB_MAP) with regime-appropriate position semantics./tmp/spec5-boot-iter-NN-midM-mfF-posPX_PY-smSX_SY-wWX_WY.png) for post-mortem cross-verification.Empirical trajectory (7 runs)
Open architectural debt (next PR)
FF1 NES shops are NPC dialog overlays inside town overlay (mapflags=1, mapId=0), NOT sub-maps. The runOutfitBootPhase
inSubShopcheck (curMapId != 0) is structurally wrong — Gemini visually reached the weapon-shop BUY menu in run #7 but Done was rejected because mapId stayed 0. Next iteration needs:mapId=0 && mapflags=1.Test plan
./gradlew :knes-agent:compileKotlinclean