Frame-synced input queue, MCP/REST game actions, and UI cleanup#7
Open
ArturSkowronski wants to merge 284 commits into
Open
Frame-synced input queue, MCP/REST game actions, and UI cleanup#7ArturSkowronski wants to merge 284 commits into
ArturSkowronski wants to merge 284 commits into
Conversation
@ArturSkowronski Emulator is no longer depending on any UI Class
Emulator moved to vnes-emulator module
AppletUI to remove dependency on AbstractNESUI and implement GUI directly
Remove BufferViewAdapter.java and AppletScreenView.java from AppletUI
Compiling Compose UI Without any reference to swing
Additional Logging and first Mario Frame
Add MCP server: control emulator from Claude as native tools
MCP tools now call the Compose UI's embedded REST API (localhost:6502) instead of running a standalone headless NES. This means: - The LLM controls the game through MCP tools - The user watches gameplay live in the Compose UI window - Same NES instance, zero duplication Flow: Claude → MCP (stdio) → REST API (:6502) → NES ← Compose UI Usage: 1. Launch Compose UI, click "API Server" 2. Configure MCP server in Claude Desktop/Code 3. Ask Claude to play the game
MCP server bridges to REST API for live visualization
Add MCP module tests (17 tests)
- Add 14 E2E tests (McpApiE2ETest): real Netty server + RestApiClient, same HTTP path as MCP production. 6 infra tests (no ROM), 5 SMB tests, 3 FF1 tests including full intro navigation. - Fix MCP server shutdown: add Job.join() so stdio transport stays alive - Enable /step and /reset in shared mode so MCP can control emulation while Compose UI renders live - Wire API controller into ComposeInputHandler so button presses from MCP reach the NES (keyboard + gamepad + API merged) - In shared mode, advanceFrames waits for UI-produced frames instead of stepping CPU directly (prevents race condition) - Add .mcp.json for Claude Code integration 464 tests, 0 failures.
MCP E2E tests and shared mode live gameplay
- NES screen canvas scales with window, maintains 256:240 aspect ratio - Remove Kotlin Dev Day logo from logo.png (keep VirtusLab, JVM Weekly)
Addresses short button presses not registering in shared mode due to race between API thread and NES joypad polling.
In shared mode, enqueues steps and awaits a latch with per-frame timeout instead of directly driving frames. Standalone mode keeps existing setButtons + advanceFrames behavior. Adds test for standalone press wiring.
Integrates InputQueue, FrameInput, and StepRequest from knes-api into NesEmulatorSession so standalone/headless MCP mode has the same queued input behavior as the shared Compose UI mode. Also moves knes-api from testImplementation to implementation in knes-mcp/build.gradle.
- Add synchronized lock to InputQueue.enqueue and advanceFrame - Use AtomicInteger for LatchEntry.remaining - Call controller.onFrameBoundary in EmulatorSession standalone advanceFrames - Rename misleading test
Screenshot flag on step, tap tool, sequence tool to reduce tool call round-trips by 67-92%.
Introduce the pluggable game actions system for knes-debug. GameAction defines automation scripts that play like a real NES player (read RAM + press buttons only). Includes static registry with register/get/list operations, ActionController interface for emulator interaction, and ActionResult data class. Tests cover registry CRUD, data classes, FF1 BattleFightAll logic, ActionRegistry loading, and mock controller execution.
BattleFightAll automates FF1 battle by having all alive characters use FIGHT until the battle screen exits. ActionRegistry provides lazy loading of game-specific actions keyed by profile ID, matching the existing GameProfile pattern.
- SessionActionController bridges ActionController to EmulatorSession
- GET /profiles/{id}/actions lists available actions
- POST /profiles/{id}/actions/{actionId} executes an action
- MCP tools: list_actions, execute_action
- ActionRegistry.ensureLoaded() on profile apply
Actions play like a real NES player: read RAM + press buttons only.
Walks randomly on overworld until a battle triggers, with stuck detection and auto-reversal when entering towns.
- gradle/wrapper-validation-action@v1 → gradle/actions/wrapper-validation@v4 - actions/setup-java@v3 → actions/setup-java@v4 Fixes wrapper validation failure on Gradle 9.4.1.
ArturSkowronski
changed the title
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three thematic groups of changes accumulated on
ui-cleanup:Frame-synchronized input
InputQueuedelivers input on frame boundaries instead of racing the emulator loop, eliminating dropped/duplicated presses./step) and standalone Compose UI paths so behavior is consistent across modes.onFrameBoundaryfollow-up.MCP / REST game actions
GameActioninterface,ActionController,ActionResult, andActionRegistryfor pluggable game-specific automation.BattleFightAll,WalkUntilEncounter./stepand/sequencegain screenshot capture; new/tapendpoint and MCPtap/sequencetools.UI + infra cleanup
Docs
Test plan
/stepand/sequencevia REST return screenshots when requestedtap/sequencework end-to-endBattleFightAllandWalkUntilEncountervia MCP / REST