ClaudePlayer

An AI agent that plays Game Boy games autonomously using Claude's vision capabilities and the PyBoy emulator.

Overview

ClaudePlayer connects Claude to a Game Boy emulator. Each turn, Claude receives a screenshot of the current game frame, reasons about what to do, and sends button inputs back to the emulator. It maintains a structured memory system to track game progress (items, NPCs, locations, quests, stats).

Prerequisites

Python 3.10+
Pipenv
An Anthropic API key
A Game Boy ROM file (.gb)

Setup

# Install dependencies
pipenv install

# Create .env with your API key
echo "ANTHROPIC_API_KEY=your_key_here" > .env

# Place a Game Boy ROM in the project directory

Configuration

Edit config.json:

| Key | Description | Default | |-----|-------------|---------| | ROM_PATH | Path to Game Boy ROM | Required | | STATE_PATH | Saved emulator state file | null | | EMULATION_MODE | "turn_based" or "continuous" | "turn_based" | | MODEL_DEFAULTS.MODEL | Claude model to use | "claude-sonnet-4-5-20250929" | | MODEL_DEFAULTS.THINKING | Enable extended thinking | true | | RATE_LIMITS.RPM_THRESHOLD | Requests per minute limit | Configured in file | | RATE_LIMITS.TPM_THRESHOLD | Tokens per minute limit | Configured in file | | REDIS_LOGS | Optional Redis logging config | null |

Running

pipenv shell

# Default config
python play.py

# Custom config
python play.py --config my_config.json

# Create a saved emulator state
python emu_setup.py

Project Structure

| Path | Purpose | |------|---------| | play.py | Launcher script | | claude_player/main.py | CLI entry point with arg parsing | | claude_player/agent/game_agent.py | Main orchestrator: emulator init, game loop, coordination | | claude_player/interface/claude_interface.py | Claude API communication and rate limiting | | claude_player/state/game_state.py | Game state tracking (memory, goals, history) | | claude_player/tools/tool_setup.py | Tool definitions (send_inputs, memory ops, etc.) | | claude_player/tools/tool_registry.py | Tool registry system | | claude_player/config/config_loader.py | Config file parsing | | claude_player/config/config_class.py | Configuration data class | | claude_player/utils/game_utils.py | Button input parsing, screenshot capture | | claude_player/utils/memory_reader.py | Game memory/stats reader | | claude_player/agent/summary_generator.py | Periodic game progress summarization | | config.json | Runtime configuration |

Available Tools (for the agent during gameplay)

| Tool | Description | |------|-------------| | send_inputs | Send button sequences to the emulator | | set_game | Identify the current game | | set_current_goal | Update the gameplay objective | | add_to_memory | Store items, NPCs, locations, quests, mechanics, stats | | remove_from_memory | Remove a memory entry | | update_memory_item | Update an existing memory entry | | toggle_thinking | Toggle extended thinking mode on/off |

How It Works

PyBoy emulator loads the ROM and renders a frame
The frame is captured as a screenshot and sent to Claude
Claude analyzes the screen, reasons about the game state, and calls tools
send_inputs presses buttons on the emulator (A, B, Up, Down, Left, Right, Start, Select)
In turn-based mode, the emulator only advances when the agent acts
The agent maintains structured memory to track long-term game progress
Periodic summaries compress the conversation history to stay within context limits