ScreenVision Skill

Local Client/Server skill: FastAPI + OpenVINO VLM + ChromaDB on http://127.0.0.1:8000.

Agent workflow

1. Check health: GET http://127.0.0.1:8000/health
   • status=running → ready
   • connection failure / not running → run setup.bat in this skill folder
   • status=initializing → wait and retry (first model export can take 5–20 min)
   • status=error → report message to the user
2. Remember current screen → run one-shot capture (preferred):

   python capture_once.py
   

   Or upload an image: POST /capture_and_index with form field file.
3. Search past screens:

   curl "http://127.0.0.1:8000/search_memory?query=python%20error"
   

APIs

POST /capture_and_index

Upload screenshot (multipart/form-data, field file). Returns summary + memory_id.

GET /search_memory?query=...

Returns top matching memory summaries.

GET /health

running | initializing | error, plus device when ready.

Setup

Windows one-click:

setup.bat

The script will:

1. Install real Python 3.11 if missing (skips Windows Store alias)
2. Install deps from requirements.txt
3. Start server.py
4. Poll /health until the model is ready (or fail with a clear error)

Optional hotkey client (Ctrl+Shift+M): python auto_capture.py
Optional silent autostart: start_hidden.vbs (uses .python_path written by setup)

Files

screenvision/
├── SKILL.md
├── setup.bat
├── wait_ready.ps1
├── server.py
├── capture_once.py      # Agent/CLI one-shot capture
├── auto_capture.py      # Hotkey listener client
├── start_hidden.vbs
├── requirements.txt
├── skill.json
├── screen_memory_db/    # created at runtime
└── ov_model_cache/      # created after first model export

Notes

• Device order: NPU → GPU → CPU automatic fallback.
• First run caches INT4 OpenVINO model under ov_model_cache/ for faster restarts.
• Needs ~30GB free disk and strong network on first model download.