model-watch

Problem → Solution

The problem: AI companies silently degrade their models. "Opus 4.7 was hallucinating a lot today... shocking to see such degradation" — r/ClaudeAI (49↑). "Anthropic admits to have made hosted models more stupid" — r/LocalLLaMA (281↑). You're paying the same price for a dumber model and you don't even know it.

The solution: Standardized benchmark suite you run yourself. 7 tests across 5 categories. Scores stored locally. Alerts when recent scores drop >10% vs your historical average. Hard data, not vibes.

Quick Start

pip install git+https://github.com/minirr890112-byte/model-watch.git

model-watch demo              # View benchmark questions
model-watch submit '{"reasoning_1":"...","coding_1":"...",...}'  # Submit outputs
model-watch history           # View score history
model-watch alert             # Check for degradation

Benchmarks (7 tests)

| Category | Tests | What it measures | |----------|-------|-----------------| | Reasoning | 2 | Logic, multi-step deduction | | Coding | 2 | Code generation, debugging | | Writing | 1 | Quality, coherence | | Instruction-following | 1 | Precision, constraint adherence | | Hallucination detection | 2 | Factual accuracy |

How It Works

1. Run the 7 benchmark questions through your AI model of choice
2. Feed the responses into model-watch submit
3. Scores are stored locally in ~/.hermes/model-watch-history.json
4. Track trends with model-watch history
5. model-watch alert flags when recent scores drop >10% vs historical average

---

⭐ Star this repo if you've noticed your favorite model getting dumber: github.com/minirr890112-byte/model-watch