Four evaluation engineering lessons learned from building agent plugins

Published 24 October 2026

Evaluation should grade an agent's execution steps, not only its final summaries. An LLM judge incorrectly failed three of 11 prompts that the plugin had already passed.