ai-evals-course
Install Write Code Eval
Pick your agent. Every command is ready to paste.
npx skills add ai-evals-course/evals-skills --skill write-code-evalThe open skills CLI detects your agents (Claude Code, Cursor, Codex and 70+ more) and installs to the right folder. Add -g to install globally.
01
Install it
Run the command above, or upload the folder in Claude.
02
Ask naturally
Describe the task. Your agent loads write-code-eval when the request matches its description.
03
Name it if needed
If it does not trigger, say “use the write-code-eval skill”. In Codex, type $write-code-eval.
Write a code evaluator
Start with a failure mode found through error analysis. Write one check for that failure mode, much like a unit test that asserts what should hold for each trace.
- State the rule and identify the trace fields or reference data the check needs. If the rule requires interpretation, use
write-judge-prompt. - Implement the check in the project's language and eval framework. Return a result and a reason in the format that framework expects.
- Test known passes and failures, including borderline cases. Run the check on available traces and inspect mistakes. If the rule uses a proxy for human judgment, compare its results with human labels.
Examples
| Failure mode | Possible check |
|---|---|
| Invalid output structure | Parse the output and check required fields |
| Missing or forbidden text | Match a string or pattern |
| Citation not in retrieved documents | Compare cited IDs with retrieved IDs |
| Bad tool call | Check arguments against the tool schema or run the call in a safe test environment |
| Wrong value | Compare the output with a reference value |
Choose the check from the failure rule. For a failure with both objective and interpretive parts, check the objective part with code and use a judge for the rest.
Tips for Write Code Eval
- • Start a fresh session after installing so your agent picks up the new skill.
- • It is part of a pack with 9 skills. Install only the ones you need with --skill to keep triggers precise.
- • If two skills overlap, mention the one you want by name.
Questions about Write Code Eval
What does the Write Code Eval skill do?
Write code evaluators for known failure modes with objective rules. Use when code can check the rule from a trace, with or without a reference answer. Use `write-judge-prompt` when the rule requires interpretation.
How do I install Write Code Eval?
Run "npx skills add ai-evals-course/evals-skills --skill write-code-eval" in your project. The skills CLI installs it for Claude Code, Cursor, Codex and 70+ other agents. In Claude on the web, upload the skill folder as a ZIP under Customize > Skills.
Does Write Code Eval work in Cursor and Codex?
Yes. Write Code Eval uses the open SKILL.md format, so it works in Claude Code, Claude, Cursor, OpenAI Codex, Gemini CLI, GitHub Copilot and other agents that support agent skills.
Is Write Code Eval free?
Write Code Eval is open source on GitHub under the Apache-2.0 license. It is free to install; check the license before commercial use.
Who made Write Code Eval?
Write Code Eval is published by ai-evals-course in the ai-evals-course/evals-skills repository, together with 8 other skills.
