Experiment Code skill

Experiment Code

Generate and iteratively improve ML experiment code for research papers.

Input

• $0 — Task: generate, improve, debug, plot
• $1 — Research plan, idea description, or error message

References

• Experiment prompts and patterns: ~/.claude/skills/experiment-code/references/experiment-prompts.md
• Code patterns (error handling, repair, hill-climbing): ~/.claude/skills/experiment-code/references/code-patterns.md

Action: generate

Generate initial experiment code following this structure:

1. Plan experiments first — List all runs needed (hyperparameter sweeps, ablations, baselines)
2. Write self-contained code — All code in project directory, no external imports from reference repos
3. Include proper logging — Save results to JSON, print intermediate metrics
4. Generate figures — At minimum Figure_1.png and Figure_2.png

Mandatory Structure

project/
├── experiment.py      # Main experiment script
├── plot.py            # Visualization script
├── notes.txt          # Experiment descriptions and results
├── run_1/             # Results from run 1
│   └── final_info.json
├── run_2/
└── ...

Constraints

• No placeholder code (pass, ..., raise NotImplementedError)
• Must use actual datasets (not toy data unless explicitly requested)
• PyTorch or scikit-learn preferred (no TensorFlow/Keras)
• Each run uses: python experiment.py --out_dir=run_i

Action: improve

Improve existing experiment code:

1. Read current code and results
2. Reflect on what worked and what didn't
3. Apply targeted edits (prefer small edits over full rewrites)
4. Re-run and compare scores
5. Keep the best-performing code variant

Action: debug

Fix experiment code errors:

1. Read the error message (truncate to last 1500 chars if very long)
2. Identify the root cause
3. Apply minimal fix
4. Up to 4 retry attempts before changing approach

Action: plot

Generate publication-quality plots from experiment results:

1. Read all run_*/final_info.json files
2. Generate comparison plots with proper labels
3. Use the figure-generation skill for styling

Rules

• Always plan experiments before writing code
• After each run, document results in notes.txt
• Include print statements explaining what results show
• Method MUST not get 0% accuracy — verify accuracy calculations
• Use seeds for reproducibility
• Before each experiment include a print statement explaining exactly what the results are meant to show

Related Skills

• Upstream: experiment-design, algorithm-design
• Downstream: data-analysis, backward-traceability
• See also: code-debugging, paper-to-code