Aaai Artifact Evaluation skill

AAAI Artifact Evaluation

Use this to prepare artifacts that reviewers can use to assess reproducibility. AAAI supplementary material is part of the submission record; after review starts, do not assume it can be updated.

Artifact package

• Provide a technical appendix for proofs, algorithms, assumptions, hyperparameters, and extended experiments.
• Provide code/data ZIPs that reproduce main tables or figures, with a short README, environment, commands, seeds, expected outputs, and runtime.
• Provide multimedia appendices only when they support the technical claim.
• Remove author names, usernames, paths, repository history, cloud buckets, API keys, and metadata.
• Avoid web pointers in the reviewed submission unless current rules explicitly allow them.
• Include licensing and access notes for datasets, models, and third-party code.

AAAI-specific discipline

• Treat the supplementary deadline as final.
• Verify ZIP integrity before submission; missing or corrupted files may not be fixable during rebuttal.
• Make the reproducibility checklist consistent with the artifact package.
• Prepare a post-acceptance public release path but keep review artifacts anonymous.

What an AAAI reviewer actually opens

AAAI does not run a separate badged artifact-evaluation committee the way some systems venues do; the same broad-AI reviewer who scores the paper also inspects whatever supplement you attach. That reviewer may be a planning, knowledge-representation, or constraint-satisfaction specialist rather than a deep-learning engineer, so the artifact has to be legible without insider tooling. Optimize for a reviewer who skims, not one who will spend an afternoon configuring a cluster.

Reviewer action	Passes	Fails
Opens the ZIP	sane tree, top README	nested archives, 0-byte files
Reads appendix	maps to numbered claims	contradicts the paper
Tries one command	reproduces one headline number	needs private data or credentials
Scans for identity	nothing reveals authors	Git logs or home paths leak

Phase-1 artifact red flags

Because clearly-below-bar papers can be cut before author feedback, a supplement that looks thin or unrunnable is a cheap reason to summary-reject. Avoid these:

• Checklist promises released code, but the ZIP only holds figures and no scripts.
• A "see our repository" pointer to a mutable, deanonymizing URL.
• Multimedia attached for spectacle that carries no technical claim, inflating size with no rigor.
• Datasets shipped with no license note, leaving reuse legality unverifiable.

Worked vignette

A constraint-solving paper claims a 30% node-expansion reduction. The team ships a large ZIP of raw solver logs but no driver script. The reproduction path is empty, so artifact status is "risky"; the fix is a small run_main.py that regenerates Table 2 from seeds, a trimmed log sample, and a license for the benchmark instances. The raw dump moves to the post-acceptance release.

Output format

[Artifact status] complete / partial / risky / unavailable
[Submitted files] technical appendix / multimedia appendix / code-data ZIP
[Reviewer reproduction path] <commands and expected output>
[Anonymity risks] <metadata, links, paths, logs>
[Missing items] <data, code, seeds, licenses, hardware>