What is the Bench Perf Oss skill?
Bench Perf Oss is a testing skill by NVIDIA for Claude Code, Cursor, Codex and other AI coding agents. Compare BioIR vs OSS folding performance: model.forward() latency, GPU use, lDDT and DockQ with matched inputs. Use for benchmark setup, execution or audits; excludes affinity heads and Ray throughput.
Input: model key + OSS source path. Output: WORKDIR harnesses, locked config, per-sample model.forward() latency + GPU artifacts, OpenStructure lDDT and DockQ vs benchmarks/dataset/groundtruth/, comparison report.
When to use it
Use for benchmark setup, execution or audits; excludes affinity heads and Ray throughput.
What the SKILL.md covers
- Mandatory gates
- Approximate options
- MSA and template contract
- Phases
- Phase 0 — WORKDIR, environments, GPU, OSS root
- Phase 1 — Manifest + locked config
- Phase 2 — Map the dataset into ossdata
- Phase 3 — Write the harnesses
- Phase 4 — Run BioIR
- Phase 5 — Run OSS e2e
- Author
- NVIDIA
- Pack
- NVIDIA/skills (30 skills)
- Category
- Testing, #48 of 313
- Alyn Score
- 78 / 99
- Files
- 56, includes scripts, includes references
- License
- Apache-2.0
- Last updated
- 4 h ago
- Works with
- Claude Code, Claude, Cursor, Codex, Gemini CLI, Copilot
Bench Perf Oss compared to similar skills
| Skill | Alyn Score | GitHub stars | Updated |
|---|---|---|---|
| Bench Perf Oss · NVIDIA | 78 | 4.1k | 4 h ago |
| Benchmark Models · Garry Tan | 84 | 142k | 19 min ago |
| Arize Experiment · GitHub | 86 | 41k | 6 h ago |
| Model Code And Result Generator · yushui2022 | 50 | 462 | 26 days ago |
Install Bench Perf Oss
Pick your agent. Every command is ready to paste.
npx alyn-cli add bench-perf-ossDetects your agents (Claude Code, Cursor, Codex and 70+ more) and installs to the right folder. Add -g to install globally.
npx skills add NVIDIA/skills --skill bench-perf-ossOr use the open skills CLI directly.
01
Install it
Run the command above, or upload the folder in Claude.
02
Ask naturally
Describe the task. Your agent loads bench-perf-oss when the request matches its description.
03
Name it if needed
If it does not trigger, say “use the bench-perf-oss skill”. In Codex, type $bench-perf-oss.
Benchmark BioIR vs OSS e2e model.forward
Input: model key + OSS source path. Output: WORKDIR harnesses, locked
config, per-sample model.forward() latency + GPU artifacts, OpenStructure
lDDT and DockQ vs benchmarks/dataset/ground_truth/,
comparison report.
This file defines scope, mandatory gates, and phase handoffs. Follow workflow.md for procedures; open references when their phase starts.
This is v1 for structure folding only (keys with a coordinate
head: Boltz-1/2, OpenFold2 / AlphaFold2, OpenFold3, protenix-v2).
The headline compared numbers are GPU-synced model.forward()
time and GPU memory. Parse, tokenize, featurize, H2D, postprocess,
write, and scoring are outside the timing window.
Both quality metrics are required: OpenStructure lDDT on every sample, and DockQ on supported protein-protein or protein-small-molecule complexes. Every row records DockQ applicability; DockQ 2.1.3 does not support RNA/DNA interfaces. Whole-structure lDDT and interface DockQ answer different questions. See measurement.md.
Do not run this playbook unchanged on a non-folding key
(boltz-2-affinity, or anything whose output is not a structure).
See Other model kinds.
Templates on both sides or neither. Retain template-bearing spec items and verify attachment on both featurized batches before timing; missing templates change both quality and workload. Follow the model profile and samples.md.
Serial, one sample per call. executor_backend=None. Feed
processor([one_record]) — never the whole manifest as one batch
(host-RAM OOM). Same one-at-a-time rule on the OSS side.
Choose a path from the support matrix
(docs/ref/support-matrix.md, Pipeline column):
- Processor path —
build_processor. The model has a factory (Boltz-1/2, OpenFold2 / AlphaFold2, OpenFold3,protenix-v2). BioIR uses serialbuild_processor; OSS uses its own e2e. Each side featurizes itself. - OSS feature path — OSS pipeline + BioIR module. A folding model with a
compute path but no pipeline factory. No current key uses it. Use the
OSS data pipeline for features, then replace the OSS
nn.Modulewith the BioIR module. See no-pipeline.md.boltz-2-affinityis not a folding bench — see Other model kinds.
Authorities:
- Phase procedures — workflow.md
- Production API —
docs/ref/api.md(build_processor,profile_inference,runtime_args) - Serial vs Ray —
docs/ref/architecture.md - Model keys and MSA rules —
docs/ref/support-matrix.md - Checkpoints —
docs/ref/model-weights.md - Dataset and sample catalog — samples.md (built by
rebuild_dataset.py) - Environment — environment.md
- Timing protocol — measurement.md
- MSA mapping (dataset -> OSS) — msa.md
- Template mapping (dataset -> OSS) — templates.md
- No BioIR pipeline — no-pipeline.md
- DeepSpeed evoformer (OF2 / OF3) — deepspeed-evoformer.md
- Model profiles — Boltz-2, OpenFold2 / AlphaFold2, OpenFold3, Protenix-v2
- Portable SKU package (optional) — bundle.md
Mandatory gates
If any gate cannot be satisfied, stop and report the blocker. Do not claim a complete bench.
- Both environments are probed and isolated first. Check BioIR and OSS imports, set up whatever is missing, and resolve package conflicts before any timed run. See environment.md. Do not install OSS pins into the BioIR interpreter.
- OSS revision is the pin. Use the exact revision selected by the
model profile; when the profile selects a checked-in submodule
gitlink, that gitlink is the pin. Boltz-1/2 =
v2.2.1. OpenFold2 = aqlaboratory/openfoldv2.2.0. Amaincheckout or a different tag is a hard failure. See environment.md. A recorded, reverted, input-side patch on top of the pin is allowed — only to make both sides honour the same inputs, never to change the compute path (templates.md). - No invented numbers. Every latency, GPU power, clock-rate, or speedup figure in the report cites an executed command, exit code, and artifact path. Parse tool output — never estimate.
- Dataset is the one
rebuild_dataset.pybuilds, only. The builder ships in this skill atdataset/; local build root:benchmarks/dataset/. Usespec_full.jsonorspec_monomer.json(samples.md). Do not substituteexamples/data/samples/, FASTA-only inputs, or a synthetic residue sweep unless the user explicitly replaces the dataset. - MSAs and templates are mandatory when they exist. For every
polymer that lists an unpaired or paired A3M (or Boltz CSV) or a
template in the spec, both BioIR and OSS must load those files.
Running without MSAs, with
use_msa_server, or with empty protein MSAs when files exist is a hard failure. - Templates are custom templates: all of them, both sides. The spec's templates are caller-supplied, so both sides featurize every one, limited only by a cap set identically on each side. Turn off the similarity, coverage, date, and keep/drop gates that exist to filter search output; patch a side if config cannot. Verify by counting populated template slots per side — a non-zero mask passes while three of four templates are missing. See MSA and template contract, msa.md, and templates.md.
- Same sample set on both sides. Write
$WORKDIR/ref_data/sample_manifest.jsonbefore any timed run. Both harnesses assert exact set equality against it. - The OSS input mapping is a script plus an index, not ad-hoc
files. When the OSS loader needs another layout or naming, build
$WORKDIR/oss_data/withbench/stage_oss_data.pyand recordoss_data/index.json; the harness reads the index. Symlink the dataset, never copy or edit it, and assert every declared input staged with one distinct key per chain (Phase 2). - Config parity is locked first. Write
$WORKDIR/ref_data/bench_config.jsonbefore Phase 4. Recycling, sampling, diffusion samples, precision, seed, and GPU id must match. - BioIR starts from its optimized pretrained config. Omit
config=and processorengine_kwargs["config"]unless applying documented model-profile overrides for input parity, fixed recycle counts, or approximate options. Derive those configs withget_pretrained_config, change only the documented fields, and record the delta. An eager arm additionally callsdisable_cuda_graphs(). Never hand-build aBaseConfig, copy OSS layer configs onto BioIR, or replace graph policies. See measurement.md. - Serial, one GPU, one sample per call.
executor_backend=None. PinCUDA_VISIBLE_DEVICESto one device. Loop the in-scope manifest and callprocessor([one_record])(or OSS featurize +forwardon that one batch). Neverprocessor(all_rows)— feature generation then holds every record in host RAM and OOM-kills before the engine. See measurement.md. - Every timed column runs on the same host and the same GPU. Run
BioIR and every OSS column on one physical machine and one device,
not just the same GPU model. Record the hostname, GPU UUID, and
driver version in
ref_data/bench_config.json. Latency from different hosts is not comparable, because CPU, driver, power limit, and thermals differ. If a column must be rerun elsewhere, rerun every column there. - Both scorers are installed before timing. Give
ostits own conda env andDockQits own venv, separate from BioIR, OSS and Miniforge base. Score written CIFs withost compare-structuresand supported interfaces withDockQ. Every row getsdockq_status: RNA/DNA interfaces useunsupported_chain_types; a monomer usesnull/single_chain, never0.0. See environment.md and environment.md. - OSS uses its recommended inference kernels, and the config has
to say so. Do not strip an optimized backend "for fairness", and
do not assume a shipped default is the recommendation; resolve the
model profile and upstream kernel documentation. If the selected
profile requires cuEquivariance, enable it in config—installing the
wheel is not enabling the kernel. If it requires DeepSpeed
Evoformer attention, build only that op from source
(
DS_BUILD_OPS=0 DS_BUILD_EVOFORMER_ATTN=1), non-editable and never-e, budgeting 5–10 min for the CUTLASS compile. See environment.md and deepspeed-evoformer.md. A fallback is a different column, and only with user approval. - Warmup 1, measure 1. One discarded forward (JIT / graph capture
/
torch.compile), then one timed forward. That single time is the headline. Do not average or take a median. - Five samples before the full sweep. After preparation, run
BioIR and OSS eager on five samples spanning the residue range,
run the fast synthetic direct-module compile probe, then run OSS
torch.compileon the same five when the compile retry ladder succeeds. Show the measured results and ask whether to proceed. Do not launch the full sweep without the user's explicit approval (five-sample smoke gate). - No hidden skips. A missing sample, missing MSA, OOM, or crash is
recorded. Do not
continuepast it silently. - Implementation notes are mandatory. Keep
$WORKDIR/implementation-notes.mdcurrent as work happens.
Approximate options
Some BioIR options are approximate: they change the sampled coordinates
instead of reproducing the exact path bit for bit. Frame-aware prediction
reuse, frame_reuse_period, is one; Protenix-v2, OpenFold3 P2 and Boltz-2
have it. Its pretrained default is "auto" for Protenix-v2 and OpenFold3 P2,
which reuses at the step counts they were evaluated with, and 0 (exact) for
Boltz-2. Rules:
- Default on for Protenix-v2 and OpenFold3 P2. Both profiles lock 200
diffusion steps, a count the option was evaluated with, so their BioIR arm
runs with
frame_reuse_period = 4. Set 4 explicitly rather than relying on"auto": a forced 4 raises where"auto"would quietly run exact. Build the config from the model's pretrained config (the profile shows how), change only that field, pass it as the engine config (engine_kwargs["config"]), and record the value inbench_config.json. Boltz-2 keeps the option off; add its approximate column only when the user asks (its profile shows how). - The exact arm sets
frame_reuse_period = 0. Build it the same way. The pretrained Protenix-v2 and OpenFold3 P2 configs reuse, so leaving them unchanged does not give an exact column. The harness written in Phase 3 takes an--exactswitch that sets 0. - Run the exact arm as well. Add one BioIR column with the option off (same seed, same inputs, same GPU), time it like every other column, and report the approximate column's speedup against it and against OSS.
- Judge fidelity on paired scores, not equality. Score both BioIR columns with the same lDDT and DockQ scorers and report the paired per-sample difference (approximate minus exact) next to each speedup. Bit equality is required only of the exact column and of the option when off.
- Label the column. Name the column
bioir_approx_<option>and list the option and value in the table caption. Never present the approximate column as the exact BioIR result.
MSA and template contract
Mapping the dataset's alignments and templates onto an OSS tree is its own job, with its own silent failure modes. Full procedure, per-tree mappings, and verification: msa.md and templates.md. The gates below are the short form.
Protein unpaired MSA is required for Boltz-1/2 and OpenFold3, and
for every AlphaFold2 / OpenFold2 key (docs/ref/support-matrix.md)
when the spec lists A3Ms. Paired MSA is used when the spec lists
paired_msas. RNA, DNA, and ligand chains carry empty msas —
that is correct; do not invent files.
BioIR load (processor path). Convert each in-scope spec item to an
InputRequest with paths resolved against
benchmarks/dataset/
(samples.md). Do not call
load_requests on spec_*.json. Attach every listed A3M and
template.
OSS load (both paths). Map the same spec item to the OSS
inference script. Boltz: boltz_yaml + boltz_msa_csv, rewriting
stale absolute msa: paths to $DATASET_ROOT/casp15/msa/. Other
OSS trees: spec polymers + resolved A3Ms. Do not call
build_processor on the OSS feature path.
After load, before any forward, assert all of:
- Every spec
msas/paired_msas(and Boltz CSV) path exists on disk under$DATASET_ROOT. - Every protein polymer whose spec lists MSAs has a non-empty
MSARecordlist on the BioIRInputRequest(processor path) and non-empty alignments on the OSS featurizer input. - Every A3M the spec lists for that item is attached (unpaired and paired). A listed file that is skipped is a hard failure.
- The OSS adapter consumes those same alignments. No ColabFold / MSA server. No FASTA-only rewrite that drops alignments.
- Every template the spec lists is attached on both sides:
populated template slots equal
min(n_supplied, cap)on each, with the same entry and chain per slot (templates.md).
Phases
Run the phases in order. Each names what to read, what it must leave in
$WORKDIR, and the check that ends it; do not start a phase before the
previous exit check passes. Harnesses live in $WORKDIR; source changes are
limited to the recorded input-side patches allowed by the mandatory gates.
Phase 0 — WORKDIR, environments, GPU, OSS root
- Read:
workflow.md,
environment.md, and the model profile under
models/. OpenFold2 / OpenFold3 also need deepspeed-evoformer.md. - Produce: the
$WORKDIRtree,$OSS_ROOTat the pin, the BioIR and OSS interpreters,envs/ostandenvs/dockq,ref_data/gpu_inventory.csv, and the environment fields plusfeature_path(processor|oss) inref_data/bench_config.json. - Exit: both sides import on the same GPU, and the checkpoint strict-loads once in each interpreter from a recorded local path with the hub untouched (weights preflight). A non-folding key stops here.
Phase 1 — Manifest + locked config
- Read: workflow.md, samples.md, measurement.md, and the profile's runtime lock.
- Produce:
ref_data/sample_manifest.jsonwith every spec item, the in-scope filter by model capability only, and every A3M and template resolved;ref_data/bench_config.jsonwith the locked runtime args, seed, precision, GPU, dataset digests, BioIR default constructor and graph activation, OSS entry script, and compile plan. - Exit: the OSS script's expanded effective config matches the lock. The config is read-only from here.
Phase 2 — Map the dataset into oss_data
- Read: workflow.md, msa.md, templates.md, and the profile's mapping section.
- Produce:
bench/stage_oss_data.py, anoss_data/tree of symlinks, andoss_data/index.jsonper sample and chain. - Exit: a rerun is idempotent and silent, every declared input is staged with one distinct key per chain, and one monomer and one multi-chain sample are checked by hand.
Phase 3 — Write the harnesses
- Read: workflow.md and
measurement.md. Copy the shared helpers into
bench/: timing.py, gpu_telemetry.py, scoring.py, check_results.py, summarize_speedup.py, and plot_latency.py. Do not rewrite them. The OSS feature path also needs no-pipeline.md. - Produce: on the processor path,
bench/run_bioir.pyandbench/run_oss.py; on the OSS feature path,bench/dump_oss_features.pyplus both runners. Every loop shows a progress bar outside the timing window. - Exit: the five-sample smoke gate passes, its results are shown, and the user explicitly approves the full sweep.
Phase 4 — Run BioIR
- Read: workflow.md.
- Produce:
results/bioir_forward.jsonand logs underdebug/. The OSS feature path writes the feature dumps first. Run with$BIOIR_PYTHONandCUTEDSL_FORCE_CUBIN=1.
Phase 5 — Run OSS e2e
- Read: workflow.md.
- Produce:
results/oss_eager.jsonalways, andresults/oss_compile.jsononly after a passing probe or retry. Same manifest, locked knobs, and GPU; the OSS feature path loads the Phase 4 dumps. Run with$OSS_PYTHON. An OSS eager kernel crash blocks that backend; never relabel a fallback asoss_e2e.
Phase 6 — Compare and report
- Read: workflow.md and measurement.md.
- Produce:
results/speedup.jsonand both charts, then print setup, dataset, the latency table, speedup geomean and median overall and per residue bin, lDDT, DockQ, GPU power and clocks from in-forward samples, commands, and the path toimplementation-notes.md. Show the chart PNGs inline. - Exit:
harness/check_results.pypasses on every result file, every number cites an artifact, and latency is in seconds to three decimals.
Optional — compact a portable bioir-perf package
Skip unless the user asks to pack completed benches so they can run on another GPU SKU or in a fresh container. Follow bundle.md.
The pack is bioir-perf/, never release/, with one directory per model.
Completion requires a GPU smoke of every included model's BioIR, OSS eager,
and supported compile arms (Protenix never compiles), plus speedup.json,
comparison_report.md, and per-column JSON. This smoke checks harnesses;
a full-dataset SKU sweep is separate. Supported keys: boltz2, of3,
protenix, of2.
Boltz-2 — MSA deletion bug
The pinned OSS v2.2.1 featurizer zeroes non-query MSA deletions, and BioIR
does not. A large BioIR vs OSS lDDT gap on MSA-bearing proteins may be that,
not the engine. Never zero BioIR deletions to match; triage it with
models/boltz2.md.
Other model kinds
v1 is a folding bench. If the user names a non-folding key
(boltz-2-affinity, embeddings, design), stop and extend this skill (or a
sibling) before writing harnesses. What carries over and what each new kind
must redefine is in
workflow.md.
Out of scope (v1)
- Nsight Systems / Nsight Compute, MFU, SOL%, kernel breakdown
- Ray replica throughput (large-dataset path; not this spec)
- Template search (HHsearch / HMMsearch); templates are attached from the spec's shipped structures only (see samples.md)
- The demo tree
examples/data/samples/as a substitute dataset - DockQ / feature-tensor equivalence (
make-data-pipeline) - Isolated same-features microbench of a single stack (Benchmark the Swap
in
docs/advanced/accelerate-custom-model.md) — the OSS feature path is e2emodel.forwardon OSS features, not a layer bench - Non-folding heads (
boltz-2-affinity, embeddings, design, …) until this skill (or a sibling) is extended
Those are separate skills or a later revision.
Key gotchas
- Launch each harness with its own python —
$BIOIR_PYTHONor$OSS_PYTHON, never a barepythonafter Phase 0. CUTEDSL_FORCE_CUBIN=1on every BioIR process. JIT compile time makes forward latency unstable.- No
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, ever — it can slowforward(measurement.md). - CUDA 12 packages → CUDA 13. Install the
cu13counterpart of anycu12pin (environment.md). - Protenix-v2 is Processor path. Never pass
return_full_data=False;build_processorrejects it. Compare quality over seeds or the sample set because OSS adds Monte Carlo pair dropout that BioIR omits. - Seed goes on the feature-generator stage, not the tokenizer
(
docs/ref/api.md). - Paired A3Ms are per chain with equal row counts (
docs/ref/api.md). - Let DockQ choose the chain mapping and record it.
- Audit graph execution on every forward. Follow the counters and
cache rules in measurement.md; the default
model captures graphs without an
optimize()call. - Sharing the GPU with another job invalidates the run.
Tips for Bench Perf Oss
- • Start a fresh session after installing so your agent picks up the new skill.
- • This skill ships scripts. Read them before the first run, and run it in a project without secrets first.
- • It bundles reference files. The agent reads them only when needed, so they cost no context until then.
- • It is part of a pack with 30 skills. Install only the ones you need with --skill to keep triggers precise.
- • If two skills overlap, mention the one you want by name.
Questions about Bench Perf Oss
What does the Bench Perf Oss skill do?
Compare BioIR vs OSS folding performance: model.forward() latency, GPU use, lDDT and DockQ with matched inputs. Use for benchmark setup, execution or audits; excludes affinity heads and Ray throughput.
How do I install Bench Perf Oss?
Run "npx alyn-cli add bench-perf-oss" in your project. The Alyn CLI installs it for Claude Code, Cursor, Codex and 70+ other agents. In Claude on the web, upload the skill folder as a ZIP under Customize > Skills.
Does Bench Perf Oss work in Cursor and Codex?
Yes. Bench Perf Oss uses the open SKILL.md format, so it works in Claude Code, Claude, Cursor, OpenAI Codex, Gemini CLI, GitHub Copilot and other agents that support agent skills.
When should I use the Bench Perf Oss skill?
Use for benchmark setup, execution or audits; excludes affinity heads and Ray throughput.
What is inside the Bench Perf Oss skill?
The SKILL.md covers Mandatory gates, Approximate options, MSA and template contract, Phases, Phase 0 — WORKDIR, environments, GPU, OSS root, Phase 1 — Manifest + locked config, and the skill folder has 56 files including scripts.
What is a good alternative to Bench Perf Oss?
Benchmark Models by Garry Tan is the highest-ranked similar skill from another creator, with an Alyn Score of 84 compared to 78 for Bench Perf Oss.
Is Bench Perf Oss free?
Bench Perf Oss is open source on GitHub under the Apache-2.0 license. It is free to install; check the license before commercial use.
Who made Bench Perf Oss?
Bench Perf Oss is published by NVIDIA in the NVIDIA/skills repository, together with 29 other skills.
