Abstract

Reproducibility and reuse of scientific artifacts are critical for scientific progress. Artifacts that are easy to reuse help researchers compare results and build on prior work. Security conferences promote reproducibility through artifact evaluation and badging. The highest-level badge–results-reproduced–indicates that independent evaluators were able to use an artifact to reproduce key claims in the corresponding paper.
We perform a small-scale study to quantify the ease of reuse of security artifacts that received results-reproduced badges. We evaluate 97 artifacts from two conferences–PETS and ACSAC–from a recent year and from 2020. Our study finds that, despite passing stringent evaluation, many artifacts exhibit usability issues. Around 12% are not findable or lack key components. Among those that are complete, 27% require resources that may not be easily obtained, and 22% fail to run within an hour, leaving only 43 artifacts (44%) successfully reused. Artifacts are packaged using a variety of frameworks, such as Docker/VM images, Python environments, Rust, Go, Popper, Poetry, and Bazel; 38% of artifacts use more than one approach. Our findings highlight the need for community-wide standards for artifact packaging and evaluation to improve artifact findability and ease of reuse.

Date

2026

Authors

Jelena Mirkovic, Qishen Sam Liang, David Balenson