tamanaganti/abtp: Reproducibility fixes and Kendall τ verification
Abstract
Description
<h2>Changes from v1.0</h2>
<h3>Fixed</h3>
<ul>
<li>The §6 verification snippet in <code>REPRODUCE.md</code> now takes <strong><code>bri_imagenet_c</code> from <code>generalization_results.json</code></strong>, the same artifact that holds all 15 models' shift metrics. Previously it merged BRI from <code>imagenet_c_summary.json</code> while reading outcomes from the generalization file, which is easy to get out of sync and is unnecessary for a headline check. <strong>Published correlation values are unchanged</strong>—only the verification script was corrected.</li>
</ul>
<h3>Added</h3>
<ul>
<li><strong>Kendall τ</strong> alongside Spearman in the correlation snippet, with expected values aligned to <code>scipy.stats.kendalltau</code> on the bundled JSON (e.g. ImageNet-R τ = −0.390, p = 0.046; ImageNet-A τ = +0.448, p = 0.021; NINCO τ = +0.619, p ≈ 0.001). Signs and significance match Spearman.</li>
<li><strong>Pre-flight</strong> Python snippet to confirm <code>bri_imagenet_c</code> is non-null for every model record before correlating.</li>
<li><strong>Instructions</strong> for re-running <code>10_generalization_eval_all15.py</code> without stale <code>*_generalization.json</code> cache issues (run Script 04 first; clear caches when regenerating).</li>
</ul>
<h3>Unchanged</h3>
<ul>
<li>Application code under <code>src/</code> and <code>scripts/</code></li>
<li>Committed result files and figures</li>
<li>Reported statistics in the paper</li>
</ul>
Keywords
Citation
Collections
Endorsement
Review
Supplemented By
Referenced By
Rights and licensing
info:eu-repo/semantics/openAccess
MIT License
https://opensource.org/licenses/MIT
MIT License
https://opensource.org/licenses/MIT