<link rel="stylesheet" href="styles.f3b1fba60ec7970c.css">

tamanaganti/abtp: Reproducibility fixes and Kendall τ verification

Loading...
Thumbnail Image

Date

Journal Title

Journal ISSN

Volume Title

Publisher

Zenodo

DOI

Abstract

Description

&lt;h2&gt;Changes from v1.0&lt;/h2&gt; &lt;h3&gt;Fixed&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;The §6 verification snippet in &lt;code&gt;REPRODUCE.md&lt;/code&gt; now takes &lt;strong&gt;&lt;code&gt;bri_imagenet_c&lt;/code&gt; from &lt;code&gt;generalization_results.json&lt;/code&gt;&lt;/strong&gt;, the same artifact that holds all 15 models' shift metrics. Previously it merged BRI from &lt;code&gt;imagenet_c_summary.json&lt;/code&gt; while reading outcomes from the generalization file, which is easy to get out of sync and is unnecessary for a headline check. &lt;strong&gt;Published correlation values are unchanged&lt;/strong&gt;—only the verification script was corrected.&lt;/li&gt; &lt;/ul&gt; &lt;h3&gt;Added&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;&lt;strong&gt;Kendall τ&lt;/strong&gt; alongside Spearman in the correlation snippet, with expected values aligned to &lt;code&gt;scipy.stats.kendalltau&lt;/code&gt; on the bundled JSON (e.g. ImageNet-R τ = −0.390, p = 0.046; ImageNet-A τ = +0.448, p = 0.021; NINCO τ = +0.619, p ≈ 0.001). Signs and significance match Spearman.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Pre-flight&lt;/strong&gt; Python snippet to confirm &lt;code&gt;bri_imagenet_c&lt;/code&gt; is non-null for every model record before correlating.&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Instructions&lt;/strong&gt; for re-running &lt;code&gt;10_generalization_eval_all15.py&lt;/code&gt; without stale &lt;code&gt;*_generalization.json&lt;/code&gt; cache issues (run Script 04 first; clear caches when regenerating).&lt;/li&gt; &lt;/ul&gt; &lt;h3&gt;Unchanged&lt;/h3&gt; &lt;ul&gt; &lt;li&gt;Application code under &lt;code&gt;src/&lt;/code&gt; and &lt;code&gt;scripts/&lt;/code&gt;&lt;/li&gt; &lt;li&gt;Committed result files and figures&lt;/li&gt; &lt;li&gt;Reported statistics in the paper&lt;/li&gt; &lt;/ul&gt;

Keywords

Citation

Collections

Endorsement

Review

Supplemented By

Referenced By

Rights and licensing

info:eu-repo/semantics/openAccess
MIT License
https://opensource.org/licenses/MIT