Replication Data and Code for: Scale-Free Attention Dynamics in Transformer-Based Language Models
Abstract
Description
<p class="">This repository contains the complete experimental framework and raw data results required to replicate the findings in the manuscript "Scale-Free Attention Dynamics in Transformer-Based Language Models."</p>
<p class=""><strong>Contents:</strong></p>
<ul>
<li><strong>Code:</strong> Jupyter notebooks for extracting attention weight matrices from multiple transformer architectures (Llama-3, GPT-2, BERT, T5, Pythia) and conducting rigorous Clauset-Shalizi-Newman (CSN) power-law fitting. It includes control experiments (Zipfian vs. Uniform training) and ablation studies (attention sink masking).</li>
<li><strong>Results:</strong> Raw CSV outputs containing the calculated power-law scaling exponents (alpha), Kolmogorov-Smirnov statistics, and Vuong likelihood ratio test verdicts across all six experiments.</li>
</ul>
Keywords
Citation
Collections
Endorsement
Review
Supplemented By
Referenced By
Rights and licensing
info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode