<link rel="stylesheet" href="styles.f3b1fba60ec7970c.css">

Replication Data and Code for: Scale-Free Attention Dynamics in Transformer-Based Language Models

Loading...
Thumbnail Image

Date

Journal Title

Journal ISSN

Volume Title

Publisher

Zenodo

DOI

Abstract

Description

&lt;p class=""&gt;This repository contains the complete experimental framework and raw data results required to replicate the findings in the manuscript "Scale-Free Attention Dynamics in Transformer-Based Language Models."&lt;/p&gt; &lt;p class=""&gt;&lt;strong&gt;Contents:&lt;/strong&gt;&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;strong&gt;Code:&lt;/strong&gt;&nbsp;Jupyter notebooks for extracting attention weight matrices from multiple transformer architectures (Llama-3, GPT-2, BERT, T5, Pythia) and conducting rigorous Clauset-Shalizi-Newman (CSN) power-law fitting. It includes control experiments (Zipfian vs. Uniform training) and ablation studies (attention sink masking).&lt;/li&gt; &lt;li&gt;&lt;strong&gt;Results:&lt;/strong&gt; Raw CSV outputs containing the calculated power-law scaling exponents (alpha), Kolmogorov-Smirnov statistics, and Vuong likelihood ratio test verdicts across all six experiments.&lt;/li&gt; &lt;/ul&gt;

Keywords

Citation

Collections

Endorsement

Review

Supplemented By

Referenced By

Rights and licensing

info:eu-repo/semantics/openAccess
Creative Commons Attribution 4.0 International
https://creativecommons.org/licenses/by/4.0/legalcode