hkalbertkim/KORA: KORA v0.1.1-alpha
Abstract
Description
<h1>KORA v0.1.1-alpha</h1>
<h2>Summary</h2>
<p>KORA v0.1.1-alpha adds CI-backed validation and the first controlled benchmark skeleton. In a deterministic-heavy benchmark skeleton, KORA-controlled execution avoided 16 of 20 simulated model invocations versus a naive direct baseline.</p>
<p>This is an alpha maintenance / benchmark skeleton release. It keeps the v0.1.0-alpha terminal-first surface intact while adding a reproducible CI and benchmark path for future technical preview work.</p>
<h2>What Changed Since v0.1.0-alpha</h2>
<ul>
<li>Added GitHub Actions CI for editable install, release smoke, and pytest validation.</li>
<li>Added the <code>experiments/</code> benchmark experiment directory.</li>
<li>Added the deterministic-heavy workload draft at <code>experiments/workloads/deterministic_heavy_v0.json</code>.</li>
<li>Added dry-run benchmark mode for workload validation and task/category counting.</li>
<li>Added direct-baseline benchmark mode to simulate one model invocation per task.</li>
<li>Added KORA-controlled benchmark mode to simulate deterministic-first execution from workload metadata.</li>
<li>Added the 2026-05-04 progress report.</li>
<li>Added the v0.1.1-alpha release note.</li>
</ul>
<h2>CI-Backed Validation</h2>
<p>GitHub Actions now validates changes to <code>main</code> with:</p>
<pre><code class="language-bash">python3 -m pip install -e ".[dev]"
./scripts/release_smoke.sh
python3 -m pytest -q
</code></pre>
<p>The release candidate was also verified locally with release smoke, benchmark modes, JSON validation, and pytest.</p>
<h2>Benchmark Skeleton</h2>
<p>This release introduces the initial benchmark structure:</p>
<pre><code class="language-text">experiments/
experiments/README.md
experiments/run_benchmark.py
experiments/workloads/deterministic_heavy_v0.json
experiments/results/.gitkeep
</code></pre>
<p>The benchmark runner currently supports:</p>
<ul>
<li><code>dry-run</code></li>
<li><code>direct-baseline</code></li>
<li><code>kora-controlled</code></li>
</ul>
<p>These modes are simulated and metadata-based. They do not call real models or external APIs.</p>
<h2>Current Controlled Benchmark Result</h2>
<p>| Metric | Value |
|---|---:|
| Total tasks | 20 |
| Direct-baseline simulated model invocations | 20 |
| KORA-controlled simulated model invocations | 4 |
| Deterministic resolutions | 16 |
| Fallback candidates | 4 |
| Avoided simulated model invocations | 16 |
| Avoided invocation rate | 80% |</p>
<h2>How To Verify Locally</h2>
<pre><code class="language-bash">./scripts/release_smoke.sh
python3 -m pytest -q
python3 experiments/run_benchmark.py --mode dry-run --workload experiments/workloads/deterministic_heavy_v0.json --output /tmp/kora_deterministic_heavy_v0.dry_run.json
python3 experiments/run_benchmark.py --mode direct-baseline --workload experiments/workloads/deterministic_heavy_v0.json --output /tmp/kora_deterministic_heavy_v0.direct_baseline.json
python3 experiments/run_benchmark.py --mode kora-controlled --workload experiments/workloads/deterministic_heavy_v0.json --output /tmp/kora_deterministic_heavy_v0.kora_controlled.json
</code></pre>
<p>Optional JSON checks:</p>
<pre><code class="language-bash">python3 -m json.tool /tmp/kora_deterministic_heavy_v0.dry_run.json > /tmp/kora_dry_run_check.json
python3 -m json.tool /tmp/kora_deterministic_heavy_v0.direct_baseline.json > /tmp/kora_direct_baseline_check.json
python3 -m json.tool /tmp/kora_deterministic_heavy_v0.kora_controlled.json > /tmp/kora_kora_controlled_check.json
</code></pre>
<h2>Limitations</h2>
<ul>
<li>The benchmark is simulated.</li>
<li>No real model calls are made.</li>
<li>No external API calls are made.</li>
<li>No real API-cost measurement is included.</li>
<li>No production cost reduction claim is supported.</li>
<li>No full KORA runtime integration is implemented yet.</li>
<li>The workload is intentionally small and deterministic-heavy.</li>
</ul>
<h2>Next Planned Work</h2>
<ul>
<li>Decide whether benchmark result artifacts should remain generated-only or be committed under a tracked path.</li>
<li>Add a concise benchmark result summary markdown if the result should be reviewed publicly.</li>
<li>Extend the benchmark runner toward real KORA runtime integration in a later alpha.</li>
<li>Continue expanding deterministic-heavy workloads beyond the initial 20-task draft.</li>
</ul>
Keywords
Citation
Collections
Endorsement
Review
Supplemented By
Referenced By
Rights and licensing
info:eu-repo/semantics/openAccess
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0