<link rel="stylesheet" href="styles.f3b1fba60ec7970c.css">

hkalbertkim/KORA: KORA v0.1.1-alpha

Loading...
Thumbnail Image

Date

Journal Title

Journal ISSN

Volume Title

Publisher

Zenodo

DOI

Abstract

Description

&lt;h1&gt;KORA v0.1.1-alpha&lt;/h1&gt; &lt;h2&gt;Summary&lt;/h2&gt; &lt;p&gt;KORA v0.1.1-alpha adds CI-backed validation and the first controlled benchmark skeleton. In a deterministic-heavy benchmark skeleton, KORA-controlled execution avoided 16 of 20 simulated model invocations versus a naive direct baseline.&lt;/p&gt; &lt;p&gt;This is an alpha maintenance / benchmark skeleton release. It keeps the v0.1.0-alpha terminal-first surface intact while adding a reproducible CI and benchmark path for future technical preview work.&lt;/p&gt; &lt;h2&gt;What Changed Since v0.1.0-alpha&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;Added GitHub Actions CI for editable install, release smoke, and pytest validation.&lt;/li&gt; &lt;li&gt;Added the &lt;code&gt;experiments/&lt;/code&gt; benchmark experiment directory.&lt;/li&gt; &lt;li&gt;Added the deterministic-heavy workload draft at &lt;code&gt;experiments/workloads/deterministic_heavy_v0.json&lt;/code&gt;.&lt;/li&gt; &lt;li&gt;Added dry-run benchmark mode for workload validation and task/category counting.&lt;/li&gt; &lt;li&gt;Added direct-baseline benchmark mode to simulate one model invocation per task.&lt;/li&gt; &lt;li&gt;Added KORA-controlled benchmark mode to simulate deterministic-first execution from workload metadata.&lt;/li&gt; &lt;li&gt;Added the 2026-05-04 progress report.&lt;/li&gt; &lt;li&gt;Added the v0.1.1-alpha release note.&lt;/li&gt; &lt;/ul&gt; &lt;h2&gt;CI-Backed Validation&lt;/h2&gt; &lt;p&gt;GitHub Actions now validates changes to &lt;code&gt;main&lt;/code&gt; with:&lt;/p&gt; &lt;pre&gt;&lt;code class="language-bash"&gt;python3 -m pip install -e &quot;.[dev]&quot; ./scripts/release_smoke.sh python3 -m pytest -q &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;The release candidate was also verified locally with release smoke, benchmark modes, JSON validation, and pytest.&lt;/p&gt; &lt;h2&gt;Benchmark Skeleton&lt;/h2&gt; &lt;p&gt;This release introduces the initial benchmark structure:&lt;/p&gt; &lt;pre&gt;&lt;code class="language-text"&gt;experiments/ experiments/README.md experiments/run_benchmark.py experiments/workloads/deterministic_heavy_v0.json experiments/results/.gitkeep &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;The benchmark runner currently supports:&lt;/p&gt; &lt;ul&gt; &lt;li&gt;&lt;code&gt;dry-run&lt;/code&gt;&lt;/li&gt; &lt;li&gt;&lt;code&gt;direct-baseline&lt;/code&gt;&lt;/li&gt; &lt;li&gt;&lt;code&gt;kora-controlled&lt;/code&gt;&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;These modes are simulated and metadata-based. They do not call real models or external APIs.&lt;/p&gt; &lt;h2&gt;Current Controlled Benchmark Result&lt;/h2&gt; &lt;p&gt;| Metric | Value | |---|---:| | Total tasks | 20 | | Direct-baseline simulated model invocations | 20 | | KORA-controlled simulated model invocations | 4 | | Deterministic resolutions | 16 | | Fallback candidates | 4 | | Avoided simulated model invocations | 16 | | Avoided invocation rate | 80% |&lt;/p&gt; &lt;h2&gt;How To Verify Locally&lt;/h2&gt; &lt;pre&gt;&lt;code class="language-bash"&gt;./scripts/release_smoke.sh python3 -m pytest -q python3 experiments/run_benchmark.py --mode dry-run --workload experiments/workloads/deterministic_heavy_v0.json --output /tmp/kora_deterministic_heavy_v0.dry_run.json python3 experiments/run_benchmark.py --mode direct-baseline --workload experiments/workloads/deterministic_heavy_v0.json --output /tmp/kora_deterministic_heavy_v0.direct_baseline.json python3 experiments/run_benchmark.py --mode kora-controlled --workload experiments/workloads/deterministic_heavy_v0.json --output /tmp/kora_deterministic_heavy_v0.kora_controlled.json &lt;/code&gt;&lt;/pre&gt; &lt;p&gt;Optional JSON checks:&lt;/p&gt; &lt;pre&gt;&lt;code class="language-bash"&gt;python3 -m json.tool /tmp/kora_deterministic_heavy_v0.dry_run.json &gt; /tmp/kora_dry_run_check.json python3 -m json.tool /tmp/kora_deterministic_heavy_v0.direct_baseline.json &gt; /tmp/kora_direct_baseline_check.json python3 -m json.tool /tmp/kora_deterministic_heavy_v0.kora_controlled.json &gt; /tmp/kora_kora_controlled_check.json &lt;/code&gt;&lt;/pre&gt; &lt;h2&gt;Limitations&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;The benchmark is simulated.&lt;/li&gt; &lt;li&gt;No real model calls are made.&lt;/li&gt; &lt;li&gt;No external API calls are made.&lt;/li&gt; &lt;li&gt;No real API-cost measurement is included.&lt;/li&gt; &lt;li&gt;No production cost reduction claim is supported.&lt;/li&gt; &lt;li&gt;No full KORA runtime integration is implemented yet.&lt;/li&gt; &lt;li&gt;The workload is intentionally small and deterministic-heavy.&lt;/li&gt; &lt;/ul&gt; &lt;h2&gt;Next Planned Work&lt;/h2&gt; &lt;ul&gt; &lt;li&gt;Decide whether benchmark result artifacts should remain generated-only or be committed under a tracked path.&lt;/li&gt; &lt;li&gt;Add a concise benchmark result summary markdown if the result should be reviewed publicly.&lt;/li&gt; &lt;li&gt;Extend the benchmark runner toward real KORA runtime integration in a later alpha.&lt;/li&gt; &lt;li&gt;Continue expanding deterministic-heavy workloads beyond the initial 20-task draft.&lt;/li&gt; &lt;/ul&gt;

Keywords

Citation

Collections

Endorsement

Review

Supplemented By

Referenced By

Rights and licensing

info:eu-repo/semantics/openAccess
Apache License 2.0
http://www.apache.org/licenses/LICENSE-2.0