Benchmark · Measured October 7–11, 2026
CI Bench v1
Time and cost per job for GitHub Actions runners, on real open-source CI.
Lower and further left is better. GitHub-hosted 2 vCPU = 1.0. Warm + parallel: warm starts and test parallelization.
Key results
- Faster than GitHub-hosted 2 vCPU
- 2.29x
- Vitko Runners with warm starts and test parallelization
- Lower cost per job
- 84%
- Than GitHub-hosted 2 vCPU, with or without those features
- Faster, without either feature
- 1.74x
- Plain Vitko Runners, at 84% lower cost per job
About
Twelve CI workloads from popular open-source projects, nine runner configurations, 930 measured runs each. The same workflow and commit ran everywhere; only the runner changed. The method, workflows and every raw job record are public.
- One self-hosted configuration, RunsOn on AWS, has a lower cost index at high volume. Limitations
- Our own benchmark: Vitko sells one of the products measured. Conflict of interest
All nine configurations
Relative to GitHub-hosted 2 vCPU (= 1.00), geometric mean over the twelve workloads. Lower is better. Small figures are 95% intervals. Sorted by time index.
| Configuration | Time index | Speed vs GitHub 2 vCPU | Cost index | Cost vs GitHub 2 vCPU | Time × cost |
|---|---|---|---|---|---|
| Vitko Runners with warm starts and test parallelization | 0.4360.423–0.459 | 2.29x faster2.18–2.36x | 0.1580.156–0.160 | 84% lower | 0.069 |
| Vitko RunnersWithout warm starts or test parallelization | 0.5740.556–0.602 | 1.74x faster1.66–1.80x | 0.1570.155–0.159 | 84% lower | 0.090 |
| GitHub-hosted, 4 vCPUA larger machine size | 0.6420.629–0.669 | 1.56x faster1.50–1.59x | 1.3871.350–1.424 | 39% higher | 0.891 |
| Blacksmith, 2 vCPU | 0.6450.606–0.759 | 1.55x faster1.32–1.65x | 0.4820.435–0.537 | 52% lower | 0.311 |
| Google Cloud pool, self-hostedn4d-standard-2, always on, on-demand; all-in at 50% utilization | 0.6670.658–0.692 | 1.50x faster1.45–1.52x | 0.4090.386–0.440 | 59% lower | 0.273 |
| RunsOn on AWS, self-hosted, 2 vCPUDefault settings (spot); all-in | 0.8280.816–0.859 | 1.21x faster1.16–1.23x | 0.1210.117–0.125 | 88% lower | 0.100 |
| AWS EC2, self-hosted, one VM per jobm8a.large, on-demand; all-in | 0.8690.833–0.947 | 1.15x faster1.06–1.20x | 0.3830.358–0.405 | 62% lower | 0.333 |
| Ubicloud standard, 2 vCPU | 0.8790.847–0.921 | 1.14x faster1.09–1.18x | 0.1730.167–0.180 | 83% lower | 0.152 |
| GitHub-hosted, 2 vCPUThe baseline (= 1.00) | 1.000 | — | 1.000 | — | 1.000 |
- With warm starts and test parallelization, Vitko Runners has the lowest time index of the nine configurations. Its interval (0.423–0.459) does not overlap any other configuration’s.
- GitHub-hosted 4 vCPU is a larger machine. Its lead over 2 vCPU comes from machine size, not speed per vCPU.
- Self-hosted options run in your own cloud account, and you operate them. Their cost depends on volume; Vitko Runners’ indices do not change with volume.
- Time × cost multiplies the two indices: lower is better on both together.
Breakdowns
By workload
With warm starts and test parallelization, Vitko Runners has the lowest median wall time on 8 of 12 workloads. Blacksmith is fastest on 1 (the burst of 20), plain Vitko Runners on 1 (codex-lint), RunsOn on 1 (container) and the always-on Google Cloud pool on 1 (flask).
Median wall time per job, seconds
Includes waiting for a runner. The lowest in each row is bold.
| Workload | Vitko (warm + parallel) | Vitko | GitHub 4 vCPU | Blacksmith | Google Cloud pool | RunsOn | AWS EC2 ephemeral | Ubicloud | GitHub 2 vCPU |
|---|---|---|---|---|---|---|---|---|---|
| burst | 131 | 160 | 154 | 97 | 153 | 177 | 161 | 203 | 259 |
| cli | 140 | 272 | 192 | 286 | 310 | 327 | 357 | 359 | 368 |
| codex-lint | 405 | 404 | 476 | 448 | 581 | 623 | 512 | 682 | 884 |
| container | 178 | 179 | 185 | 386 | 178 | 174 | 247 | 213 | 239 |
| fd | 45 | 49 | 66 | 65 | 64 | 89 | 114 | 87 | 92 |
| flask | 13 | 16 | 21 | 22 | 11 | 38 | 70 | 36 | 23 |
| guava | 651 | 655 | 946 | 735 | 820 | 897 | 717 | 881 | 1167 |
| monorepo | 172 | 180 | 220 | 177 | 278 | 288 | 281 | 276 | 384 |
| pydantic | 69 | 105 | 154 | 103 | 119 | 147 | 154 | 159 | 179 |
| ts | 243 | 372 | 435 | 401 | 533 | 570 | 433 | 560 | 784 |
| vite | 199 | 350 | 288 | 360 | 401 | 437 | 403 | 502 | 603 |
| vue | 32 | 63 | 67 | 76 | 76 | 101 | 117 | 114 | 121 |
Mean cost per job, US dollars
Self-hosted options at 100,000 jobs a month. The lowest in each row is bold.
| Workload | Vitko (warm + parallel) | Vitko | GitHub 4 vCPU | Blacksmith | Google Cloud pool | RunsOn | AWS EC2 ephemeral | Ubicloud | GitHub 2 vCPU |
|---|---|---|---|---|---|---|---|---|---|
| burst | $0.0520 | $0.0556 | $0.6356 | $0.1656 | $0.1351 | $0.0511 | $0.1112 | $0.0623 | $0.4998 |
| cli | $0.0117 | $0.0097 | $0.0452 | $0.0204 | $0.0173 | $0.0041 | $0.0139 | $0.0078 | $0.0394 |
| codex-lint | $0.0135 | $0.0135 | $0.0989 | $0.0336 | $0.0636 | $0.0101 | $0.0489 | $0.0149 | $0.0872 |
| container | $0.0061 | $0.0060 | $0.0424 | $0.0307 | $0.0532 | $0.0060 | $0.0436 | $0.0047 | $0.0266 |
| fd | $0.0014 | $0.0016 | $0.0192 | $0.0055 | $0.0035 | $0.0018 | $0.0047 | $0.0021 | $0.0120 |
| flask | $0.0004 | $0.0005 | $0.0120 | $0.0040 | $0.0011 | $0.0014 | $0.0036 | $0.0013 | $0.0060 |
| guava | $0.0216 | $0.0218 | $0.1808 | $0.0512 | $0.0449 | $0.0096 | $0.0318 | $0.0192 | $0.1090 |
| monorepo | $0.0056 | $0.0058 | $0.0464 | $0.0140 | $0.0158 | $0.0039 | $0.0116 | $0.0061 | $0.0396 |
| pydantic | $0.0022 | $0.0034 | $0.0332 | $0.0087 | $0.0065 | $0.0024 | $0.0064 | $0.0037 | $0.0196 |
| ts | $0.0274 | $0.0122 | $0.0992 | $0.0299 | $0.0290 | $0.0061 | $0.0168 | $0.0120 | $0.0774 |
| vite | $0.0127 | $0.0115 | $0.0604 | $0.0264 | $0.0231 | $0.0053 | $0.0163 | $0.0107 | $0.0609 |
| vue | $0.0017 | $0.0020 | $0.0200 | $0.0077 | $0.0044 | $0.0020 | $0.0051 | $0.0027 | $0.0140 |
Bursts of 20
Twenty runs dispatched at once, per round. At the 95th percentile, queue time was 115 s with warm starts and test parallelization and 295 s for plain Vitko Runners, against 31 s for GitHub-hosted 2 vCPU and 3 s for the always-on pool.
| Configuration | Queue, median (s) | Queue, 95th pct (s) | Wall time, median (s) |
|---|---|---|---|
| Vitko Runners with warm starts and test parallelization | 17 | 115 | 131 |
| Vitko Runners | 18 | 295 | 160 |
| GitHub-hosted, 4 vCPU | 3 | 39 | 154 |
| Blacksmith, 2 vCPU | 9 | 16 | 97 |
| Google Cloud pool, self-hosted | 2 | 3 | 153 |
| RunsOn on AWS, self-hosted, 2 vCPU | 23 | 194 | 177 |
| AWS EC2, self-hosted, one VM per job | 65 | 93 | 161 |
| Ubicloud standard, 2 vCPU | 18 | 127 | 203 |
| GitHub-hosted, 2 vCPU | 3 | 31 | 259 |
Cost by volume
Self-hosted options carry fixed monthly costs (control plane, license) spread over the job count, so their cost index depends on volume. The results table uses 100,000 jobs a month.
| Configuration | 100,000 jobs a month | 10,000 | 1,000 |
|---|---|---|---|
| Vitko Runners with warm starts and test parallelization | 0.158 | 0.158 | 0.158 |
| Vitko Runners | 0.157 | 0.157 | 0.157 |
| GitHub-hosted, 4 vCPU | 1.387 | 1.387 | 1.387 |
| Blacksmith, 2 vCPU | 0.482 | 0.482 | 0.482 |
| Google Cloud pool, self-hosted | 0.409 | 0.409 | 0.409 |
| RunsOn on AWS, self-hosted, 2 vCPU | 0.121 | 0.250 | 1.431 |
| AWS EC2, self-hosted, one VM per job | 0.383 | 0.395 | 0.508 |
| Ubicloud standard, 2 vCPU | 0.173 | 0.173 | 0.173 |
| GitHub-hosted, 2 vCPU | 1.000 | 1.000 | 1.000 |
Per session
Time index / cost index in each of the three sessions; pooled is computed from all three.
| Configuration | Session 1 | Session 2 | Session 3 | Pooled |
|---|---|---|---|---|
| Vitko Runners with warm starts and test parallelization | 0.43 / 0.16 | 0.45 / 0.16 | 0.43 / 0.16 | 0.44 / 0.16 |
| Vitko Runners | 0.57 / 0.16 | 0.58 / 0.16 | 0.57 / 0.16 | 0.57 / 0.16 |
| GitHub-hosted, 4 vCPU | 0.66 / 1.41 | 0.64 / 1.37 | 0.64 / 1.38 | 0.64 / 1.39 |
| Blacksmith, 2 vCPU | 0.77 / 0.54 | 0.66 / 0.47 | 0.60 / 0.43 | 0.64 / 0.48 |
| Google Cloud pool, self-hosted | 0.68 / 0.43 | 0.67 / 0.39 | 0.66 / 0.39 | 0.67 / 0.41 |
| RunsOn on AWS, self-hosted, 2 vCPU | 0.85 / 0.12 | 0.83 / 0.12 | 0.82 / 0.12 | 0.83 / 0.12 |
| AWS EC2, self-hosted, one VM per job | 0.94 / 0.40 | 0.90 / 0.39 | 0.83 / 0.36 | 0.87 / 0.38 |
| Ubicloud standard, 2 vCPU | 0.89 / 0.17 | 0.91 / 0.18 | 0.85 / 0.17 | 0.88 / 0.17 |
| GitHub-hosted, 2 vCPU | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |
How we measured
- Workloads. Eleven open-source projects’ own CI jobs at pinned releases, plus a burst of 20 concurrent runs. The same workflow and commit on every provider; only the runner changed.
- Providers. Nine configurations, listed above. A provider is included only if its terms allow publishing benchmark results.
- Runs. Three sessions of 10 measured rounds each, after 2 warm-up rounds: 930 measured runs per configuration. Time comes from GitHub’s own job timestamps and includes waiting for a runner. Failures are counted, never replaced.
- Pre-registered. The method was fixed before the sessions ran (tag
prereg-v1), and every later change is dated with its reason. Full method: METHOD.md.
How cost and the indices are calculated
- Cost is each provider’s public list price for the size used, applied the way that provider bills (Vitko bills per second; GitHub rounds each job up to a minute). Waiting is not billed by anyone. Free minutes and discounts are left out. Test parallelization is billed on the parts only while a split step runs.
- Managed and self-hosted. Managed providers are priced at list price under their own billing rule. Self-hosted options are priced all-in: everything their jobs use in your cloud account, measured per job, plus fixed monthly costs spread over the job count. List prices, with sources and dates, are in the repository (prices/2026-10-05.json).
- Indices. Each configuration’s time and cost are divided by GitHub-hosted 2 vCPU’s on the same workload, then combined by geometric mean over the twelve workloads.
- Intervals. 95% block-bootstrap intervals over sessions and rounds.
- Bursts. Time is from the first job created to the last job finished.
Limitations
- RunsOn costs less per job at high volume. At 100,000 jobs a month, RunsOn (self-hosted on AWS) has the lowest cost index, 0.121 against 0.157 for plain Vitko Runners, and the lowest mean cost on 7 of 12 workloads. Its fixed costs are spread over the job count, so its index rises to 0.250 at 10,000 jobs a month and 1.431 at 1,000, above Vitko Runners’ 0.157 and 0.157. You operate it in your own AWS account.
- One workload’s end-to-end tests are flaky across providers. Apart from one GitHub-hosted 4 vCPU codex-lint run that hung, every failed run was in vite’s end-to-end tests (the build or serve step), which failed on most providers. Of the 15 failed runs, 3 were Vitko Runners with warm starts and test parallelization (vite succeeded in 27 of 30 runs there). Plain Vitko Runners had no failed measured run.
- Bursts of 20: queue time at the 95th percentile was 115 s for Vitko with warm starts and test parallelization and 295 s for plain Vitko Runners, against 31 s for GitHub-hosted 2 vCPU. Vitko Runners is not the fastest on every workload (fastest on 8 of 12).
- Not every provider was tested. Depot was planned and excluded because its runners never started a job during the test window.
- Scope. The 2 vCPU class (plus one 4 vCPU reference), Linux x64, one region per provider, public repositories. No GPU, macOS, Windows or Arm. Hardware differs by provider; each job is reported as it ran.
- Warm starts were measured on an earlier build. The Vitko Runners build used for the benchmark predates later cleanup of restored setups between jobs. The repository’s list of deviations has the details.
Conflict of interest
Vitko, Inc. maintains this benchmark and sells Vitko Runners, one of the products measured. We pre-registered the method, published the workflows and raw records, list every failed or re-run run, and state the limitations. We operated the self-hosted options, with the settings published in the repository. Providers may report errors, and corrections are recorded in a dated changelog.
Data and downloads
- Results and write-up: github.com/vitko-inc/runner-benchmark, results/v1 (tag
results-v1). Method, workflows and analysis are in the same repository. - Downloads: results.json (aggregate output), summary.csv (per workload and configuration) and runs.csv (every measured run).
- License: data under CC BY 4.0, code under Apache-2.0.
- Cite as: Vitko, Inc. CI Bench v1 (2026). https://runners.vitko.inc/benchmarks/