Benchmark · Measured October 7–11, 2026
CI Bench v1: an open, reproducible benchmark of GitHub Actions runners
Time and cost per job for nine runner configurations on twelve real open-source CI jobs. The method, the workflows and every raw job record are public, so you can check or rerun the numbers.
Lower and further left is better. GitHub-hosted 2 vCPU = 1.0. Warm + parallel: warm starts and test parallelization.
Key results
- Faster than GitHub-hosted 2 vCPU
- 2.29x
- Vitko Runners with warm starts and test parallelization
- Lower cost per job
- 84%
- Than GitHub-hosted 2 vCPU, with or without those features
- Faster, without either feature
- 1.74x
- Plain Vitko Runners, at 84% lower cost per job
About
Twelve CI workloads from popular open-source projects, nine runner configurations, 930 measured runs each. The same workflow and commit ran everywhere; only the runner changed. The method, workflows and every raw job record are public.
- One self-hosted configuration, RunsOn on AWS, has a lower cost index at high volume. Limitations
- Our own benchmark: Vitko sells one of the products measured. Conflict of interest
All nine configurations
Relative to GitHub-hosted 2 vCPU (= 1.00), geometric mean over the twelve workloads. Lower is better. Small figures are 95% intervals. Sorted by time index.
| Configuration | Time index | Speed vs GitHub 2 vCPU | Cost index | Cost vs GitHub 2 vCPU | Time × cost |
|---|---|---|---|---|---|
| Vitko Runners with warm starts and test parallelization | 0.4360.423–0.459 | 2.29x faster2.18–2.36x | 0.1580.156–0.160 | 84% lower | 0.069 |
| Vitko RunnersWithout warm starts or test parallelization | 0.5740.556–0.602 | 1.74x faster1.66–1.80x | 0.1570.155–0.159 | 84% lower | 0.090 |
| GitHub-hosted, 4 vCPUA larger machine size | 0.6420.629–0.669 | 1.56x faster1.50–1.59x | 1.3871.350–1.424 | 39% higher | 0.891 |
| Blacksmith, 2 vCPU | 0.6450.606–0.759 | 1.55x faster1.32–1.65x | 0.4820.435–0.537 | 52% lower | 0.311 |
| Google Cloud pool, self-hostedn4d-standard-2, always on, on-demand; all-in at 50% utilization | 0.6670.658–0.692 | 1.50x faster1.45–1.52x | 0.4090.386–0.440 | 59% lower | 0.273 |
| RunsOn on AWS, self-hosted, 2 vCPUDefault settings (spot); all-in | 0.8280.816–0.859 | 1.21x faster1.16–1.23x | 0.1210.117–0.125 | 88% lower | 0.100 |
| AWS EC2, self-hosted, one VM per jobm8a.large, on-demand; all-in | 0.8690.833–0.947 | 1.15x faster1.06–1.20x | 0.3830.358–0.405 | 62% lower | 0.333 |
| Ubicloud standard, 2 vCPU | 0.8790.847–0.921 | 1.14x faster1.09–1.18x | 0.1730.167–0.180 | 83% lower | 0.152 |
| GitHub-hosted, 2 vCPUThe baseline (= 1.00) | 1.000 | — | 1.000 | — | 1.000 |
- With warm starts and test parallelization, Vitko Runners has the lowest time index of the nine configurations. Its interval (0.423–0.459) does not overlap any other configuration’s.
- GitHub-hosted 4 vCPU is a larger machine. Its lead over 2 vCPU comes from machine size, not speed per vCPU.
- Self-hosted options run in your own cloud account, and you operate them. Their cost depends on volume; Vitko Runners’ indices do not change with volume.
- Time × cost multiplies the two indices: lower is better on both together.
Vitko Runners against each runner
Measured in CI Bench v1 (eleven of the twelve real CI workloads, 2 vCPU configurations, October 7 to 11, 2026; the container workload is left out for every runner because Blacksmith’s Docker layer cache did not hit in that run; Vitko maintains the benchmark and sells one of the runners measured; the method and every raw record are public). Speed is median job time across the workloads; cost per job uses list prices dated October 5, 2026.
| Compared with | Vitko Runners speed against it | With warm starts and test splitting | Cost per job, Vitko Runners | Workloads where Vitko Runners had the lower median | Detail |
|---|---|---|---|---|---|
| GitHub-hosted runners (2 vCPU) | 1.78x as fast | 2.41x as fast | 85% lower | 11 of 11 (without container) | Vitko Runners vs GitHub-hosted runners |
| Blacksmith | 1.06x as fast | 1.43x as fast | 66% lower | 10 of 11 (without container) | Vitko Runners vs Blacksmith |
| Ubicloud | 1.57x as fast | 2.11x as fast | 12% lower | 11 of 11 (without container) | Vitko Runners vs Ubicloud |
| RunsOn | 1.49x as fast | 2.02x as fast | 33% higher | 11 of 11 (without container) | Vitko Runners vs RunsOn |
| Depot | Not measured in CI Bench v1, so no speed or cost-per-job comparison. Published list price: $0.006 a minute, per second; Vitko Runners $0.002 a minute, per second. | n/a | Vitko Runners vs Depot | ||
| WarpBuild | Not measured in CI Bench v1, so no speed or cost-per-job comparison. Published list price: $0.004 a minute, per minute; Vitko Runners $0.002 a minute, per second. | n/a | Vitko Runners vs WarpBuild | ||
| Namespace | Not measured in CI Bench v1, so no speed or cost-per-job comparison. Published list price: $0.004 a minute prepaid (unit minutes); Vitko Runners $0.002 a minute, per second. | n/a | Vitko Runners vs Namespace | ||
Blacksmith’s Docker layer cache did not hit in the container workload run, which is why that workload is left out above; see the CI Bench method. Vitko Runners is in private beta: Ubuntu 24.04 on x64, private repositories only. List price per minute is not cost per job; faster runners finish sooner.
Breakdowns
By workload
With warm starts and test parallelization, Vitko Runners has the lowest median wall time on 8 of 12 workloads. Blacksmith is fastest on 1 (the burst of 20), plain Vitko Runners on 1 (codex-lint), RunsOn on 1 (container) and the always-on Google Cloud pool on 1 (flask).
Median wall time per job, seconds
Includes waiting for a runner. The lowest in each row is bold.
| Workload | Vitko (warm + parallel) | Vitko | GitHub 4 vCPU | Blacksmith | Google Cloud pool | RunsOn | AWS EC2 ephemeral | Ubicloud | GitHub 2 vCPU |
|---|---|---|---|---|---|---|---|---|---|
| burst | 131 | 160 | 154 | 97 | 153 | 177 | 161 | 203 | 259 |
| cli | 140 | 272 | 192 | 286 | 310 | 327 | 357 | 359 | 368 |
| codex-lint | 405 | 404 | 476 | 448 | 581 | 623 | 512 | 682 | 884 |
| container | 178 | 179 | 185 | 386 | 178 | 174 | 247 | 213 | 239 |
| fd | 45 | 49 | 66 | 65 | 64 | 89 | 114 | 87 | 92 |
| flask | 13 | 16 | 21 | 22 | 11 | 38 | 70 | 36 | 23 |
| guava | 651 | 655 | 946 | 735 | 820 | 897 | 717 | 881 | 1167 |
| monorepo | 172 | 180 | 220 | 177 | 278 | 288 | 281 | 276 | 384 |
| pydantic | 69 | 105 | 154 | 103 | 119 | 147 | 154 | 159 | 179 |
| ts | 243 | 372 | 435 | 401 | 533 | 570 | 433 | 560 | 784 |
| vite | 199 | 350 | 288 | 360 | 401 | 437 | 403 | 502 | 603 |
| vue | 32 | 63 | 67 | 76 | 76 | 101 | 117 | 114 | 121 |
Mean cost per job, US dollars
Self-hosted options at 100,000 jobs a month. The lowest in each row is bold.
| Workload | Vitko (warm + parallel) | Vitko | GitHub 4 vCPU | Blacksmith | Google Cloud pool | RunsOn | AWS EC2 ephemeral | Ubicloud | GitHub 2 vCPU |
|---|---|---|---|---|---|---|---|---|---|
| burst | $0.0520 | $0.0556 | $0.6356 | $0.1656 | $0.1351 | $0.0511 | $0.1112 | $0.0623 | $0.4998 |
| cli | $0.0117 | $0.0097 | $0.0452 | $0.0204 | $0.0173 | $0.0041 | $0.0139 | $0.0078 | $0.0394 |
| codex-lint | $0.0135 | $0.0135 | $0.0989 | $0.0336 | $0.0636 | $0.0101 | $0.0489 | $0.0149 | $0.0872 |
| container | $0.0061 | $0.0060 | $0.0424 | $0.0307 | $0.0532 | $0.0060 | $0.0436 | $0.0047 | $0.0266 |
| fd | $0.0014 | $0.0016 | $0.0192 | $0.0055 | $0.0035 | $0.0018 | $0.0047 | $0.0021 | $0.0120 |
| flask | $0.0004 | $0.0005 | $0.0120 | $0.0040 | $0.0011 | $0.0014 | $0.0036 | $0.0013 | $0.0060 |
| guava | $0.0216 | $0.0218 | $0.1808 | $0.0512 | $0.0449 | $0.0096 | $0.0318 | $0.0192 | $0.1090 |
| monorepo | $0.0056 | $0.0058 | $0.0464 | $0.0140 | $0.0158 | $0.0039 | $0.0116 | $0.0061 | $0.0396 |
| pydantic | $0.0022 | $0.0034 | $0.0332 | $0.0087 | $0.0065 | $0.0024 | $0.0064 | $0.0037 | $0.0196 |
| ts | $0.0274 | $0.0122 | $0.0992 | $0.0299 | $0.0290 | $0.0061 | $0.0168 | $0.0120 | $0.0774 |
| vite | $0.0127 | $0.0115 | $0.0604 | $0.0264 | $0.0231 | $0.0053 | $0.0163 | $0.0107 | $0.0609 |
| vue | $0.0017 | $0.0020 | $0.0200 | $0.0077 | $0.0044 | $0.0020 | $0.0051 | $0.0027 | $0.0140 |
Bursts of 20
Twenty runs dispatched at once, per round: queue time, job time and cost for every configuration are on cibench.com, the open benchmark site.
Cost by volume
Self-hosted options carry fixed monthly costs (control plane, license) spread over the job count, so their cost index depends on volume. The results table uses 100,000 jobs a month.
| Configuration | 100,000 jobs a month | 10,000 | 1,000 |
|---|---|---|---|
| Vitko Runners with warm starts and test parallelization | 0.158 | 0.158 | 0.158 |
| Vitko Runners | 0.157 | 0.157 | 0.157 |
| GitHub-hosted, 4 vCPU | 1.387 | 1.387 | 1.387 |
| Blacksmith, 2 vCPU | 0.482 | 0.482 | 0.482 |
| Google Cloud pool, self-hosted | 0.409 | 0.409 | 0.409 |
| RunsOn on AWS, self-hosted, 2 vCPU | 0.121 | 0.250 | 1.431 |
| AWS EC2, self-hosted, one VM per job | 0.383 | 0.395 | 0.508 |
| Ubicloud standard, 2 vCPU | 0.173 | 0.173 | 0.173 |
| GitHub-hosted, 2 vCPU | 1.000 | 1.000 | 1.000 |
Per session
Time index / cost index in each of the three sessions; pooled is computed from all three.
| Configuration | Session 1 | Session 2 | Session 3 | Pooled |
|---|---|---|---|---|
| Vitko Runners with warm starts and test parallelization | 0.43 / 0.16 | 0.45 / 0.16 | 0.43 / 0.16 | 0.44 / 0.16 |
| Vitko Runners | 0.57 / 0.16 | 0.58 / 0.16 | 0.57 / 0.16 | 0.57 / 0.16 |
| GitHub-hosted, 4 vCPU | 0.66 / 1.41 | 0.64 / 1.37 | 0.64 / 1.38 | 0.64 / 1.39 |
| Blacksmith, 2 vCPU | 0.77 / 0.54 | 0.66 / 0.47 | 0.60 / 0.43 | 0.64 / 0.48 |
| Google Cloud pool, self-hosted | 0.68 / 0.43 | 0.67 / 0.39 | 0.66 / 0.39 | 0.67 / 0.41 |
| RunsOn on AWS, self-hosted, 2 vCPU | 0.85 / 0.12 | 0.83 / 0.12 | 0.82 / 0.12 | 0.83 / 0.12 |
| AWS EC2, self-hosted, one VM per job | 0.94 / 0.40 | 0.90 / 0.39 | 0.83 / 0.36 | 0.87 / 0.38 |
| Ubicloud standard, 2 vCPU | 0.89 / 0.17 | 0.91 / 0.18 | 0.85 / 0.17 | 0.88 / 0.17 |
| GitHub-hosted, 2 vCPU | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 |
How we measured
- Workloads. Eleven open-source projects’ own CI jobs at pinned releases, plus a burst of 20 concurrent runs. The same workflow and commit on every provider; only the runner changed.
- Providers. Nine configurations, listed above. A provider is included only if its terms allow publishing benchmark results.
- Runs. Three sessions of 10 measured rounds each, after 2 warm-up rounds: 930 measured runs per configuration. Time comes from GitHub’s own job timestamps and includes waiting for a runner. Failures are counted, never replaced.
- Pre-registered. The method was fixed before the sessions ran (tag
prereg-v1), and every later change is dated with its reason. Full method: METHOD.md.
How cost and the indices are calculated
- Cost is each provider’s public list price for the size used, applied the way that provider bills (Vitko bills per second; GitHub rounds each job up to a minute). Waiting is not billed by anyone. Free minutes and discounts are left out. Test parallelization is billed on the parts only while a split step runs.
- Managed and self-hosted. Managed providers are priced at list price under their own billing rule. Self-hosted options are priced all-in: everything their jobs use in your cloud account, measured per job, plus fixed monthly costs spread over the job count. List prices, with sources and dates, are in the repository (prices/2026-10-05.json).
- Indices. Each configuration’s time and cost are divided by GitHub-hosted 2 vCPU’s on the same workload, then combined by geometric mean over the twelve workloads.
- Intervals. 95% block-bootstrap intervals over sessions and rounds.
- Bursts. Time is from the first job created to the last job finished.
Limitations
- RunsOn costs less per job at high volume. At 100,000 jobs a month, RunsOn (self-hosted on AWS) has the lowest cost index, 0.121 against 0.157 for plain Vitko Runners, and the lowest mean cost on 7 of 12 workloads. Its fixed costs are spread over the job count, so its index rises to 0.250 at 10,000 jobs a month and 1.431 at 1,000, above Vitko Runners’ 0.157 and 0.157. You operate it in your own AWS account.
- One workload’s end-to-end tests are flaky across providers. Apart from one GitHub-hosted 4 vCPU codex-lint run that hung, every failed run was in vite’s end-to-end tests (the build or serve step), which failed on most providers. Of the 15 failed runs, 3 were Vitko Runners with warm starts and test parallelization (vite succeeded in 27 of 30 runs there). Plain Vitko Runners had no failed measured run.
- Bursts of 20: queue time at the 95th percentile was 115 s for Vitko with warm starts and test parallelization and 295 s for plain Vitko Runners, against 31 s for GitHub-hosted 2 vCPU. Vitko Runners is not the fastest on every workload (fastest on 8 of 12).
- Not every provider was tested. Depot was planned and excluded because its runners never started a job during the test window.
- Scope. The 2 vCPU class (plus one 4 vCPU reference), Linux x64, one region per provider, public repositories. No GPU, macOS, Windows or Arm. Hardware differs by provider; each job is reported as it ran.
- Warm starts were measured on an earlier build. The Vitko Runners build used for the benchmark predates a later change to how a saved setup is cleaned between jobs. The repository’s list of deviations has the details.
Conflict of interest
Vitko, Inc. maintains this benchmark and sells Vitko Runners, one of the products measured. We pre-registered the method, published the workflows and raw records, list every failed or re-run run, and state the limitations. We operated the self-hosted options, with the settings published in the repository. Providers may report errors, and corrections are recorded in a dated changelog.
Data and downloads
- Results and write-up: github.com/vitko-inc/runner-benchmark, results/v1 (tag
results-v1). Method, workflows and analysis are in the same repository. The results also have their own site, cibench.com. - Downloads: results.json (aggregate output), summary.csv (per workload and configuration) and runs.csv (every measured run).
- License: data under CC BY 4.0, code under Apache-2.0.
- Cite as: Vitko, Inc. CI Bench v1 (2026). https://runners.vitko.inc/benchmarks/
Questions this page answers
Is there an open, reproducible benchmark comparing GitHub Actions runner providers on real open-source CI jobs?
Yes: CI Bench v1, measured October 7 to 11, 2026 on twelve CI workloads from open-source projects and nine runner configurations, 30 runs per cell. The pinned workflows, the plan written before the runs, the analysis code and every raw job record, failed runs included, are public at github.com/vitko-inc/runner-benchmark, and results are CC BY 4.0. Vitko maintains it and sells one of the runners measured; it is shown in the same columns as the others.
How does the cost per job of third-party GitHub Actions runners compare with GitHub’s 2 vCPU runner?
In CI Bench v1, at 100,000 jobs a month, cost per job against GitHub-hosted 2 vCPU was 84% lower for Vitko Runners, 83% lower for Ubicloud, 52% lower for Blacksmith and 88% lower for RunsOn (self-hosted in your own AWS account, so excluding setup and operating work). Median job time against GitHub-hosted: Vitko Runners 1.7x as fast, Blacksmith 1.6x. Blacksmith’s container result is flagged because its Docker layer cache did not hit in that run. Per-workload tables are on this page and at cibench.com.