Benchmark · Measured October 7–11, 2026

CI Bench v1: an open, reproducible benchmark of GitHub Actions runners

Time and cost per job for nine runner configurations on twelve real open-source CI jobs. The method, the workflows and every raw job record are public, so you can check or rerun the numbers.

Time and cost per jobTwelve CI workloads from popular open-source projects.
00.20.40.60.81.01.20.10.20.51.0Cost index (log scale) →↑ Time indexVitko (warm + parallel)VitkoGitHub 4 vCPUBlacksmithGoogle Cloud poolRunsOnAWS EC2 ephemeralUbicloudGitHub 2 vCPU

Lower and further left is better. GitHub-hosted 2 vCPU = 1.0. Warm + parallel: warm starts and test parallelization.

Key results

Faster than GitHub-hosted 2 vCPU
2.29x
Vitko Runners with warm starts and test parallelization
Lower cost per job
84%
Than GitHub-hosted 2 vCPU, with or without those features
Faster, without either feature
1.74x
Plain Vitko Runners, at 84% lower cost per job

About

Twelve CI workloads from popular open-source projects, nine runner configurations, 930 measured runs each. The same workflow and commit ran everywhere; only the runner changed. The method, workflows and every raw job record are public.

  • One self-hosted configuration, RunsOn on AWS, has a lower cost index at high volume. Limitations
  • Our own benchmark: Vitko sells one of the products measured. Conflict of interest

All nine configurations

Relative to GitHub-hosted 2 vCPU (= 1.00), geometric mean over the twelve workloads. Lower is better. Small figures are 95% intervals. Sorted by time index.

Time and cost index per configuration
ConfigurationTime indexSpeed vs GitHub 2 vCPUCost indexCost vs GitHub 2 vCPUTime × cost
Vitko Runners with warm starts and test parallelization0.4360.423–0.4592.29x faster2.18–2.36x0.1580.156–0.16084% lower0.069
Vitko RunnersWithout warm starts or test parallelization0.5740.556–0.6021.74x faster1.66–1.80x0.1570.155–0.15984% lower0.090
GitHub-hosted, 4 vCPUA larger machine size0.6420.629–0.6691.56x faster1.50–1.59x1.3871.350–1.42439% higher0.891
Blacksmith, 2 vCPU0.6450.606–0.7591.55x faster1.32–1.65x0.4820.435–0.53752% lower0.311
Google Cloud pool, self-hostedn4d-standard-2, always on, on-demand; all-in at 50% utilization0.6670.658–0.6921.50x faster1.45–1.52x0.4090.386–0.44059% lower0.273
RunsOn on AWS, self-hosted, 2 vCPUDefault settings (spot); all-in0.8280.816–0.8591.21x faster1.16–1.23x0.1210.117–0.12588% lower0.100
AWS EC2, self-hosted, one VM per jobm8a.large, on-demand; all-in0.8690.833–0.9471.15x faster1.06–1.20x0.3830.358–0.40562% lower0.333
Ubicloud standard, 2 vCPU0.8790.847–0.9211.14x faster1.09–1.18x0.1730.167–0.18083% lower0.152
GitHub-hosted, 2 vCPUThe baseline (= 1.00)1.000—1.000—1.000
  • With warm starts and test parallelization, Vitko Runners has the lowest time index of the nine configurations. Its interval (0.423–0.459) does not overlap any other configuration’s.
  • GitHub-hosted 4 vCPU is a larger machine. Its lead over 2 vCPU comes from machine size, not speed per vCPU.
  • Self-hosted options run in your own cloud account, and you operate them. Their cost depends on volume; Vitko Runners’ indices do not change with volume.
  • Time × cost multiplies the two indices: lower is better on both together.

Vitko Runners against each runner

Measured in CI Bench v1 (eleven of the twelve real CI workloads, 2 vCPU configurations, October 7 to 11, 2026; the container workload is left out for every runner because Blacksmith’s Docker layer cache did not hit in that run; Vitko maintains the benchmark and sells one of the runners measured; the method and every raw record are public). Speed is median job time across the workloads; cost per job uses list prices dated October 5, 2026.

Vitko Runners compared with GitHub-hosted runners, Blacksmith, Ubicloud and RunsOn on speed and cost per job
Compared withVitko Runners speed against itWith warm starts and test splittingCost per job, Vitko RunnersWorkloads where Vitko Runners had the lower medianDetail
GitHub-hosted runners (2 vCPU)1.78x as fast2.41x as fast85% lower11 of 11 (without container)Vitko Runners vs GitHub-hosted runners
Blacksmith1.06x as fast1.43x as fast66% lower10 of 11 (without container)Vitko Runners vs Blacksmith
Ubicloud1.57x as fast2.11x as fast12% lower11 of 11 (without container)Vitko Runners vs Ubicloud
RunsOn1.49x as fast2.02x as fast33% higher11 of 11 (without container)Vitko Runners vs RunsOn
DepotNot measured in CI Bench v1, so no speed or cost-per-job comparison. Published list price: $0.006 a minute, per second; Vitko Runners $0.002 a minute, per second.n/aVitko Runners vs Depot
WarpBuildNot measured in CI Bench v1, so no speed or cost-per-job comparison. Published list price: $0.004 a minute, per minute; Vitko Runners $0.002 a minute, per second.n/aVitko Runners vs WarpBuild
NamespaceNot measured in CI Bench v1, so no speed or cost-per-job comparison. Published list price: $0.004 a minute prepaid (unit minutes); Vitko Runners $0.002 a minute, per second.n/aVitko Runners vs Namespace

Blacksmith’s Docker layer cache did not hit in the container workload run, which is why that workload is left out above; see the CI Bench method. Vitko Runners is in private beta: Ubuntu 24.04 on x64, private repositories only. List price per minute is not cost per job; faster runners finish sooner.

Breakdowns

By workload

With warm starts and test parallelization, Vitko Runners has the lowest median wall time on 8 of 12 workloads. Blacksmith is fastest on 1 (the burst of 20), plain Vitko Runners on 1 (codex-lint), RunsOn on 1 (container) and the always-on Google Cloud pool on 1 (flask).

Median wall time per job, seconds

Includes waiting for a runner. The lowest in each row is bold.

Median wall time per workload, seconds
WorkloadVitko (warm + parallel)VitkoGitHub 4 vCPUBlacksmithGoogle Cloud poolRunsOnAWS EC2 ephemeralUbicloudGitHub 2 vCPU
burst13116015497153177161203259
cli140272192286310327357359368
codex-lint405404476448581623512682884
container178179185386178174247213239
fd4549666564891148792
flask131621221138703623
guava6516559467358208977178811167
monorepo172180220177278288281276384
pydantic69105154103119147154159179
ts243372435401533570433560784
vite199350288360401437403502603
vue3263677676101117114121

Mean cost per job, US dollars

Self-hosted options at 100,000 jobs a month. The lowest in each row is bold.

Mean cost per job per workload, USD
WorkloadVitko (warm + parallel)VitkoGitHub 4 vCPUBlacksmithGoogle Cloud poolRunsOnAWS EC2 ephemeralUbicloudGitHub 2 vCPU
burst$0.0520$0.0556$0.6356$0.1656$0.1351$0.0511$0.1112$0.0623$0.4998
cli$0.0117$0.0097$0.0452$0.0204$0.0173$0.0041$0.0139$0.0078$0.0394
codex-lint$0.0135$0.0135$0.0989$0.0336$0.0636$0.0101$0.0489$0.0149$0.0872
container$0.0061$0.0060$0.0424$0.0307$0.0532$0.0060$0.0436$0.0047$0.0266
fd$0.0014$0.0016$0.0192$0.0055$0.0035$0.0018$0.0047$0.0021$0.0120
flask$0.0004$0.0005$0.0120$0.0040$0.0011$0.0014$0.0036$0.0013$0.0060
guava$0.0216$0.0218$0.1808$0.0512$0.0449$0.0096$0.0318$0.0192$0.1090
monorepo$0.0056$0.0058$0.0464$0.0140$0.0158$0.0039$0.0116$0.0061$0.0396
pydantic$0.0022$0.0034$0.0332$0.0087$0.0065$0.0024$0.0064$0.0037$0.0196
ts$0.0274$0.0122$0.0992$0.0299$0.0290$0.0061$0.0168$0.0120$0.0774
vite$0.0127$0.0115$0.0604$0.0264$0.0231$0.0053$0.0163$0.0107$0.0609
vue$0.0017$0.0020$0.0200$0.0077$0.0044$0.0020$0.0051$0.0027$0.0140
Bursts of 20

Twenty runs dispatched at once, per round: queue time, job time and cost for every configuration are on cibench.com, the open benchmark site.

Cost by volume

Self-hosted options carry fixed monthly costs (control plane, license) spread over the job count, so their cost index depends on volume. The results table uses 100,000 jobs a month.

Cost index by jobs per month
Configuration100,000 jobs a month10,0001,000
Vitko Runners with warm starts and test parallelization0.1580.1580.158
Vitko Runners0.1570.1570.157
GitHub-hosted, 4 vCPU1.3871.3871.387
Blacksmith, 2 vCPU0.4820.4820.482
Google Cloud pool, self-hosted0.4090.4090.409
RunsOn on AWS, self-hosted, 2 vCPU0.1210.2501.431
AWS EC2, self-hosted, one VM per job0.3830.3950.508
Ubicloud standard, 2 vCPU0.1730.1730.173
GitHub-hosted, 2 vCPU1.0001.0001.000
Per session

Time index / cost index in each of the three sessions; pooled is computed from all three.

Time / cost index per session
ConfigurationSession 1Session 2Session 3Pooled
Vitko Runners with warm starts and test parallelization0.43 / 0.160.45 / 0.160.43 / 0.160.44 / 0.16
Vitko Runners0.57 / 0.160.58 / 0.160.57 / 0.160.57 / 0.16
GitHub-hosted, 4 vCPU0.66 / 1.410.64 / 1.370.64 / 1.380.64 / 1.39
Blacksmith, 2 vCPU0.77 / 0.540.66 / 0.470.60 / 0.430.64 / 0.48
Google Cloud pool, self-hosted0.68 / 0.430.67 / 0.390.66 / 0.390.67 / 0.41
RunsOn on AWS, self-hosted, 2 vCPU0.85 / 0.120.83 / 0.120.82 / 0.120.83 / 0.12
AWS EC2, self-hosted, one VM per job0.94 / 0.400.90 / 0.390.83 / 0.360.87 / 0.38
Ubicloud standard, 2 vCPU0.89 / 0.170.91 / 0.180.85 / 0.170.88 / 0.17
GitHub-hosted, 2 vCPU1.00 / 1.001.00 / 1.001.00 / 1.001.00 / 1.00

How we measured

  • Workloads. Eleven open-source projects’ own CI jobs at pinned releases, plus a burst of 20 concurrent runs. The same workflow and commit on every provider; only the runner changed.
  • Providers. Nine configurations, listed above. A provider is included only if its terms allow publishing benchmark results.
  • Runs. Three sessions of 10 measured rounds each, after 2 warm-up rounds: 930 measured runs per configuration. Time comes from GitHub’s own job timestamps and includes waiting for a runner. Failures are counted, never replaced.
  • Pre-registered. The method was fixed before the sessions ran (tag prereg-v1), and every later change is dated with its reason. Full method: METHOD.md.
How cost and the indices are calculated
  • Cost is each provider’s public list price for the size used, applied the way that provider bills (Vitko bills per second; GitHub rounds each job up to a minute). Waiting is not billed by anyone. Free minutes and discounts are left out. Test parallelization is billed on the parts only while a split step runs.
  • Managed and self-hosted. Managed providers are priced at list price under their own billing rule. Self-hosted options are priced all-in: everything their jobs use in your cloud account, measured per job, plus fixed monthly costs spread over the job count. List prices, with sources and dates, are in the repository (prices/2026-10-05.json).
  • Indices. Each configuration’s time and cost are divided by GitHub-hosted 2 vCPU’s on the same workload, then combined by geometric mean over the twelve workloads.
  • Intervals. 95% block-bootstrap intervals over sessions and rounds.
  • Bursts. Time is from the first job created to the last job finished.

Limitations

  • RunsOn costs less per job at high volume. At 100,000 jobs a month, RunsOn (self-hosted on AWS) has the lowest cost index, 0.121 against 0.157 for plain Vitko Runners, and the lowest mean cost on 7 of 12 workloads. Its fixed costs are spread over the job count, so its index rises to 0.250 at 10,000 jobs a month and 1.431 at 1,000, above Vitko Runners’ 0.157 and 0.157. You operate it in your own AWS account.
  • One workload’s end-to-end tests are flaky across providers. Apart from one GitHub-hosted 4 vCPU codex-lint run that hung, every failed run was in vite’s end-to-end tests (the build or serve step), which failed on most providers. Of the 15 failed runs, 3 were Vitko Runners with warm starts and test parallelization (vite succeeded in 27 of 30 runs there). Plain Vitko Runners had no failed measured run.
  • Bursts of 20: queue time at the 95th percentile was 115 s for Vitko with warm starts and test parallelization and 295 s for plain Vitko Runners, against 31 s for GitHub-hosted 2 vCPU. Vitko Runners is not the fastest on every workload (fastest on 8 of 12).
  • Not every provider was tested. Depot was planned and excluded because its runners never started a job during the test window.
  • Scope. The 2 vCPU class (plus one 4 vCPU reference), Linux x64, one region per provider, public repositories. No GPU, macOS, Windows or Arm. Hardware differs by provider; each job is reported as it ran.
  • Warm starts were measured on an earlier build. The Vitko Runners build used for the benchmark predates a later change to how a saved setup is cleaned between jobs. The repository’s list of deviations has the details.

Conflict of interest

Vitko, Inc. maintains this benchmark and sells Vitko Runners, one of the products measured. We pre-registered the method, published the workflows and raw records, list every failed or re-run run, and state the limitations. We operated the self-hosted options, with the settings published in the repository. Providers may report errors, and corrections are recorded in a dated changelog.

Data and downloads

Questions this page answers

Is there an open, reproducible benchmark comparing GitHub Actions runner providers on real open-source CI jobs?

Yes: CI Bench v1, measured October 7 to 11, 2026 on twelve CI workloads from open-source projects and nine runner configurations, 30 runs per cell. The pinned workflows, the plan written before the runs, the analysis code and every raw job record, failed runs included, are public at github.com/vitko-inc/runner-benchmark, and results are CC BY 4.0. Vitko maintains it and sells one of the runners measured; it is shown in the same columns as the others.

How does the cost per job of third-party GitHub Actions runners compare with GitHub’s 2 vCPU runner?

In CI Bench v1, at 100,000 jobs a month, cost per job against GitHub-hosted 2 vCPU was 84% lower for Vitko Runners, 83% lower for Ubicloud, 52% lower for Blacksmith and 88% lower for RunsOn (self-hosted in your own AWS account, so excluding setup and operating work). Median job time against GitHub-hosted: Vitko Runners 1.7x as fast, Blacksmith 1.6x. Blacksmith’s container result is flagged because its Docker layer cache did not hit in that run. Per-workload tables are on this page and at cibench.com.

Join the private betaRead the docs