Benchmark · Measured October 7–11, 2026 # CI Bench v1 Time and cost per job for GitHub Actions runners, on real open-source CI. Time and cost per jobTwelve CI workloads from popular open-source projects. Lower and further left is better. GitHub-hosted 2 vCPU = 1.0. Warm + parallel: warm starts and test parallelization. ## Key results Faster than GitHub-hosted 2 vCPU2.29xVitko Runners with warm starts and test parallelizationLower cost per job84%Than GitHub-hosted 2 vCPU, with or without those featuresFaster, without either feature1.74xPlain Vitko Runners, at 84% lower cost per job ## About Twelve CI workloads from popular open-source projects, nine runner configurations, 930 measured runs each. The same workflow and commit ran everywhere; only the runner changed. The method, workflows and every raw job record are public. - One self-hosted configuration, RunsOn on AWS, has a lower cost index at high volume. [Limitations](#limitations) - Our own benchmark: Vitko sells one of the products measured. [Conflict of interest](#conflict) ## All nine configurations Relative to GitHub-hosted 2 vCPU (= 1.00), geometric mean over the twelve workloads. Lower is better. Small figures are 95% intervals. Sorted by time index. | Configuration | Time index | Speed vs GitHub 2 vCPU | Cost index | Cost vs GitHub 2 vCPU | Time × cost | | --- | --- | --- | --- | --- | --- | | Vitko Runners with warm starts and test parallelization | 0.4360.423–0.459 | 2.29x faster2.18–2.36x | 0.1580.156–0.160 | 84% lower | 0.069 | | Vitko RunnersWithout warm starts or test parallelization | 0.5740.556–0.602 | 1.74x faster1.66–1.80x | 0.1570.155–0.159 | 84% lower | 0.090 | | GitHub-hosted, 4 vCPUA larger machine size | 0.6420.629–0.669 | 1.56x faster1.50–1.59x | 1.3871.350–1.424 | 39% higher | 0.891 | | Blacksmith, 2 vCPU | 0.6450.606–0.759 | 1.55x faster1.32–1.65x | 0.4820.435–0.537 | 52% lower | 0.311 | | Google Cloud pool, self-hostedn4d-standard-2, always on, on-demand; all-in at 50% utilization | 0.6670.658–0.692 | 1.50x faster1.45–1.52x | 0.4090.386–0.440 | 59% lower | 0.273 | | RunsOn on AWS, self-hosted, 2 vCPUDefault settings (spot); all-in | 0.8280.816–0.859 | 1.21x faster1.16–1.23x | 0.1210.117–0.125 | 88% lower | 0.100 | | AWS EC2, self-hosted, one VM per jobm8a.large, on-demand; all-in | 0.8690.833–0.947 | 1.15x faster1.06–1.20x | 0.3830.358–0.405 | 62% lower | 0.333 | | Ubicloud standard, 2 vCPU | 0.8790.847–0.921 | 1.14x faster1.09–1.18x | 0.1730.167–0.180 | 83% lower | 0.152 | | GitHub-hosted, 2 vCPUThe baseline (= 1.00) | 1.000 | — | 1.000 | — | 1.000 | - With warm starts and test parallelization, Vitko Runners has the lowest time index of the nine configurations. Its interval (0.423–0.459) does not overlap any other configuration’s. - GitHub-hosted 4 vCPU is a larger machine. Its lead over 2 vCPU comes from machine size, not speed per vCPU. - Self-hosted options run in your own cloud account, and you operate them. Their cost depends on volume; Vitko Runners’ indices do not change with volume. - Time × cost multiplies the two indices: lower is better on both together. ## Breakdowns By workload With warm starts and test parallelization, Vitko Runners has the lowest median wall time on 8 of 12 workloads. Blacksmith is fastest on 1 (the burst of 20), plain Vitko Runners on 1 (codex-lint), RunsOn on 1 (container) and the always-on Google Cloud pool on 1 (flask). ### Median wall time per job, seconds Includes waiting for a runner. The lowest in each row is bold. | Workload | Vitko (warm + parallel) | Vitko | GitHub 4 vCPU | Blacksmith | Google Cloud pool | RunsOn | AWS EC2 ephemeral | Ubicloud | GitHub 2 vCPU | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | burst | 131 | 160 | 154 | **97** | 153 | 177 | 161 | 203 | 259 | | cli | **140** | 272 | 192 | 286 | 310 | 327 | 357 | 359 | 368 | | codex-lint | 405 | **404** | 476 | 448 | 581 | 623 | 512 | 682 | 884 | | container | 178 | 179 | 185 | 386 | 178 | **174** | 247 | 213 | 239 | | fd | **45** | 49 | 66 | 65 | 64 | 89 | 114 | 87 | 92 | | flask | 13 | 16 | 21 | 22 | **11** | 38 | 70 | 36 | 23 | | guava | **651** | 655 | 946 | 735 | 820 | 897 | 717 | 881 | 1167 | | monorepo | **172** | 180 | 220 | 177 | 278 | 288 | 281 | 276 | 384 | | pydantic | **69** | 105 | 154 | 103 | 119 | 147 | 154 | 159 | 179 | | ts | **243** | 372 | 435 | 401 | 533 | 570 | 433 | 560 | 784 | | vite | **199** | 350 | 288 | 360 | 401 | 437 | 403 | 502 | 603 | | vue | **32** | 63 | 67 | 76 | 76 | 101 | 117 | 114 | 121 | ### Mean cost per job, US dollars Self-hosted options at 100,000 jobs a month. The lowest in each row is bold. | Workload | Vitko (warm + parallel) | Vitko | GitHub 4 vCPU | Blacksmith | Google Cloud pool | RunsOn | AWS EC2 ephemeral | Ubicloud | GitHub 2 vCPU | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | burst | $0.0520 | $0.0556 | $0.6356 | $0.1656 | $0.1351 | **$0.0511** | $0.1112 | $0.0623 | $0.4998 | | cli | $0.0117 | $0.0097 | $0.0452 | $0.0204 | $0.0173 | **$0.0041** | $0.0139 | $0.0078 | $0.0394 | | codex-lint | $0.0135 | $0.0135 | $0.0989 | $0.0336 | $0.0636 | **$0.0101** | $0.0489 | $0.0149 | $0.0872 | | container | $0.0061 | $0.0060 | $0.0424 | $0.0307 | $0.0532 | $0.0060 | $0.0436 | **$0.0047** | $0.0266 | | fd | **$0.0014** | $0.0016 | $0.0192 | $0.0055 | $0.0035 | $0.0018 | $0.0047 | $0.0021 | $0.0120 | | flask | **$0.0004** | $0.0005 | $0.0120 | $0.0040 | $0.0011 | $0.0014 | $0.0036 | $0.0013 | $0.0060 | | guava | $0.0216 | $0.0218 | $0.1808 | $0.0512 | $0.0449 | **$0.0096** | $0.0318 | $0.0192 | $0.1090 | | monorepo | $0.0056 | $0.0058 | $0.0464 | $0.0140 | $0.0158 | **$0.0039** | $0.0116 | $0.0061 | $0.0396 | | pydantic | **$0.0022** | $0.0034 | $0.0332 | $0.0087 | $0.0065 | $0.0024 | $0.0064 | $0.0037 | $0.0196 | | ts | $0.0274 | $0.0122 | $0.0992 | $0.0299 | $0.0290 | **$0.0061** | $0.0168 | $0.0120 | $0.0774 | | vite | $0.0127 | $0.0115 | $0.0604 | $0.0264 | $0.0231 | **$0.0053** | $0.0163 | $0.0107 | $0.0609 | | vue | **$0.0017** | $0.0020 | $0.0200 | $0.0077 | $0.0044 | $0.0020 | $0.0051 | $0.0027 | $0.0140 | Bursts of 20 Twenty runs dispatched at once, per round. At the 95th percentile, queue time was 115 s with warm starts and test parallelization and 295 s for plain Vitko Runners, against 31 s for GitHub-hosted 2 vCPU and 3 s for the always-on pool. | Configuration | Queue, median (s) | Queue, 95th pct (s) | Wall time, median (s) | | --- | --- | --- | --- | | Vitko Runners with warm starts and test parallelization | 17 | 115 | 131 | | Vitko Runners | 18 | 295 | 160 | | GitHub-hosted, 4 vCPU | 3 | 39 | 154 | | Blacksmith, 2 vCPU | 9 | 16 | 97 | | Google Cloud pool, self-hosted | 2 | 3 | 153 | | RunsOn on AWS, self-hosted, 2 vCPU | 23 | 194 | 177 | | AWS EC2, self-hosted, one VM per job | 65 | 93 | 161 | | Ubicloud standard, 2 vCPU | 18 | 127 | 203 | | GitHub-hosted, 2 vCPU | 3 | 31 | 259 | Cost by volume Self-hosted options carry fixed monthly costs (control plane, license) spread over the job count, so their cost index depends on volume. The results table uses 100,000 jobs a month. | Configuration | 100,000 jobs a month | 10,000 | 1,000 | | --- | --- | --- | --- | | Vitko Runners with warm starts and test parallelization | 0.158 | 0.158 | 0.158 | | Vitko Runners | 0.157 | 0.157 | 0.157 | | GitHub-hosted, 4 vCPU | 1.387 | 1.387 | 1.387 | | Blacksmith, 2 vCPU | 0.482 | 0.482 | 0.482 | | Google Cloud pool, self-hosted | 0.409 | 0.409 | 0.409 | | RunsOn on AWS, self-hosted, 2 vCPU | 0.121 | 0.250 | 1.431 | | AWS EC2, self-hosted, one VM per job | 0.383 | 0.395 | 0.508 | | Ubicloud standard, 2 vCPU | 0.173 | 0.173 | 0.173 | | GitHub-hosted, 2 vCPU | 1.000 | 1.000 | 1.000 | Per session Time index / cost index in each of the three sessions; pooled is computed from all three. | Configuration | Session 1 | Session 2 | Session 3 | Pooled | | --- | --- | --- | --- | --- | | Vitko Runners with warm starts and test parallelization | 0.43 / 0.16 | 0.45 / 0.16 | 0.43 / 0.16 | 0.44 / 0.16 | | Vitko Runners | 0.57 / 0.16 | 0.58 / 0.16 | 0.57 / 0.16 | 0.57 / 0.16 | | GitHub-hosted, 4 vCPU | 0.66 / 1.41 | 0.64 / 1.37 | 0.64 / 1.38 | 0.64 / 1.39 | | Blacksmith, 2 vCPU | 0.77 / 0.54 | 0.66 / 0.47 | 0.60 / 0.43 | 0.64 / 0.48 | | Google Cloud pool, self-hosted | 0.68 / 0.43 | 0.67 / 0.39 | 0.66 / 0.39 | 0.67 / 0.41 | | RunsOn on AWS, self-hosted, 2 vCPU | 0.85 / 0.12 | 0.83 / 0.12 | 0.82 / 0.12 | 0.83 / 0.12 | | AWS EC2, self-hosted, one VM per job | 0.94 / 0.40 | 0.90 / 0.39 | 0.83 / 0.36 | 0.87 / 0.38 | | Ubicloud standard, 2 vCPU | 0.89 / 0.17 | 0.91 / 0.18 | 0.85 / 0.17 | 0.88 / 0.17 | | GitHub-hosted, 2 vCPU | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | 1.00 / 1.00 | ## How we measured - **Workloads.** Eleven open-source projects’ own CI jobs at pinned releases, plus a burst of 20 concurrent runs. The same workflow and commit on every provider; only the runner changed. - **Providers.** Nine configurations, listed above. A provider is included only if its terms allow publishing benchmark results. - **Runs.** Three sessions of 10 measured rounds each, after 2 warm-up rounds: 930 measured runs per configuration. Time comes from GitHub’s own job timestamps and includes waiting for a runner. Failures are counted, never replaced. - **Pre-registered.** The method was fixed before the sessions ran (tag `prereg-v1`), and every later change is dated with its reason. Full method: [METHOD.md](https://github.com/vitko-inc/runner-benchmark/blob/results-v1/METHOD.md).How cost and the indices are calculated - **Cost** is each provider’s public list price for the size used, applied the way that provider bills (Vitko bills per second; GitHub rounds each job up to a minute). Waiting is not billed by anyone. Free minutes and discounts are left out. Test parallelization is billed on the parts only while a split step runs. - **Managed and self-hosted.** Managed providers are priced at list price under their own billing rule. Self-hosted options are priced all-in: everything their jobs use in your cloud account, measured per job, plus fixed monthly costs spread over the job count. List prices, with sources and dates, are in the repository (prices/2026-10-05.json). - **Indices.** Each configuration’s time and cost are divided by GitHub-hosted 2 vCPU’s on the same workload, then combined by geometric mean over the twelve workloads. - **Intervals.** 95% block-bootstrap intervals over sessions and rounds. - **Bursts.** Time is from the first job created to the last job finished. ## Limitations - **RunsOn costs less per job at high volume.** At 100,000 jobs a month, RunsOn (self-hosted on AWS) has the lowest cost index, 0.121 against 0.157 for plain Vitko Runners, and the lowest mean cost on 7 of 12 workloads. Its fixed costs are spread over the job count, so its index rises to 0.250 at 10,000 jobs a month and 1.431 at 1,000, above Vitko Runners’ 0.157 and 0.157. You operate it in your own AWS account. - **One workload’s end-to-end tests are flaky across providers.** Apart from one GitHub-hosted 4 vCPU codex-lint run that hung, every failed run was in vite’s end-to-end tests (the build or serve step), which failed on most providers. Of the 15 failed runs, 3 were Vitko Runners with warm starts and test parallelization (vite succeeded in 27 of 30 runs there). Plain Vitko Runners had no failed measured run. - **Bursts of 20:** queue time at the 95th percentile was 115 s for Vitko with warm starts and test parallelization and 295 s for plain Vitko Runners, against 31 s for GitHub-hosted 2 vCPU. Vitko Runners is not the fastest on every workload (fastest on 8 of 12). - **Not every provider was tested.** Depot was planned and excluded because its runners never started a job during the test window. - **Scope.** The 2 vCPU class (plus one 4 vCPU reference), Linux x64, one region per provider, public repositories. No GPU, macOS, Windows or Arm. Hardware differs by provider; each job is reported as it ran. - **Warm starts were measured on an earlier build.** The Vitko Runners build used for the benchmark predates later cleanup of restored setups between jobs. The repository’s list of deviations has the details. ## Conflict of interest Vitko, Inc. maintains this benchmark and sells Vitko Runners, one of the products measured. We pre-registered the method, published the workflows and raw records, list every failed or re-run run, and state the limitations. We operated the self-hosted options, with the settings published in the repository. Providers may report errors, and corrections are recorded in a dated changelog. ## Data and downloads - Results and write-up: [github.com/vitko-inc/runner-benchmark, results/v1](https://github.com/vitko-inc/runner-benchmark/tree/results-v1/results/v1) (tag `results-v1`). Method, workflows and analysis are in the same [repository](https://github.com/vitko-inc/runner-benchmark). - Downloads: [results.json](https://runners.vitko.inc/benchmarks/data/ci-bench-v1/results.json) (aggregate output), [summary.csv](https://runners.vitko.inc/benchmarks/data/ci-bench-v1/summary.csv) (per workload and configuration) and [runs.csv](https://runners.vitko.inc/benchmarks/data/ci-bench-v1/runs.csv) (every measured run). - License: data under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), code under Apache-2.0. - Cite as: Vitko, Inc. CI Bench v1 (2026). https://runners.vitko.inc/benchmarks/ [Join the private beta](https://runners.vitko.inc/#join)[Read the docs](https://runners.vitko.inc/docs/)