Comparison · CI Bench v1, measured October 7–11, 2026
GitHub Actions runners compared: time and cost per job
Nine runner configurations, managed and self-hosted, on twelve real CI workloads from popular open-source projects, each against GitHub-hosted 2 vCPU.
In short
Vitko Runners with warm starts and test parallelization was the fastest of the nine configurations: 2.29x faster than GitHub-hosted 2 vCPU at 84% lower cost per job. Plain Vitko Runners was 1.74x faster at 84% lower cost. Among managed runners, Vitko Runners had the lowest cost per job. RunsOn, self-hosted on AWS, had a lower cost per job at 100,000 jobs a month.
- Our own benchmark: Vitko sells one of the products measured. Conflict of interest
Every configuration
Relative to GitHub-hosted 2 vCPU, geometric mean over the twelve workloads, at each provider’s list price. Self-hosted options are priced all-in at 100,000 jobs a month; the last column shows 10,000 jobs a month.
| Runner | Type | Speed vs GitHub 2 vCPU | Cost per job vs GitHub 2 vCPU | Cost at 10,000 jobs a month |
|---|---|---|---|---|
| Vitko Runners with warm starts and test parallelization | Managed | 2.29x faster | 84% lower | 84% lower |
| Vitko RunnersWithout warm starts or test parallelization | Managed | 1.74x faster | 84% lower | 84% lower |
| GitHub-hosted, 4 vCPUA larger machine size | Managed | 1.56x faster | 39% higher | 39% higher |
| Blacksmith, 2 vCPU | Managed | 1.55x faster | 52% lower | 52% lower |
| Google Cloud pool, self-hostedn4d-standard-2, always on, on-demand; all-in at 50% utilization | Self-hosted | 1.50x faster | 59% lower | 59% lower |
| RunsOn on AWS, self-hosted, 2 vCPUDefault settings (spot); all-in | Self-hosted | 1.21x faster | 88% lower | 75% lower |
| AWS EC2, self-hosted, one VM per jobm8a.large, on-demand; all-in | Self-hosted | 1.15x faster | 62% lower | 60% lower |
| Ubicloud standard, 2 vCPU | Managed | 1.14x faster | 83% lower | 83% lower |
| GitHub-hosted, 2 vCPUThe baseline (= 1.00) | Managed | baseline | baseline | baseline |
Which GitHub Actions runner was fastest per job?
Vitko Runners with warm starts and test parallelization: 2.29x faster than GitHub-hosted 2 vCPU, the lowest time index of the nine configurations in CI Bench v1. Plain Vitko Runners was 1.74x faster.
How does the cost per job of third-party runners compare with GitHub's 2 vCPU runner?
At list price, per job, against GitHub-hosted 2 vCPU: Vitko Runners with warm starts and test parallelization 84% lower; Vitko Runners 84% lower; GitHub-hosted, 4 vCPU 39% higher; Blacksmith, 2 vCPU 52% lower; Ubicloud standard, 2 vCPU 83% lower. The lowest-cost managed runner was Vitko Runners.
What about self-hosted runners in my own cloud account?
Self-hosted options are priced all-in: everything their jobs use in the cloud account, plus fixed monthly costs spread over the job count. RunsOn on AWS had the lowest cost index at 100,000 jobs a month (0.121, against 0.157 for Vitko Runners), but 0.250 at 10,000 and 1.431 at 1,000 jobs a month, above Vitko Runners' 0.157 at both volumes. Self-hosted options also need you to operate them.
Is the benchmark open and reproducible?
Yes. The method was pre-registered before the sessions ran, and the workflows, the analysis and every raw job record are public, with instructions to rerun it against any provider. Changes after pre-registration are dated with their reason.
What was measured, and what was not?
Twelve workloads: eleven open-source projects’ own CI jobs at pinned releases and a burst of 20 concurrent runs, in three sessions of ten measured rounds (930 measured runs per configuration). The 2 vCPU class plus one 4 vCPU reference, Linux x64, one region per provider. No GPU, macOS, Windows or Arm. Depot was planned and excluded because its runners never started a job during the test window.
Sources
- Full results, method, limitations and downloads: CI Bench v1
- Workflows, analysis and every raw job record: github.com/vitko-inc/runner-benchmark