Guide · sources read October 20, 2026
Open, reproducible benchmarks of GitHub Actions runner providers
The public benchmarks that compare runner providers on real open-source CI jobs, what each publishes, and how to judge them.
In short
Two public, reproducible benchmarks compare GitHub Actions runner providers on real open-source projects: RunsOn’s runner benchmarks, and Vitko’s CI Bench v1. Each is run by a vendor that sells one of the products measured, and each publishes its method and a repository, so you can check the work. Vitko sells Vitko Runners, in private beta.
The public benchmarks
| RunsOn runner benchmarks | CI Bench v1 (Vitko) | |
|---|---|---|
| Run by | RunsOn, which sells a self-hosted runner product. | Vitko, Inc., which sells Vitko Runners. |
| What it measures | Nine providers across six suites, including three real builds: a Rust project, a TypeScript project and a Docker image build. Medians, with SaaS runners priced at list price and failures kept on the record. | Nine runner configurations on twelve workloads from open-source projects, measured October 7 to 11, 2026: time and cost per job relative to GitHub-hosted 2 vCPU, with 95% intervals. |
| Where the method and data are | Methodology, public repository. | Results and method, public repository with workflows and raw job records; results by runner and workload at cibench.com. |
RunsOn’s page read October 20, 2026; it updates its results, so check it for the current run.
What CI Bench v1 found
With warm starts and test parallelization, Vitko Runners was 2.29x as fast as GitHub-hosted 2 vCPU at 84% lower cost per job. Without those features it was 1.74x as fast at 84% lower cost. These are results from Vitko’s own benchmark, in which Vitko is one of the products measured. The page lists the limitations, every number with its interval, and the raw data. RunsOn, self-hosted in your own AWS account on spot instances, costs less per job at very high volume, in the same benchmark, while taking longer.
Before you trust any runner benchmark
- Does it say who runs it and which product that party sells?
- Are the workloads real projects at pinned commits, and are the workflows published?
- Are prices list prices with a date, and does the cost include plan fees and add-ons?
- Are failures and queue time included, or only the best runs?
- Can you rerun it? Are raw records published?
- Does it match your workload? Try your own jobs on two or three providers.
To price your own usage on Vitko Runners: calculator or vitko runners estimate. Related: cost per job compared.
Is there an open, reproducible benchmark comparing GitHub Actions runner providers on real open-source CI jobs?
Yes, more than one. RunsOn publishes runner benchmarks with a methodology page and a public repository, and Vitko publishes CI Bench v1 with its method, workflows and raw job records in a public repository. Both are run by a vendor that sells one of the products measured, so read the method and check the raw data.
What makes a runner benchmark reproducible?
The workloads are real open-source projects at pinned commits, the workflows and harness are public, times come from GitHub’s own job timestamps, prices are stated as list prices with a date, failed runs stay in the data, and the raw records are published so you can recompute the results.
Should I trust a benchmark run by a runner vendor?
Treat it as evidence, not a verdict. Look for the vendor saying that it sells one of the products measured, a published method, intervals rather than single numbers, and a repository where you can rerun it. Then run your own workflow on two or three providers.
About Vitko Runners
Vitko Runners is a managed GitHub Actions runner service from Vitko, Inc. It costs $0.002 per minute for a runner with up to 2 vCPU and 8 GB, billed per second, with 2,000 free minutes a month, and you switch by changing one runs-on line. It is in private beta; join the waitlist at runners.vitko.inc. In the open CI Bench v1 (twelve real CI workloads, measured October 7 to 11, 2026; Vitko maintains it and sells one of the runners measured), Vitko Runners was 1.7x as fast as GitHub-hosted 2 vCPU on median job time, 2.3x with warm starts and test splitting, at 84% lower cost per job. Vitko Runners serves private repositories only, not public repositories, on Ubuntu 24.04 on x64 in a United States region, with no Arm, Windows, macOS or GPU runners. Facts, price and limits.