Benchmark: GitHub-hosted macOS runner vs the doublespeed fleet

Two benchmarks: a hello-world app (measures overhead) and, further down, Wikipedia iOS (a real app with 367 tests, compared with Namespace's published numbers).

The same iOS app, the same commit, the same xcodebuild test on an iPhone 16 Pro simulator with Xcode 16.4 — run two ways on every push to doublespeed-main/mac-ci-benchmark:

• github-mac — a GitHub-hosted macos-15 runner runs xcodebuild natively.
• doublespeed-mac — an ubuntu-latest runner runs ds xcode test .; the Mac work happens on our fleet and the results come back as JSON.

Measured on 2026-08-27, six rounds, both workflows dispatched together each round. Timings come from the GitHub API (run.created_at → job completed_at), so they include waiting for a runner and runner boot — the time between pushing and seeing the check go green. Raw data: benchmark.json.

Results

Per round (push → job complete):

On the fleet side, what the 37–57 s is made of (from the job's timings): source sync and API round-trips from the Linux runner ~10–15 s, queue wait 0.7–2.1 s, xcodebuild test 19–34 s, Mac slot held 22–37 s in total. The first round was the slowest on both sides (cold caches on ours; nothing to warm on GitHub's, each run is a fresh VM).

Real app: Wikipedia iOS

The hello-world run above measures overhead. This one measures a real, mid-size app: wikimedia/wikipedia-ios — the same project Namespace benchmarks — scheme Wikipedia, 367 unit tests, run on an iPhone 16 simulator. Everything is pinned to commit 84f87f1 (the last one that builds with Xcode 16.4) so both of our columns use the identical toolchain. Workflows: wikipedia-github-mac.yml and wikipedia-doublespeed.yml; the xcodebuild invocation mirrors Wikipedia's own CI.

"Repeat build" means the worker already holds this org's unchanged source blobs and Swift package clones in its per-tenant cache (added during this benchmark, 2026-08-27) — the normal case for CI, where each push changes a handful of files. The first-ever build of a 4,700-file repo pays ~150 s to materialise the source; after that, only changed files are fetched.

What this shows

• On a real app, 4.0× faster than GitHub-hosted macOS runners push-to-green (2m 15s vs 9m 06s, medians of 4 and 6 runs), and ~6× cheaper per run. On a cold first build, 1.9× faster.
• Namespace's published cached number for the same repository is 3m 18s; ours is 2m 15s on smaller hardware. We don't claim a multiplier against it. Caveats, honestly: their runs are on Xcode 26 at a newer commit and include their full workflow; ours are Xcode 16.4 at an older commit measured from GitHub's own timestamps. Same shape, not identical inputs. The raw xcodebuild step is where hardware matters (their M4/M5 Max would out-compile our base M4); the difference is that our job starts on a warm machine with the source already there.
• Compile time itself is ~108 s on the Mini (one run of four took 155 s) vs 461–646 s on GitHub's 3-core M1 — the rest of GitHub's time is VM boot, simulator boot and cold caches. The GitHub workflow uses no actions/cache; caching there saves the Swift package clone, not the boot or the compile. GitHub's M2 Pro XL runner ($0.10/min) was not measured.

What the benchmark found and fixed

Running a 4,700-file repository through the pipeline surfaced five bugs, all fixed the same day: symlinked headers were dropped from snapshots; a forced -configuration broke the scheme's test host; one-in-4,600 transient storage errors failed uploads and downloads outright; the CLI stopped following a job that had been requeued; and (methodology) CODE_SIGNING_ALLOWED=NO, harmless for archives, disables app-group entitlements and made Wikipedia's tests hang for an hour on GitHub. The per-tenant blob and Swift package caches came out of the same work.

Reproduce

gh repo clone doublespeed-main/mac-ci-benchmark && cd mac-ci-benchmark
gh secret set DS_API_KEY
gh workflow run wikipedia-doublespeed.yml && gh workflow run wikipedia-github-mac.yml

Run wikipedia-doublespeed.yml twice: the first run fills the cache, the second is the repeat-build number.

Cost per run (list prices)

GitHub bills macOS at 10× the Linux rate and rounds each job up to the next minute. The fleet side is $0.05 per Mac-minute, metered per second of Mac time actually held (GET /v1/xcode/usage), with no charge while queued — about 15× cheaper per run here.

Hardware

What this does and doesn't show

• The app is tiny (one SwiftUI view, two XCTest cases). Compilation is a few seconds on either machine, so the gap is almost entirely overhead: a GitHub macOS run boots a fresh VM, then a fresh simulator, then warms Xcode from nothing, every time. Our workers keep Xcode and the simulator runtime resident between jobs, and only the changed files are uploaded on each push.
• A bigger app narrows the ratio (compile time is the same order on both, the M4 is faster per core) but not the absolute ~4 minutes of overhead on the GitHub side, which you pay on every push.
• Our fleet is small today; when it is saturated, jobs queue in FIFO order and the wait shows up in timings.queued_ms, which is reported on every job so you can see it.
• Six rounds is a small sample. The repo's scripts/benchmark.mjs recomputes this table from the last N runs, and the workflows run on every push, so the numbers keep accumulating.

Reproduce it

gh repo clone doublespeed-main/mac-ci-benchmark && cd mac-ci-benchmark
gh secret set DS_API_KEY            # your key
gh workflow run github-mac.yml && gh workflow run doublespeed-mac.yml
node scripts/benchmark.mjs --runs 5 --json results.json

The doublespeed-mac workflow is ten lines; it is the same one shown in the docs.