Back to feed

sledtools/pika branch #155

mac-ci-todos

Add mac ci roadmap

Target branch: master

Merge Commit: 4d915de2b97ca469e64a9ebe652b6e77f5a12126

branch: merged tutorial: ready ci: success
Open CI Details

Continuous Integration

CI: success

Compact status on the review page, with full logs on the CI page.

Open CI Details

Latest run #193 success

head 64897470a00a9792e9825197ad57196b46cd4e16 · queued 2026-03-28 12:31:14 · 0 lane(s)

queued 0s

No lanes were selected for this branch head.

Summary

This branch introduces a comprehensive macOS CI roadmap document (todos/mac-ci.md) to the sledtools/pika repository. The document captures the current state of Apple CI — which recently gained compile-only pre-merge lanes, desktop/iOS split selection, and improved Nix integration — and lays out a structured plan to reach parity with Linux CI and eventually surpass it. The roadmap covers six parity workstreams (compile concurrency, cache formalization, sccache validation, Xcode cost reduction, operational visibility, and worker model hardening), four beyond-parity initiatives (burst capacity, VM isolation, selector precision, and compile heuristics), a recommended execution sequence, and concrete open design questions.

Tutorial Steps

Establish the living Mac CI roadmap document

Intent: Create a single authoritative document that records what the recent Apple CI improvements achieved, what gaps remain relative to Linux CI, and the full plan to close and eventually exceed those gaps.

Affected files: todos/mac-ci.md

Evidence 
@@ -0,0 +1,327 @@
+# Mac CI
+
+This is the living plan for getting macOS CI from "good enough" to "as strong as Linux CI", and then past it.

A new file todos/mac-ci.md is added at the repository root under the todos/ directory. It is framed as a living plan, signaling that it is expected to be updated as work progresses. The document opens by acknowledging the concrete wins already deployed.

Document the current Apple CI state and recent wins

Intent: Provide a clear baseline so that anyone reading the roadmap understands what has already been accomplished and what the current lane design looks like.

Affected files: todos/mac-ci.md

Evidence 
@@ -0,0 +1,327 @@
+- pre-merge Apple is now compile-only
+- pre-merge Apple is split into desktop and iOS lanes
+- Apple compile work now leans much harder on Nix
+- Apple host assets are pinned
+- host-local iOS build reuse is much better
@@ -0,0 +1,327 @@
+- `check-apple-desktop-compile`
+- `check-apple-ios-compile`

The document lists five recent improvements and names the two concrete pre-merge lanes (check-apple-desktop-compile, check-apple-ios-compile). It also enumerates what already works well — unrelated branches skip Apple, and desktop/iOS changes are properly scoped — before listing three remaining weaknesses: single-slot compile throughput, dependence on one remote host with mutable caches, and lack of Linux-grade operational clarity.

Articulate the Linux CI advantages Apple must match

Intent: Frame the parity gap precisely so that subsequent workstreams are well-motivated and scoped.

Affected files: todos/mac-ci.md

Evidence 
@@ -0,0 +1,327 @@
+1. Better cache model
+...
+2. Better horizontal scaling
+...
+3. Better operational simplicity
+...
+4. Better failure isolation

Four dimensions are identified where Linux CI remains ahead:

  1. Cache model — Linux is closer to derivation-first CI; Apple still relies on hybrid Nix + host-local Xcode mutable state.
  2. Horizontal scaling — Linux can fan out independent jobs; Apple serializes on one compile slot.
  3. Operational simplicity — Linux avoids Xcode, simulator runtimes, and DerivedData.
  4. Failure isolation — Linux lanes map more cleanly onto hermetic prepare/run boundaries; Apple has shared host-local state that creates confusing reuse or contention.

Define the six parity workstreams

Intent: Break the path to Linux parity into concrete, sequenceable workstreams with clear target end-states.

Affected files: todos/mac-ci.md

Evidence 
@@ -0,0 +1,327 @@
+### 1. Increase Apple compile concurrency safely
+...
+### 2. Formalize Apple cache layers
+...
+### 3. Add stronger Rust cache reuse on Apple
+...
+### 4. Reduce the remaining Xcode-native cost
+...
+### 5. Improve Apple operational visibility
+...
+### 6. Harden the Apple worker model

Six numbered workstreams make up the parity plan:

  1. Compile concurrency — Split the apple-compile forge concurrency group and remote lock into separate desktop and iOS scopes so both can run in parallel.
  2. Cache formalization — Document and enforce a three-tier cache stack: Nix-managed immutable layers, Apple-managed host assets (Xcode, simulator runtimes), and host-local mutable caches (DerivedData, xcresult).
  3. sccache validation — Prove that sccache (already on the Apple shell path) is materially helping expensive Rust rebuilds, measure hit rates, and persist state properly.
  4. Xcode cost reduction — Profile xcodebuild timing, remove avoidable script phases from compile-only lanes, and keep compile lanes ruthlessly compile-only.
  5. Operational visibility — Surface cache hit/miss, lock contention, prepare vs. bundle duration, and stale state indicators in the forge UI and ph CLI.
  6. Worker model hardening — Move from one fragile Mac host toward a clean lifecycle/scheduling model before adding parallel capacity.

Outline the beyond-parity ambitions

Intent: Capture the stretch goals that could make Mac CI *better* than Linux CI, while clearly marking them as non-blocking on parity.

Affected files: todos/mac-ci.md

Evidence 
@@ -0,0 +1,327 @@
+### 1. Opportunistic Apple burst capacity
+...
+### 2. VM-backed Apple isolation with preserved warm caches
+...
+### 3. Better selector precision than Linux
+...
+### 4. Apple-specific compile heuristics

Four beyond-parity initiatives are described:

  1. Burst capacity — Use a rented always-on Mac as baseline plus a volunteer MacBook Pro runner during working hours, letting daytime Apple queue time potentially beat Linux.
  2. VM-backed isolation — Run Apple runtime jobs in Tart (or similar) VMs with shared persistent cache layers to increase parallelism and isolate flaky stateful work.
  3. Selector precision — Push desktop-only vs. iOS-only scoping even tighter so Apple CI triggers fewer unnecessary lanes than some Linux lanes.
  4. Compile heuristics — Skip unnecessary assembly steps, reuse prebuilt intermediates, and apply lane-specific shortcuts keyed by exact change surface.

Propose the recommended execution sequence

Intent: Give the team a concrete ordering that respects dependencies and avoids premature investment in parallelism before cache and compile paths are solid.

Affected files: todos/mac-ci.md

Evidence 
@@ -0,0 +1,327 @@
+1. Keep the deployed forge and checked-in `ph` in sync
+...
+2. Split Apple compile throughput
+...
+3. Measure real cache performance
+...
+4. Profile remaining Xcode cost
+...
+5. Design the next Apple capacity model
+...
+6. Add optional Apple parallelism beyond one host

The recommended sequence is:

  1. Keep forge deploy and ph CLI in sync to avoid stale client/server mismatches.
  2. Split compile throughput (concurrency groups + remote locks).
  3. Measure real cache performance (sccache hit rates, warm vs. cold xcframework and DerivedData times).
  4. Profile remaining Xcode cost with timing summaries and cut always-run work.
  5. Design the next capacity model (rented Mac mini vs. volunteer laptop burst).
  6. Add optional parallelism beyond one host (Tart VMs), only after host/cache boundaries are explicit.

This ordering ensures cache correctness and compile-path efficiency are proven before capacity is scaled out.

Record concrete open design questions

Intent: Capture the key architectural decisions that still need team input so they are not lost or deferred indefinitely.

Affected files: todos/mac-ci.md

Evidence 
@@ -0,0 +1,327 @@
+- Should desktop and iOS compile lanes get separate forge concurrency groups immediately?
+- Should desktop and iOS compile lanes share one `CARGO_TARGET_DIR`, or should concurrent mode split them?
+- Should Apple compile lanes stay host-native forever, with only runtime lanes moving into VMs?
+- Should the long-term Apple baseline be a rented `64 GiB` mini, the new laptop as a volunteer runner, or both?
+- How much of the Apple Rust layer should be pushed further into dedicated Darwin Nix derivations?

Five explicit open questions are recorded:

  • Whether to split forge concurrency groups immediately.
  • Whether concurrent desktop/iOS compiles should share or split CARGO_TARGET_DIR.
  • Whether compile lanes stay host-native forever while only runtime lanes move to VMs.
  • Whether the long-term baseline is a rented 64 GiB Mac mini, a volunteer laptop, or both.
  • How much of the Apple Rust layer should migrate into dedicated Darwin Nix derivations.

These questions are designed to be answerable independently, letting the team make incremental decisions as each workstream matures.

Diff