pika-git-1

A new module ci_state.rs is added and registered in main.rs. It defines the enums and structs that represent the lifecycle of a CI lane beyond simple queued/running/passed/failed:

• CiLaneExecutionReason captures why a lane is in its current status. A lane with status = queued might have execution_reason = target_unhealthy to indicate it is deliberately held back.
• CiLaneFailureKind classifies failures into test_failure, timeout, or infrastructure. The classify_ci_failure function inspects log text for patterns like "timed out", "ci runner error", or "infrastructure" to automatically assign the kind.
• CiTargetHealthState and CiTargetHealthSnapshot model per-target health. A target becomes unhealthy after CI_TARGET_HEALTH_INFRA_FAILURE_THRESHOLD (default 2) consecutive infrastructure failures. Once unhealthy, next_target_cooloff_until computes a 15-minute cooloff window.
• configured_target_key_for_lane resolves the target key from lane configuration, preferring pikaci_target_id, then falling back to well-known concurrency groups like apple-host.

All enums implement FromStr for database deserialization and as_str()/label() for serialization and display.

Migration 0019_ci_queue_state_and_target_health.sql makes five categories of schema changes:

1. New columns on branch_ci_run_lanes and nightly_run_lanes: execution_reason (NOT NULL, defaults to 'queued'), failure_kind (nullable), and ci_target_key (nullable).

2. Backfill execution_reason: Already-running lanes get 'running' instead of the default 'queued'.

3. Backfill ci_target_key: Populates the target key from pikaci_target_id if present, otherwise from well-known concurrency group names (apple-host, nightly-android).

4. Composite indexes: Four new indexes on (status, execution_reason, id) and (ci_target_key, status, id) for both lane tables, enabling efficient claim queries and execution reason refresh scans.

5. ci_target_health table: Tracks per-target health with state, consecutive_infra_failure_count, timestamps for last success/failure, the kind of last failure, and a cooloff_until timestamp.

Both BranchCiLaneRecord and NightlyLaneRecord gain four new fields:

• execution_reason: CiLaneExecutionReason — why the lane is in its current state
• failure_kind: Option<CiLaneFailureKind> — classification of the failure if the lane failed
• ci_target_key: Option<String> — the resolved CI target identifier
• target_health: Option<CiTargetHealthSnapshot> — the current health snapshot of the lane's target, hydrated after query

The rerun source structs (BranchLaneRerunSource, NightlyLaneRerunSource) also gain ci_target_key so reruns preserve target assignment.

Every INSERT INTO branch_ci_run_lanes and INSERT INTO nightly_run_lanes statement is updated to:

1. Call configured_target_key_for_lane(lane) to resolve and store ci_target_key at creation time.
2. Include execution_reason with an initial value of 'queued'.

This applies to four code paths:

• Initial lane creation for branch CI suites
• Initial lane creation for nightly runs
• Rerun lane creation for branch CI
• Rerun lane creation for nightly runs

The rerun queries also now SELECT and propagate ci_target_key from the source lane.

Six distinct lane-reset code paths are updated:

1. Cancel branch CI lane → sets execution_reason = 'queued', failure_kind = NULL
2. Cancel all branch CI lanes → same
3. Cancel nightly lane → same
4. Cancel all nightly lanes → same
5. Stale branch lane recovery → sets execution_reason = 'stale_recovered' (a distinct reason to track lanes that were recovered from stale state)
6. Stale nightly lane recovery → same

The lease-lost failure path also clears failure_kind = NULL since lease loss is not a test/infra failure.

The claim_pending_branch_ci_lane_runs and claim_pending_nightly_lane_runs methods now:

1. Load the set of currently unhealthy target IDs via current_unhealthy_target_ids(), which queries ci_target_health for targets that are in unhealthy state with an active cooloff window.
2. For each candidate lane, look up its ci_target_key and skip it if the target is unhealthy.
3. When a lane is claimed (transitioned to running), set execution_reason = 'running' and failure_kind = NULL.

This creates a natural back-pressure mechanism: lanes targeting unhealthy infrastructure stay queued until the cooloff expires or the target recovers.

The finish_branch_ci_lane_run and finish_nightly_lane_run methods are refactored into a two-layer pattern:

1. Public entry point calls classify_ci_failure(log_text) for failed lanes to auto-detect the failure kind, then delegates to finish_*_with_kind.
2. finish_*_with_kind persists failure_kind alongside the status update and calls update_target_health_after_lane_finish.

The update_target_health_after_lane_finish function implements the health state machine:

• Infrastructure failure: Increments consecutive_infra_failure_count. If it reaches the threshold (2), marks the target as unhealthy and sets cooloff_until to 15 minutes in the future.
• Non-infrastructure failure (test failure): Does not count toward unhealthy state, but records the failure.
• Success (passed): Resets consecutive_infra_failure_count to 0 and marks the target as healthy.

The function uses INSERT ... ON CONFLICT ... DO UPDATE (upsert) to handle both new and existing target health records.

Lane listing functions (list_branch_ci_run_lanes, list_nightly_run_lanes) now call hydration functions after loading lanes:

1. hydrate_lane_target_health / hydrate_nightly_lane_target_health: Collect unique ci_target_key values from the loaded lanes, then batch-load their health snapshots.
2. load_ci_target_health_snapshots: Builds a dynamic IN (...) query against ci_target_health and returns a HashMap<String, CiTargetHealthSnapshot>.
3. Each lane's target_health field is populated by looking up its ci_target_key in the map.

The lane SELECT queries are also updated to read the new columns (execution_reason, failure_kind, ci_target_key) and parse them via parse_execution_reason and parse_optional_failure_kind helper functions.

A new refresh_ci_lane_execution_reasons method on Store is called periodically (integrated into ci.rs polling) to keep execution reasons accurate:

1. Running lanes: Forces execution_reason = 'running' for any lane with status = 'running' that somehow has a stale reason.
2. Queued lanes: Iterates all queued lanes in ID order (nightly first, then branch) and assigns the reason based on priority:
   • TargetUnhealthy if the lane's target is in the unhealthy set
   • BlockedByConcurrencyGroup if the lane's concurrency group is occupied
   • WaitingForCapacity if global CI concurrency slots are exhausted
   • StaleRecovered preserved if already set (to maintain audit trail)
   • Queued otherwise (the lane would be picked up on next claim)
3. Simulates claim order: lanes that would be claimed consume available slots and occupy concurrency groups, so downstream lanes get accurate reasons.

This runs within a single IMMEDIATE transaction to provide a consistent snapshot.

Both update_branch_ci_lane_pikaci_ids and update_nightly_lane_pikaci_ids now include ci_target_key = COALESCE(ci_target_key, ?2) in their UPDATE statements. This means:

• If the lane already had a ci_target_key from configuration, it is preserved.
• If it was NULL (no configured target), it gets populated from the pikaci_target_id reported at runtime.

This ensures that even lanes without preconfigured targets participate in target health tracking once they are assigned to a runner.

The CLI's CiLane struct gains six new fields (all #[serde(default)] for backward compatibility with older servers):

• execution_reason, failure_kind, ci_target_key, target_health_state, target_health_summary

Corresponding enums (CiLaneExecutionReason, CiLaneFailureKind, CiTargetHealthState) are defined with display logic.

Rendering is refactored:

• render_branch_status replaces the direct println! calls in print_branch_status with a String-returning function for testability.
• render_lane_status_line builds a rich single-line display: lane_id status [· reason] [· failure=kind] [· health summary] [target run_id].
• render_lane_snapshot_fragment replaces the old active_lane_titles-based snapshot with a detailed id:status:reason:failure:health format, enabling --wait to detect changes in execution reason and health state, not just status.

A comprehensive test branch_status_renders_blocked_unhealthy_and_failure_details validates all rendering paths.

The web layer's lane serialization (used for both branch CI and nightly lane API responses) is updated to include:

• execution_reason — the string representation of the execution reason enum
• failure_kind — optional string classification of the failure
• ci_target_key — the resolved target identifier
• target_health_state — "healthy" or "unhealthy" from the hydrated snapshot
• target_health_summary — a human-readable one-line summary like "target apple-host unhealthy · consecutive infra failures 2 · cooloff until 2026-03-24T00:15:00Z"

This applies to both branch lane and nightly lane serialization paths.

Both branch_ci_live.html and nightly_live.html templates are updated to render the new fields:

• Execution reason badge: Displayed for queued/running lanes when the reason is not the default (e.g., shows "blocked by concurrency group", "waiting for capacity", "target unhealthy").
• Failure kind badge: Shown for failed lanes, classifying the failure as "test failure", "timeout", or "infrastructure".
• Target health indicator: When a lane's target is unhealthy, a warning is displayed with the health summary (including consecutive failure count and cooloff deadline).

These indicators give operators immediate visibility into why lanes are stalled or what kind of failure occurred, directly in the live dashboard.

The CI module's main polling function is updated to call store.refresh_ci_lane_execution_reasons(ci_concurrency) on each tick. This ensures that:

1. Lanes blocked by concurrency groups are correctly labeled even if the blocking lane started between refreshes.
2. Lanes targeting newly-unhealthy targets are immediately flagged.
3. Lanes that were waiting for capacity but now have slots available get their reason updated to queued.
4. The dashboard and CLI --wait snapshots always reflect the latest blocking reasons.

The refresh runs within the same transaction as the rest of the poll tick, maintaining consistency.