Back to feed

sledtools/pika branch #158

jericho

Hard-cut remote staged Jericho fulfillment

Target branch: master

branch: open tutorial: ready ci: success
Open CI Details

Continuous Integration

CI: success

Compact status on the review page, with full logs on the CI page.

Open CI Details

Latest run #197 success

12 passed

head 35325ba17ec705955bdf4c2335bc6aeaf0865f4d · queued 2026-03-31 08:27:00 · 12 lane(s)

queued 17s · ran 9s

check-agent-contracts · success check-pika-rust · success check-notifications · success check-pikachat · success check-pikachat-typescript · success check-pikachat-openclaw-e2e · success check-pika-followup · success check-apple-host-sanity · success check-apple-desktop-compile · success check-apple-ios-compile · success check-fixture · success check-rmp · success

Summary

This branch performs a comprehensive rebrand of the CI system from 'pikaci' to 'jerichoci' (Jericho CI) across the entire codebase, while simultaneously introducing several reliability and correctness improvements: SSH keepalive options for remote connections, retry logic for remote directory syncing, a more forgiving cleanup-failure policy for successful remote Linux VM executions, hard-cut remote staged fulfillment that forces live environment values instead of replaying recorded metadata, database migrations to rename CI lane metadata and structured CI target columns, a new jerichoci_store module, renamed Incus guest roles and image aliases, updated Nix derivations and shell scripts, and refreshed web templates and API endpoints to reflect the new Jericho naming throughout.

Tutorial Steps

Rebrand environment variables from PIKACI to JERICHOCI

Intent: Rename all environment variable constants from the PIKACI_ prefix to JERICHOCI_ so that the executor, run orchestrator, and model layers consistently reference the new Jericho CI identity. This is the largest mechanical change in the branch and touches every configuration knob exposed through the environment.

Affected files: crates/jerichoci/src/executor.rs, crates/jerichoci/src/model.rs, crates/jerichoci/src/run.rs

Evidence
@@ -203,24 +203,24 @@ const TART_BASE_VM_ENV: &str = "JERICHOCI_TART_BASE_VM";
@@ -205,8 +205,8 @@ const REMOTE_LINUX_VM_BACKEND_ENV: &str = "JERICHOCI_REMOTE_LINUX_VM_BACKEND";
@@ -82,36 +82,37 @@ const STAGED_LINUX_RUST_SUBPROCESS_MODE_ENV: &str = "JERICHOCI_PRE_MERGE_PIKA_RUST_SUBPROCESS_FULFILL";

Every PIKACI_* environment variable constant across the executor, model, and run modules has been renamed to JERICHOCI_*. This includes:

  • Tart VM configuration: JERICHOCI_TART_BASE_VM, JERICHOCI_TART_USE_HOST_XCODE, JERICHOCI_TART_XCODE_APP
  • Remote Linux VM settings: JERICHOCI_REMOTE_LINUX_VM_BACKEND, JERICHOCI_REMOTE_LINUX_VM_INCUS_GUEST_ROLE, JERICHOCI_REMOTE_LINUX_VM_INCUS_PROJECT, JERICHOCI_REMOTE_LINUX_VM_INCUS_PROFILE, JERICHOCI_REMOTE_LINUX_VM_INCUS_IMAGE_ALIAS
  • Prepared output fulfillment: JERICHOCI_PREPARED_OUTPUT_FULFILL_SSH_BINARY, JERICHOCI_PREPARED_OUTPUT_FULFILL_SSH_HOST, JERICHOCI_PREPARED_OUTPUT_FULFILL_SSH_REMOTE_WORK_DIR, and many more
  • Host-local nix shell: JERICHOCI_HOST_LOCAL_NIX_SHELL

All log prefixes have also been changed from [pikaci] to [jerichoci] for consistency in log output. The Tart VM name prefix changes from pikaci- to jerichoci-, and the Incus instance name prefix follows suit. Error messages now reference "Jericho CI" instead of "pikaci Wave 1".

Add SSH keepalive options for remote connections

Intent: Prevent long-running SSH sessions from being silently dropped by firewalls or NAT devices by adding ServerAliveInterval and ServerAliveCountMax options to all remote SSH commands.

Affected files: crates/jerichoci/src/executor.rs

Evidence
@@ -229,15 +229,20 @@ const SSH_CONNECT_TIMEOUT_SECS: &str = "5";
@@ -1935,7 +1960,13 @@ fn run_ssh_command(remote_host: &str, command: &str) -> Command {

Three new constants govern SSH session resilience:

const SSH_CONNECT_TIMEOUT_SECS: &str = "5";
const SSH_SERVER_ALIVE_INTERVAL_SECS: &str = "15";
const SSH_SERVER_ALIVE_COUNT_MAX: &str = "4";

The run_ssh_command function now passes -o ServerAliveInterval=15 and -o ServerAliveCountMax=4 alongside the existing ConnectTimeout and BatchMode options. This means the SSH client will send a keepalive probe every 15 seconds and tolerate up to 4 missed responses (60 seconds of silence) before disconnecting. A dedicated unit test remote_ssh_commands_set_keepalive_options verifies the exact argument vector.

Introduce retry logic for remote directory syncing

Intent: Make remote snapshot and payload syncing resilient to transient network or SSH failures by wrapping the single-attempt sync in a retry loop with configurable attempts and delay.

Affected files: crates/jerichoci/src/executor.rs

Evidence
@@ -229,15 +229,20 @@ const REMOTE_SYNC_MAX_ATTEMPTS: usize = 3;
@@ -2164,6 +2195,47 @@ fn sync_directory_to_remote(

Two new constants control retry behavior:

const REMOTE_SYNC_MAX_ATTEMPTS: usize = 3;
const REMOTE_SYNC_RETRY_DELAY: Duration = Duration::from_secs(2);

The existing sync_directory_to_remote function is refactored into a retry wrapper that delegates to the new sync_directory_to_remote_once function. On failure, the wrapper logs the attempt number and error, sleeps for the retry delay, and tries again up to 3 times. The log line includes the remote host and the specific label (e.g., snapshot vs. payload) to aid debugging:

[jerichoci] sync {label} attempt {n}/{max} failed for {host}: {err}; retrying after 2s

If all attempts fail, the last error is propagated.

Tolerate cleanup failures after successful remote VM execution

Intent: Prevent a successful CI job from being reported as failed just because post-execution VM cleanup encountered a transient error, improving the overall reliability of the remote Linux VM backend.

Affected files: crates/jerichoci/src/executor.rs

Evidence
@@ -1057,15 +1062,7 @@ fn run_remote_linux_vm_job(job: &JobSpec, ctx: &HostContext) -> anyhow::Result<J
@@ -1758,6 +1755,32 @@ fn finalize_remote_linux_vm_execution(

Previously, the match in run_remote_linux_vm_job treated (Ok(_outcome), Err(cleanup_err)) as a hard failure, discarding the successful execution result. The new finalize_remote_linux_vm_execution function extracts this logic and changes the behavior:

  • (Ok, Ok) — return the successful outcome (unchanged)
  • (Err, Ok) — return the execution error (unchanged)
  • (Ok, Err)NEW: log the cleanup failure to the host log but return the successful outcome
  • (Err, Err) — return the execution error with cleanup context (unchanged)

The key insight is that once the guest job has succeeded and produced its artifacts, a cleanup failure (e.g., failing to delete the ephemeral VM) is an infrastructure nuisance, not a job failure. A new test successful_remote_linux_vm_execution_ignores_cleanup_failure validates this behavior.

Hard-cut remote staged Linux fulfillment to use live environment

Intent: Force remote staged fulfillment parameters (wrapper program, transport host, remote launcher, remote helper, remote work dir) to always come from the current environment rather than replaying previously-recorded metadata, ensuring that infrastructure changes take effect immediately without requiring re-recording.

Affected files: crates/jerichoci/src/run.rs

Evidence
@@ -294,11 +295,13 @@ fn run_jobs_against_snapshot(
@@ -2101,11 +2110,19 @@ fn resolve_run_prepared_output_invocation_wrapper_program(
@@ -2120,12 +2137,18 @@ fn resolve_run_prepared_output_launcher_transport_mode(
@@ -2133,6 +2156,7 @@ fn resolve_run_prepared_output_launcher_transport_program(
@@ -2207,14 +2250,19 @@ fn resolve_run_prepared_output_launcher_transport_remote_work_dir(

A new predicate jobs_require_remote_staged_linux_fulfillment checks whether any job in the run uses a remote staged Linux execution path. When it returns true, all eight resolve_run_prepared_output_* functions now receive the jobs slice and bypass recorded metadata to resolve values directly from the live environment:

  1. Wrapper program — calls prepared_output_fulfillment_launcher_program() instead of using recorded value
  2. Transport mode — forces SshLauncherTransportV1
  3. Transport program — reads JERICHOCI_PREPARED_OUTPUT_FULFILL_SSH_BINARY or defaults
  4. Transport host — calls prepared_output_ssh_host() directly
  5. Remote launcher program — calls prepared_output_remote_launcher_binary()
  6. Remote helper program — calls prepared_output_remote_helper_binary()
  7. Remote work dir — calls prepared_output_remote_work_dir() directly

This "hard-cut" approach means operators can update infrastructure (e.g., move to a new build host) and have it take effect on the next run without needing to re-record fulfillment metadata.

Rename Incus guest roles and image aliases

Intent: Align the Incus guest role enum variants and default image alias strings with the Jericho CI naming convention.

Affected files: crates/pika-incus-guest-role/src/lib.rs, crates/jerichoci/src/executor.rs, crates/jerichoci/src/executor/incus.rs, crates/jerichoci/src/model.rs, crates/pika-cloud/src/incus.rs, crates/pika-cloud/src/spec.rs, nix/incus/roles.nix

Evidence
@@ -1899,7 +1922,7 @@ fn remote_linux_vm_incus_image_alias(guest_role: IncusGuestRole) -> String {
@@ -651,7 +651,7 @@ guest_role: pika_incus_guest_role::IncusGuestRole::JerichoRunner,

The IncusGuestRole enum variant PikaciRunner is renamed to JerichoRunner throughout the codebase. This affects:

  • The pika-incus-guest-role crate's enum definition and serialization (now serializes as "jericho-runner")
  • All test fixtures and assertions that reference the guest role
  • Default image aliases that shift from pikaci/dev to jericho/dev
  • The Incus run binary path changes from /run/current-system/sw/bin/pikaci-incus-run to /run/current-system/sw/bin/jerichoci-incus-run
  • Error messages that mention valid guest role values
  • The Nix roles definition in nix/incus/roles.nix

Deserialization remains backward-compatible through the role alias on the image record struct, allowing older records with "pikaci-runner" to still be decoded.

Database migrations: rename CI columns

Intent: Rename the database columns that stored CI metadata under the old naming scheme to reflect the Jericho terminology, keeping the schema aligned with the codebase.

Affected files: crates/pika-git/migrations/0027_rename_ci_lane_metadata.sql, crates/pika-git/migrations/0028_rename_structured_ci_target.sql, crates/pika-git/src/ci_store.rs, crates/pika-git/src/ci_state.rs, crates/pika-git/src/ci.rs, crates/pika-git/src/ci_manifest.rs

Evidence
@@ -0,0 +1,2 @@ ALTER TABLE ci_lane_states RENAME COLUMN ci_lane_metadata TO ci_lane_run_metadata;
@@ -0,0 +1,2 @@ ALTER TABLE ci_lane_states RENAME COLUMN structured_ci_target TO ci_target;

Two new SQL migrations are added:

  • 0027: Renames ci_lane_metadataci_lane_run_metadata in the ci_lane_states table
  • 0028: Renames structured_ci_targetci_target in the ci_lane_states table

The corresponding Rust code in ci_store.rs updates all SQL queries to use the new column names. The CiLaneState struct fields and the CiManifest / CiState modules are updated accordingly to use ci_lane_run_metadata and ci_target consistently. The ci_manifest.rs module also renames CiTarget references and the ci.rs module adjusts its lane state construction.

Add jerichoci_store module and update branch store

Intent: Introduce a dedicated store module for Jericho CI state persistence and update the branch store to work with the renamed CI data structures.

Affected files: crates/pika-git/src/jerichoci_store.rs, crates/pika-git/src/branch_store.rs, crates/pika-git/src/storage.rs

Evidence
@@ -0,0 +1 @@ crates/pika-git/src/jerichoci_store.rs

A new jerichoci_store module is added under pika-git/src/ and registered in the crate's module tree via storage.rs. The branch store is updated to reference CI state through the new Jericho-named structures. This module provides the storage layer for Jericho CI run metadata, working alongside the existing ci_store module which handles the renamed columns from the database migrations.

Update Nix derivations and shell scripts for Jericho naming

Intent: Ensure the Nix build system, CI image definitions, and operational shell scripts all reference Jericho CI artifacts and paths consistently.

Affected files: nix/ci/linux-rust.nix, nix/incus/pikaci-image.nix, nix/incus/roles.nix, scripts/incus-role-image.sh, scripts/pikaci-incus-image.sh, scripts/lib/pikaci-tools.sh, scripts/pika-build-run-workspace-deps.sh, scripts/pikaci-run-summary.py, scripts/pikaci-staged-linux-remote.sh, scripts/test_pikaci_tools_json.py, flake.nix, infra/justfile, just/checks.just

Evidence
@@ nix/incus/pikaci-image.nix
@@ scripts/pikaci-staged-linux-remote.sh

The Nix and scripting infrastructure receives matching updates:

  • nix/incus/pikaci-image.nix — Image alias references shift to jericho/dev pattern; the guest image Nix expression is updated
  • nix/incus/roles.nix — Guest role definitions updated from pikaci-runner to jericho-runner
  • nix/ci/linux-rust.nix — CI derivation references updated
  • scripts/pikaci-staged-linux-remote.sh — Remote staging script updated with new paths (/var/tmp/jerichoci-prepared-output) and environment variable names
  • scripts/lib/pikaci-tools.sh — Shared shell library updated for new variable names
  • scripts/pikaci-incus-image.sh — Incus image build/import script references updated
  • flake.nix — Nix flake outputs adjusted for renamed derivations
  • infra/justfile and just/checks.just — Just recipes updated to reference new binary and path names

The .gitignore also adds .jerichoci/ to ignore the new local working directory.

Update web views, templates, and API endpoints

Intent: Ensure the web UI and API layer reflect the Jericho CI naming in templates, route handlers, and test fixtures.

Affected files: crates/pika-git/src/web.rs, crates/pika-git/src/web/api.rs, crates/pika-git/src/web/views.rs, crates/pika-git/src/web/tests.rs, crates/pika-git/src/web/tests/api.rs, crates/pika-git/src/web/tests/live.rs, crates/pika-git/src/web/tests/render.rs, crates/pika-git/src/web/tests/unit.rs, crates/pika-git/templates/branch_ci_live.html, crates/pika-git/templates/nightly_live.html

Evidence
@@ crates/pika-git/src/web/api.rs
@@ crates/pika-git/templates/branch_ci_live.html

The web layer is updated across views, API handlers, and HTML templates:

  • web/api.rs — API response structures use the renamed ci_lane_run_metadata and ci_target fields
  • web/views.rs — View models updated to match the renamed CI state fields
  • branch_ci_live.html and nightly_live.html — Templates reference updated field names for rendering CI lane states
  • Test modules (api.rs, live.rs, render.rs, unit.rs) — All test fixtures and assertions updated to use the new naming. This includes JSON snapshot assertions and rendered HTML checks.

The web tests are particularly important as they serve as integration-level verification that the renamed columns flow correctly from the database through the store layer to the API and rendered views.

Update forge model and pikaci catalog for Jericho terminology

Intent: Align the forge integration model and CI catalog with the Jericho naming, updating target resolution logic and lane definitions.

Affected files: crates/pika-forge-model/src/lib.rs, crates/pikaci/src/ci_catalog.rs, crates/pikaci/src/forge_lanes.rs, crates/pikaci/src/lib.rs, crates/pikaci/src/main.rs, crates/pikaci/src/targets.rs, crates/pika-git/src/forge.rs

Evidence
@@ crates/pikaci/src/targets.rs
@@ crates/pika-forge-model/src/lib.rs

The forge model crate and pikaci orchestrator are updated:

  • pika-forge-model — The CI target struct and its serialization are updated to use the renamed ci_target field name
  • pikaci/src/targets.rs — Target resolution logic references updated field and type names
  • pikaci/src/ci_catalog.rs — CI catalog definitions updated for Jericho naming
  • pikaci/src/forge_lanes.rs — Forge lane construction uses renamed types
  • pikaci/src/main.rs and pikaci/src/lib.rs — Entry points and library root updated
  • pika-git/src/forge.rs — Forge integration store queries use the renamed columns

These changes ensure that when the forge (e.g., GitHub/GitLab integration) triggers CI lanes, the data flows through with consistent Jericho naming from target resolution through to state persistence.

Update pika-cloud and server agent API

Intent: Ensure the cloud infrastructure layer and server-side agent API use updated Incus guest role names and CI structures.

Affected files: crates/pika-cloud/src/incus.rs, crates/pika-cloud/src/spec.rs, crates/pika-server/src/agent_api.rs

Evidence
@@ crates/pika-cloud/src/incus.rs
@@ crates/pika-server/src/agent_api.rs

The cloud and server layers receive targeted updates:

  • pika-cloud/src/incus.rs — Incus VM provisioning references the renamed JerichoRunner guest role and updated image aliases
  • pika-cloud/src/spec.rs — Cloud spec definitions use the new role names
  • pika-server/src/agent_api.rs — The agent API that CI runners call back to is updated to handle the renamed CI metadata fields

These changes ensure that the full lifecycle — from cloud VM provisioning through agent registration and result reporting — uses consistent Jericho naming.

Update ph commands, snapshot module, and documentation

Intent: Align the ph CLI tool, snapshot handling, README, and todo tracking documents with the Jericho rebrand.

Affected files: crates/ph/src/commands.rs, crates/ph/src/tests.rs, crates/jerichoci/src/snapshot.rs, crates/pika-git/README.md, todos/jericho.md, todos/pika-cloud-arbitrary-guest-config-spike.md, todos/pika-cloud.md, todos/pika-git.md, todos/professional-infra.md, .gitignore

Evidence
@@ crates/jerichoci/src/snapshot.rs
@@ .gitignore

Supporting files are updated for completeness:

  • ph/src/commands.rs and ph/src/tests.rs — The ph CLI tool's CI-related commands reference updated types and field names
  • jerichoci/src/snapshot.rs — The snapshot metadata marker file is renamed from pikaci-snapshot.json to jerichoci-snapshot.json, and all path references (temporary directories, remote staging directories) use the jerichoci- prefix
  • pika-git/README.md — Documentation updated to reference Jericho CI
  • Todo documents (jericho.md, pika-cloud.md, pika-git.md, professional-infra.md) — Planning documents updated
  • .gitignore — Adds .jerichoci/ to the ignore list alongside the existing .pikaci/ entry
  • crates/pikahut/tests/guardrails.rs — Guardrail tests updated for the new naming

Diff