This branch performs a comprehensive rebrand of the CI system from 'pikaci' to 'jerichoci' (Jericho CI) across the entire codebase, while simultaneously introducing several reliability and correctness improvements: SSH keepalive options for remote connections, retry logic for remote directory syncing, a more forgiving cleanup-failure policy for successful remote Linux VM executions, hard-cut remote staged fulfillment that forces live environment values instead of replaying recorded metadata, database migrations to rename CI lane metadata and structured CI target columns, a new jerichoci_store module, renamed Incus guest roles and image aliases, updated Nix derivations and shell scripts, and refreshed web templates and API endpoints to reflect the new Jericho naming throughout.
Tutorial Steps
Rebrand environment variables from PIKACI to JERICHOCI
Intent: Rename all environment variable constants from the PIKACI_ prefix to JERICHOCI_ so that the executor, run orchestrator, and model layers consistently reference the new Jericho CI identity. This is the largest mechanical change in the branch and touches every configuration knob exposed through the environment.
Every PIKACI_* environment variable constant across the executor, model, and run modules has been renamed to JERICHOCI_*. This includes:
Tart VM configuration: JERICHOCI_TART_BASE_VM, JERICHOCI_TART_USE_HOST_XCODE, JERICHOCI_TART_XCODE_APP
Remote Linux VM settings: JERICHOCI_REMOTE_LINUX_VM_BACKEND, JERICHOCI_REMOTE_LINUX_VM_INCUS_GUEST_ROLE, JERICHOCI_REMOTE_LINUX_VM_INCUS_PROJECT, JERICHOCI_REMOTE_LINUX_VM_INCUS_PROFILE, JERICHOCI_REMOTE_LINUX_VM_INCUS_IMAGE_ALIAS
Prepared output fulfillment: JERICHOCI_PREPARED_OUTPUT_FULFILL_SSH_BINARY, JERICHOCI_PREPARED_OUTPUT_FULFILL_SSH_HOST, JERICHOCI_PREPARED_OUTPUT_FULFILL_SSH_REMOTE_WORK_DIR, and many more
All log prefixes have also been changed from [pikaci] to [jerichoci] for consistency in log output. The Tart VM name prefix changes from pikaci- to jerichoci-, and the Incus instance name prefix follows suit. Error messages now reference "Jericho CI" instead of "pikaci Wave 1".
Add SSH keepalive options for remote connections
Intent: Prevent long-running SSH sessions from being silently dropped by firewalls or NAT devices by adding ServerAliveInterval and ServerAliveCountMax options to all remote SSH commands.
The run_ssh_command function now passes -o ServerAliveInterval=15 and -o ServerAliveCountMax=4 alongside the existing ConnectTimeout and BatchMode options. This means the SSH client will send a keepalive probe every 15 seconds and tolerate up to 4 missed responses (60 seconds of silence) before disconnecting. A dedicated unit test remote_ssh_commands_set_keepalive_options verifies the exact argument vector.
Introduce retry logic for remote directory syncing
Intent: Make remote snapshot and payload syncing resilient to transient network or SSH failures by wrapping the single-attempt sync in a retry loop with configurable attempts and delay.
The existing sync_directory_to_remote function is refactored into a retry wrapper that delegates to the new sync_directory_to_remote_once function. On failure, the wrapper logs the attempt number and error, sleeps for the retry delay, and tries again up to 3 times. The log line includes the remote host and the specific label (e.g., snapshot vs. payload) to aid debugging:
[jerichoci] sync {label} attempt {n}/{max} failed for {host}: {err}; retrying after 2s
If all attempts fail, the last error is propagated.
Tolerate cleanup failures after successful remote VM execution
Intent: Prevent a successful CI job from being reported as failed just because post-execution VM cleanup encountered a transient error, improving the overall reliability of the remote Linux VM backend.
Previously, the match in run_remote_linux_vm_job treated (Ok(_outcome), Err(cleanup_err)) as a hard failure, discarding the successful execution result. The new finalize_remote_linux_vm_execution function extracts this logic and changes the behavior:
(Ok, Ok) — return the successful outcome (unchanged)
(Err, Ok) — return the execution error (unchanged)
(Ok, Err) — NEW: log the cleanup failure to the host log but return the successful outcome
(Err, Err) — return the execution error with cleanup context (unchanged)
The key insight is that once the guest job has succeeded and produced its artifacts, a cleanup failure (e.g., failing to delete the ephemeral VM) is an infrastructure nuisance, not a job failure. A new test successful_remote_linux_vm_execution_ignores_cleanup_failure validates this behavior.
Hard-cut remote staged Linux fulfillment to use live environment
Intent: Force remote staged fulfillment parameters (wrapper program, transport host, remote launcher, remote helper, remote work dir) to always come from the current environment rather than replaying previously-recorded metadata, ensuring that infrastructure changes take effect immediately without requiring re-recording.
A new predicate jobs_require_remote_staged_linux_fulfillment checks whether any job in the run uses a remote staged Linux execution path. When it returns true, all eight resolve_run_prepared_output_* functions now receive the jobs slice and bypass recorded metadata to resolve values directly from the live environment:
Wrapper program — calls prepared_output_fulfillment_launcher_program() instead of using recorded value
Transport mode — forces SshLauncherTransportV1
Transport program — reads JERICHOCI_PREPARED_OUTPUT_FULFILL_SSH_BINARY or defaults
Transport host — calls prepared_output_ssh_host() directly
Remote launcher program — calls prepared_output_remote_launcher_binary()
Remote helper program — calls prepared_output_remote_helper_binary()
Remote work dir — calls prepared_output_remote_work_dir() directly
This "hard-cut" approach means operators can update infrastructure (e.g., move to a new build host) and have it take effect on the next run without needing to re-record fulfillment metadata.
Rename Incus guest roles and image aliases
Intent: Align the Incus guest role enum variants and default image alias strings with the Jericho CI naming convention.
The IncusGuestRole enum variant PikaciRunner is renamed to JerichoRunner throughout the codebase. This affects:
The pika-incus-guest-role crate's enum definition and serialization (now serializes as "jericho-runner")
All test fixtures and assertions that reference the guest role
Default image aliases that shift from pikaci/dev to jericho/dev
The Incus run binary path changes from /run/current-system/sw/bin/pikaci-incus-run to /run/current-system/sw/bin/jerichoci-incus-run
Error messages that mention valid guest role values
The Nix roles definition in nix/incus/roles.nix
Deserialization remains backward-compatible through the role alias on the image record struct, allowing older records with "pikaci-runner" to still be decoded.
Database migrations: rename CI columns
Intent: Rename the database columns that stored CI metadata under the old naming scheme to reflect the Jericho terminology, keeping the schema aligned with the codebase.
@@ -0,0 +1,2 @@ ALTER TABLE ci_lane_states RENAME COLUMN ci_lane_metadata TO ci_lane_run_metadata;
@@ -0,0 +1,2 @@ ALTER TABLE ci_lane_states RENAME COLUMN structured_ci_target TO ci_target;
Two new SQL migrations are added:
0027: Renames ci_lane_metadata → ci_lane_run_metadata in the ci_lane_states table
0028: Renames structured_ci_target → ci_target in the ci_lane_states table
The corresponding Rust code in ci_store.rs updates all SQL queries to use the new column names. The CiLaneState struct fields and the CiManifest / CiState modules are updated accordingly to use ci_lane_run_metadata and ci_target consistently. The ci_manifest.rs module also renames CiTarget references and the ci.rs module adjusts its lane state construction.
Add jerichoci_store module and update branch store
Intent: Introduce a dedicated store module for Jericho CI state persistence and update the branch store to work with the renamed CI data structures.
A new jerichoci_store module is added under pika-git/src/ and registered in the crate's module tree via storage.rs. The branch store is updated to reference CI state through the new Jericho-named structures. This module provides the storage layer for Jericho CI run metadata, working alongside the existing ci_store module which handles the renamed columns from the database migrations.
Update Nix derivations and shell scripts for Jericho naming
Intent: Ensure the Nix build system, CI image definitions, and operational shell scripts all reference Jericho CI artifacts and paths consistently.
The Nix and scripting infrastructure receives matching updates:
nix/incus/pikaci-image.nix — Image alias references shift to jericho/dev pattern; the guest image Nix expression is updated
nix/incus/roles.nix — Guest role definitions updated from pikaci-runner to jericho-runner
nix/ci/linux-rust.nix — CI derivation references updated
scripts/pikaci-staged-linux-remote.sh — Remote staging script updated with new paths (/var/tmp/jerichoci-prepared-output) and environment variable names
scripts/lib/pikaci-tools.sh — Shared shell library updated for new variable names
The web layer is updated across views, API handlers, and HTML templates:
web/api.rs — API response structures use the renamed ci_lane_run_metadata and ci_target fields
web/views.rs — View models updated to match the renamed CI state fields
branch_ci_live.html and nightly_live.html — Templates reference updated field names for rendering CI lane states
Test modules (api.rs, live.rs, render.rs, unit.rs) — All test fixtures and assertions updated to use the new naming. This includes JSON snapshot assertions and rendered HTML checks.
The web tests are particularly important as they serve as integration-level verification that the renamed columns flow correctly from the database through the store layer to the API and rendered views.
Update forge model and pikaci catalog for Jericho terminology
Intent: Align the forge integration model and CI catalog with the Jericho naming, updating target resolution logic and lane definitions.
The forge model crate and pikaci orchestrator are updated:
pika-forge-model — The CI target struct and its serialization are updated to use the renamed ci_target field name
pikaci/src/targets.rs — Target resolution logic references updated field and type names
pikaci/src/ci_catalog.rs — CI catalog definitions updated for Jericho naming
pikaci/src/forge_lanes.rs — Forge lane construction uses renamed types
pikaci/src/main.rs and pikaci/src/lib.rs — Entry points and library root updated
pika-git/src/forge.rs — Forge integration store queries use the renamed columns
These changes ensure that when the forge (e.g., GitHub/GitLab integration) triggers CI lanes, the data flows through with consistent Jericho naming from target resolution through to state persistence.
Update pika-cloud and server agent API
Intent: Ensure the cloud infrastructure layer and server-side agent API use updated Incus guest role names and CI structures.
The cloud and server layers receive targeted updates:
pika-cloud/src/incus.rs — Incus VM provisioning references the renamed JerichoRunner guest role and updated image aliases
pika-cloud/src/spec.rs — Cloud spec definitions use the new role names
pika-server/src/agent_api.rs — The agent API that CI runners call back to is updated to handle the renamed CI metadata fields
These changes ensure that the full lifecycle — from cloud VM provisioning through agent registration and result reporting — uses consistent Jericho naming.
Update ph commands, snapshot module, and documentation
Intent: Align the ph CLI tool, snapshot handling, README, and todo tracking documents with the Jericho rebrand.
ph/src/commands.rs and ph/src/tests.rs — The ph CLI tool's CI-related commands reference updated types and field names
jerichoci/src/snapshot.rs — The snapshot metadata marker file is renamed from pikaci-snapshot.json to jerichoci-snapshot.json, and all path references (temporary directories, remote staging directories) use the jerichoci- prefix
pika-git/README.md — Documentation updated to reference Jericho CI
Todo documents (jericho.md, pika-cloud.md, pika-git.md, professional-infra.md) — Planning documents updated
.gitignore — Adds .jerichoci/ to the ignore list alongside the existing .pikaci/ entry
crates/pikahut/tests/guardrails.rs — Guardrail tests updated for the new naming