Back to feed

sledtools/pika branch #35

pika-orch-incus-cleanup-2

pikaci: simplify Incus guest runtime contract

Target branch: master

branch: closed tutorial: ready ci: failed
Open CI Details

Continuous Integration

CI: failed

Compact status on the review page, with full logs on the CI page.

Open CI Details

Latest run #44 failed

9 passed 1 failed

head 8ccfbb4ea089c2f4fc72090f085dd241076a2400 · queued 2026-03-24 16:22:27 · 10 lane(s)

queued 11s · ran 8m 50s

check-pika-rust · success check-pika-followup · success check-notifications · success check-agent-contracts · success check-rmp · success check-pikachat · success check-pikachat-typescript · success check-apple-host-sanity · failed check-pikachat-openclaw-e2e · success check-fixture · success

Summary

This branch simplifies the Incus guest runtime contract in pikaci by eliminating the dual-mode (transfer vs. single-host-shared) execution model in favor of a single shared-mount path. The RemoteLinuxVmIncusMode enum, its environment variable, and all transfer-mode codepaths (nix-store closure import, snapshot push, guest filesystem preparation scripts, workspace finalize logic) are removed from executor.rs. The guest bootstrap script that was previously synthesized per-job in Rust is moved into the Incus NixOS image as a proper pikaci-incus-run binary at /run/current-system/sw/bin/pikaci-incus-run, which reads its configuration from environment variables (PIKACI_INCUS_GUEST_COMMAND, PIKACI_INCUS_TIMEOUT_SECS, PIKACI_INCUS_RUN_AS_ROOT) passed via incus exec env. Mount paths are renamed from /mnt/pikaci-shared-* to their final guest paths (/workspace/snapshot, /staged/linux-rust/*), eliminating the symlink indirection the old prepare script created. The migration plan documentation is updated to reflect the simplified single-path architecture and remove transfer-mode timing data.

Tutorial Steps

Remove the RemoteLinuxVmIncusMode enum and mode selection logic

Intent: Eliminate the dual-mode abstraction (transfer vs. single-host-shared) so the Incus backend has exactly one execution path. This removes the mode enum, its string conversion, the environment variable parsing function, the mode field from RemoteLinuxVmContext, and the mode-dependent branching throughout the executor.

Affected files: crates/pikaci/src/executor.rs

Evidence
@@ -81,7 +81,6 @@ struct GuestRunnerConfig {
 #[derive(Clone, Debug)]
 struct RemoteLinuxVmContext {
     backend: RemoteLinuxVmBackend,
-    incus_mode: RemoteLinuxVmIncusMode,
@@ -97,22 +96,6 @@ struct RemoteLinuxVmContext {
-    remote_incus_closure_dir: PathBuf,
-}
-
-#[derive(Clone, Copy, Debug, Eq, PartialEq)]
-enum RemoteLinuxVmIncusMode {
-    Transfer,
-    SingleHostShared,
-}
@@ -2128,10 +2110,6 @@
-fn remote_linux_vm_incus_mode_label(mode: RemoteLinuxVmIncusMode) -> &'static str {
-    mode.as_str()
-}
@@ -2261,20 +2233,6 @@
-fn remote_linux_vm_incus_mode() -> anyhow::Result<RemoteLinuxVmIncusMode> {

The RemoteLinuxVmIncusMode enum (Transfer / SingleHostShared) and all supporting infrastructure are deleted:

  • The incus_mode field is removed from RemoteLinuxVmContext.
  • The remote_incus_closure_dir field (only used in transfer mode) is removed.
  • The PIKACI_REMOTE_LINUX_VM_INCUS_MODE environment variable constant and its default are deleted.
  • The remote_linux_vm_incus_mode() parsing function and remote_linux_vm_incus_mode_label() display helper are removed.
  • In remote_linux_vm_context(), the backend-conditional mode selection is replaced by direct construction without a mode field.

This collapses a two-branch runtime decision into a single unconditional path, significantly reducing the number of match arms and conditional blocks throughout the executor.

Simplify ensure_remote_incus_runtime to a single execution path

Intent: Remove mode-conditional branching from the Incus runtime setup sequence. Previously, transfer mode required closure imports, guest filesystem preparation, snapshot staging, and workspace finalization, while single-host-shared mode only needed artifact reset and device configuration. Now only the shared-mount path remains.

Affected files: crates/pikaci/src/executor.rs

Evidence
@@ -2941,14 +2904,11 @@ fn ensure_remote_incus_runtime(
-    if remote.incus_mode == RemoteLinuxVmIncusMode::SingleHostShared {
-        reset_remote_linux_vm_artifacts(remote, log_path)?;
-    }
+    reset_remote_linux_vm_artifacts(remote, log_path)?;
@@ -2973,9 +2933,7 @@
-    if remote.incus_mode == RemoteLinuxVmIncusMode::SingleHostShared {
-        configure_remote_incus_single_host_shared_devices(job, remote, log_path)?;
-    }
+    configure_remote_incus_devices(job, remote, log_path)?;
@@ -2987,18 +2945,7 @@
-    wait_for_remote_incus_instance(remote, log_path)?;
-    match remote.incus_mode {
-        RemoteLinuxVmIncusMode::Transfer => {
-            import_remote_incus_closures(remote, log_path)?;
-            prepare_remote_incus_guest_filesystem(job, remote, log_path)?;
-            stage_snapshot_into_incus_guest(remote, log_path)?;
-            finalize_remote_incus_transfer_workspace(job, remote, log_path)
-        }
-        RemoteLinuxVmIncusMode::SingleHostShared => {
-            prepare_remote_incus_guest_filesystem(job, remote, log_path)
-        }
-    }
+    wait_for_remote_incus_instance(remote, log_path)

The ensure_remote_incus_runtime function is simplified from a multi-branch orchestration to a linear sequence:

  1. Artifact directories are always reset (previously conditional on single-host-shared mode).
  2. Device configuration (configure_remote_incus_devices, renamed from configure_remote_incus_single_host_shared_devices) is always called.
  3. The function now returns after wait_for_remote_incus_instance—it no longer performs any guest-internal filesystem preparation.

The entire post-boot guest setup (closure imports, prepare scripts, snapshot push, workspace finalization) is eliminated because the guest image now owns that contract.

Delete transfer-mode codepaths: closure import, prepare script, snapshot push, finalize

Intent: Remove approximately 280 lines of transfer-mode-specific functions that are no longer reachable after the mode enum deletion. These functions handled importing Nix store closures into the guest, generating per-job bash prepare scripts, pushing snapshot files via `incus file push`, and remounting the workspace read-only.

Affected files: crates/pikaci/src/executor.rs

Evidence
@@ -3152,293 +3093,13 @@
-fn build_remote_incus_run_command(
-fn build_remote_incus_prepare_script(
-fn build_remote_incus_transfer_workspace_finalize_command(
-fn import_remote_path_closure_into_incus(
-fn import_remote_incus_closures(
-fn prepare_remote_incus_guest_filesystem(
-fn stage_snapshot_into_incus_guest(
-fn finalize_remote_incus_transfer_workspace(

The following functions are entirely removed:

  • build_remote_incus_run_command — generated the runuser/timeout wrapper for the guest command
  • build_remote_incus_prepare_script — generated a multi-hundred-line bash script that set up the guest filesystem layout, created symlinks, set ownership, and embedded the run script at /usr/local/bin/pikaci-incus-run
  • build_remote_incus_transfer_workspace_finalize_command — generated mount --bind / mv commands for read-only workspace remounting
  • import_remote_path_closure_into_incus — ran nix-store --export | incus exec -- nix-store --import to push Nix closures
  • import_remote_incus_closures — orchestrated closure import for workspace-deps and workspace-build
  • prepare_remote_incus_guest_filesystem — resolved store paths and executed the prepare script inside the guest
  • stage_snapshot_into_incus_guest — used incus file push -r to copy the snapshot tree
  • finalize_remote_incus_transfer_workspace — executed the workspace finalize command

This is the largest single deletion in the branch and represents the core simplification.

Move guest bootstrap logic into the Incus NixOS image

Intent: Relocate the guest job bootstrap contract from dynamically-generated bash in executor.rs to a proper NixOS package (`pikaci-incus-run`) baked into the Incus guest image. The script reads its configuration from environment variables rather than being templated per-job.

Affected files: nix/incus/pikaci-image.nix

Evidence
@@ -1,5 +1,87 @@
+let
+  pikaciIncusRun = pkgs.writeShellScriptBin "pikaci-incus-run" ''
+    set -euo pipefail
+    export PATH="/run/current-system/sw/bin:$PATH"
+    : "''${PIKACI_INCUS_GUEST_COMMAND:?missing PIKACI_INCUS_GUEST_COMMAND}"
+    : "''${PIKACI_INCUS_TIMEOUT_SECS:?missing PIKACI_INCUS_TIMEOUT_SECS}"
+    : "''${PIKACI_INCUS_RUN_AS_ROOT:?missing PIKACI_INCUS_RUN_AS_ROOT}"
@@ -83,6 +165,7 @@
+    pikaciIncusRun

A new pikaciIncusRun derivation is added to nix/incus/pikaci-image.nix using pkgs.writeShellScriptBin. This script:

  1. Requires three environment variables: PIKACI_INCUS_GUEST_COMMAND, PIKACI_INCUS_TIMEOUT_SECS, and PIKACI_INCUS_RUN_AS_ROOT (fails immediately with :? if any are missing).
  2. Owns directory setup: uses a defensive ensure_owned_dir helper that attempts chown but gracefully degrades on read-only mounts.
  3. Sets the standard environment: CARGO_HOME, CARGO_TARGET_DIR, XDG_CACHE_HOME, SSL_CERT_FILE, etc.
  4. Conditionally detects the host Nix store mount: exports PIKACI_STAGED_HOST_NIX_STORE_ROOT only if /mnt/pikaci-nix-store exists.
  5. Runs the guest command either directly or via runuser -u pikaci -m depending on PIKACI_INCUS_RUN_AS_ROOT.
  6. Writes /artifacts/result.json with status, exit code, timestamp, and message.

The package is added to environment.systemPackages, making it available at /run/current-system/sw/bin/pikaci-incus-run.

Pass guest command configuration via incus exec environment variables

Intent: Update the Incus launch command builder to pass the job's guest command, timeout, and root-execution flag as environment variables through `incus exec ... env VAR=value`, pointing at the image-resident binary instead of a dynamically-created script.

Affected files: crates/pikaci/src/executor.rs

Evidence
@@ -2815,7 +2772,9 @@
-fn build_remote_incus_launch_command(remote: &RemoteLinuxVmContext) -> String {
+fn build_remote_incus_launch_command(job: &JobSpec, remote: &RemoteLinuxVmContext) -> String {
+    let (guest_command, run_as_root) = compiled_guest_command(job);
+    let run_as_root_value = if run_as_root { "1" } else { "0" };
@@ -2824,7 +2783,11 @@
-                "/usr/local/bin/pikaci-incus-run",
+                "env",
+                &format!("PIKACI_INCUS_GUEST_COMMAND={guest_command}"),
+                &format!("PIKACI_INCUS_TIMEOUT_SECS={}", job.timeout_secs),
+                &format!("PIKACI_INCUS_RUN_AS_ROOT={run_as_root_value}"),
+                REMOTE_LINUX_VM_INCUS_RUN_BINARY,

build_remote_incus_launch_command now accepts a &JobSpec parameter alongside the remote context. It computes the guest command and root flag, then constructs an incus exec invocation that prepends env to set:

  • PIKACI_INCUS_GUEST_COMMAND — the compiled shell command string
  • PIKACI_INCUS_TIMEOUT_SECS — the job timeout
  • PIKACI_INCUS_RUN_AS_ROOT1 or 0

The target binary is now the constant REMOTE_LINUX_VM_INCUS_RUN_BINARY (/run/current-system/sw/bin/pikaci-incus-run) instead of the old /usr/local/bin/pikaci-incus-run path. The call site in spawn_remote_linux_vm_process is updated to pass job to the function.

Rename mount path constants from shared-prefixed to final guest paths

Intent: Align the host-side mount path constants with the actual guest filesystem layout. The old `SHARED_*` prefix was an artifact of the dual-mode design. The new paths use the actual mount targets (`/workspace/snapshot`, `/staged/linux-rust/*`) that the guest image expects.

Affected files: crates/pikaci/src/executor.rs

Evidence
@@ -238,16 +221,15 @@
-const REMOTE_LINUX_VM_INCUS_SHARED_SNAPSHOT_MOUNT_PATH: &str = "/mnt/pikaci-snapshot";
+const REMOTE_LINUX_VM_INCUS_RUN_BINARY: &str = "/run/current-system/sw/bin/pikaci-incus-run";
 const REMOTE_LINUX_VM_INCUS_SHARED_NIX_STORE_MOUNT_PATH: &str = "/mnt/pikaci-nix-store";
-const REMOTE_LINUX_VM_INCUS_SHARED_WORKSPACE_DEPS_MOUNT_PATH: &str = "/mnt/pikaci-workspace-deps";
-const REMOTE_LINUX_VM_INCUS_SHARED_WORKSPACE_BUILD_MOUNT_PATH: &str = "/mnt/pikaci-workspace-build";
+const REMOTE_LINUX_VM_INCUS_SNAPSHOT_MOUNT_PATH: &str = "/workspace/snapshot";
+const REMOTE_LINUX_VM_INCUS_WORKSPACE_DEPS_MOUNT_PATH: &str = "/staged/linux-rust/workspace-deps";
+const REMOTE_LINUX_VM_INCUS_WORKSPACE_BUILD_MOUNT_PATH: &str = "/staged/linux-rust/workspace-build";

Four constant renames reflect the simplified mount topology:

Old constantNew constantOld valueNew value
REMOTE_LINUX_VM_INCUS_SHARED_SNAPSHOT_MOUNT_PATHREMOTE_LINUX_VM_INCUS_SNAPSHOT_MOUNT_PATH/mnt/pikaci-snapshot/workspace/snapshot
REMOTE_LINUX_VM_INCUS_SHARED_WORKSPACE_DEPS_MOUNT_PATHREMOTE_LINUX_VM_INCUS_WORKSPACE_DEPS_MOUNT_PATH/mnt/pikaci-workspace-deps/staged/linux-rust/workspace-deps
REMOTE_LINUX_VM_INCUS_SHARED_WORKSPACE_BUILD_MOUNT_PATHREMOTE_LINUX_VM_INCUS_WORKSPACE_BUILD_MOUNT_PATH/mnt/pikaci-workspace-build/staged/linux-rust/workspace-build
(new)REMOTE_LINUX_VM_INCUS_RUN_BINARY/run/current-system/sw/bin/pikaci-incus-run

The old paths went to /mnt/* intermediaries that the prepare script symlinked into final locations. Now the virtiofs mounts go directly to the paths the guest expects, eliminating the symlink layer.

Add pre-created staging directories to the Incus image tmpfiles rules

Intent: Ensure the guest image creates the workspace-deps and workspace-build directories at boot so that virtiofs mounts have valid targets even before any job runs.

Affected files: nix/incus/pikaci-image.nix

Evidence
@@ -93,6 +176,8 @@
+    "d /staged/linux-rust/workspace-deps 0755 root root -"
+    "d /staged/linux-rust/workspace-build 0755 root root -"

Two systemd.tmpfiles.rules entries are added to guarantee /staged/linux-rust/workspace-deps and /staged/linux-rust/workspace-build exist at boot time. This is necessary because the virtiofs disk devices are mounted at these paths—without pre-existing directories, the mount would fail. The directories are root-owned since they will hold read-only content.

Rename configure_remote_incus_single_host_shared_devices to configure_remote_incus_devices

Intent: Drop the mode qualifier from the device configuration function name since there is no longer an alternative mode. Simplify the error message for writable workspace rejection to remove the mode label.

Affected files: crates/pikaci/src/executor.rs

Evidence
@@ -3103,29 +3050,23 @@
-fn configure_remote_incus_single_host_shared_devices(
+fn configure_remote_incus_devices(
@@ -3103,29 +3050,23 @@
-        bail!(
-            "remote Linux VM backend `incus` mode `{}` does not support writable workspace jobs",
-            remote_linux_vm_incus_mode_label(remote.incus_mode)
-        );
+        bail!("remote Linux VM backend `incus` does not support writable workspace jobs");

The function configure_remote_incus_single_host_shared_devices is renamed to configure_remote_incus_devices. The writable-workspace error message is simplified from a mode-parameterized string to a static message. All disk device additions (pikaci-snapshot, pikaci-nix-store, pikaci-workspace-deps, pikaci-workspace-build) now use the renamed mount path constants.

Remove one directory from ensure_remote_linux_vm_directories

Intent: Stop creating the remote Incus closure directory since nix-store closure import is no longer performed.

Affected files: crates/pikaci/src/executor.rs

Evidence
@@ -2354,7 +2312,7 @@
-            "mkdir -p {} {} {} {} {} {} {}; ",
+            "mkdir -p {} {} {} {} {} {}; ",
@@ -2364,7 +2322,6 @@
-        shell_single_quote(&remote.remote_incus_closure_dir.display().to_string()),

The mkdir -p command in ensure_remote_linux_vm_directories drops the remote_incus_closure_dir path from the list of directories to create. This directory was used to store guest-store-paths.json during transfer-mode closure imports and is no longer needed.

Simplify and consolidate test infrastructure

Intent: Replace verbose inline RemoteLinuxVmContext construction in every test with shared helper functions (`sample_remote_context` and `sample_shell_job`), remove all transfer-mode-specific tests, and update remaining test assertions to match the new launch command shape.

Affected files: crates/pikaci/src/executor.rs

Evidence
@@ -4108,36 +3763,6 @@
-    fn with_incus_mode_env<T>(value: Option<&str>, action: impl FnOnce() -> T) -> T {
@@ -4151,6 +3776,44 @@
+    fn sample_shell_job(command: &'static str) -> JobSpec {
@@ -4151,6 +3776,44 @@
+    fn sample_remote_context(backend: RemoteLinuxVmBackend) -> RemoteLinuxVmContext {
@@ -4334,39 +3973,21 @@
-        let command = build_remote_incus_launch_command(&RemoteLinuxVmContext {
+        let command = build_remote_incus_launch_command(
+            &sample_shell_job("actionlint"),
+            &sample_remote_context(RemoteLinuxVmBackend::Incus),
+        );
@@ -4334,39 +3973,21 @@
+        assert!(command.contains("'env'"));
+        assert!(command.contains("PIKACI_INCUS_GUEST_COMMAND=bash --noprofile --norc -lc"));
+        assert!(command.contains("actionlint"));
+        assert!(command.contains("'PIKACI_INCUS_TIMEOUT_SECS=120'"));
+        assert!(command.contains("'PIKACI_INCUS_RUN_AS_ROOT=0'"));
+        assert!(command.contains("'/run/current-system/sw/bin/pikaci-incus-run'"));

The test module undergoes substantial cleanup:

Removed tests (no longer applicable):

  • remote_linux_incus_transfer_finalize_remounts_read_only_workspace
  • remote_linux_incus_single_host_prepare_script_avoids_snapshot_staging_copy
  • remote_linux_incus_transfer_prepare_script_keeps_guest_store_paths
  • remote_linux_incus_mode_env_accepts_single_host_shared
  • remote_linux_incus_mode_defaults_to_single_host_shared

Removed helper: with_incus_mode_env (environment variable no longer exists).

Added helpers: sample_shell_job() and sample_remote_context() provide reusable test fixtures, replacing 20+ line inline struct literals repeated across tests.

Updated tests:

  • remote_linux_incus_launch_uses_incus_exec_runner now asserts on env, PIKACI_INCUS_GUEST_COMMAND, PIKACI_INCUS_TIMEOUT_SECS, PIKACI_INCUS_RUN_AS_ROOT, and the new binary path.
  • remote_linux_incus_read_only_disk_device_uses_virtiofs_bus (renamed from *_single_host_shared_*) uses the helper and updated mount path constant.
  • ensure_remote_linux_vm_directories_skips_existing_staged_output_symlinks drops the incus_mode and remote_incus_closure_dir fields.

Update migration plan documentation

Intent: Reflect the architectural simplification in the design document: remove references to transfer mode, update the backend shape description, document the new guest bootstrap contract, and remove transfer-mode timing data.

Affected files: docs/incus-migration-plan.md

Evidence
@@ -858,24 +858,26 @@
-- transfer the workspace snapshot
+- mount the prepared workspace snapshot
@@ -858,24 +858,26 @@
-- `transfer` remains an explicit fallback via `PIKACI_REMOTE_LINUX_VM_INCUS_MODE=transfer`
+- the steady-state backend shape is now one shared-mount Incus path rather than a mode split
@@ -858,24 +858,26 @@
+- the guest bootstrap contract now lives in the Incus image:
+  the image owns the mounted-path layout and
+  `/run/current-system/sw/bin/pikaci-incus-run` owns the guest env/log/result contract
@@ -889,15 +891,11 @@
-  - `pika-actionlint` transfer Incus: about `155s` wall
-  - `pika-doc-contracts` transfer Incus: about `67s` wall

The migration plan is updated to match the new architecture:

  • "transfer the workspace snapshot" becomes "mount the prepared workspace snapshot"
  • The transfer-mode fallback documentation is replaced with a description of the single shared-mount path
  • A new bullet documents the guest bootstrap contract: the image owns the path layout and pikaci-incus-run owns the env/log/result contract
  • Transfer-mode timing data (pika-actionlint transfer at ~155s, pika-doc-contracts transfer at ~67s) is removed since the mode no longer exists
  • The RMP parity statement changes from "both fast-path Incus and transfer fallback" to "the default Incus path"

Diff