Testing boot_recovery produced a guest that cannot boot: boot-repair read
/boot/vmlinuz-linux and initramfs-linux.img on this BIOS/GRUB guest as
"orphaned kernel files shadowing the ESP" and deleted them, leaving the
sandbox at a GRUB prompt. That is a legitimate sandbox outcome -- destroying
the guest is what the tool is for -- but vmbox handled it by hanging.
Two independent unbounded waits, both fixed:
- vm_wait_ssh counted only its `sleep 2` calls, ignoring the up-to-5s
ConnectTimeout each failed probe spends inside ssh. A nominal 150s wait ran
past 400s (measured, then cut short by the harness). It now works to a wall
clock deadline: the same guest is now reported unreachable in 156s.
- _run_recover_rc reboots the stopped guest to read the recorded exit code,
and fell through to an unbounded serial console read: 180s for a prompt
that never comes plus 300s for a command that never runs, all AFTER the
verdict had already been printed. Recovery is best-effort by definition, so
both bounds are now short (60s/30s) and overridable.
serial_exec's timeouts become env-overridable to make that possible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
The image ships Xorg + i3 on tty1 precisely so locker and i3 tests can run,
but `vm run` executes through `sh -c`, which reads no profile. Every X tool
therefore landed with no DISPLAY and no XAUTHORITY and failed with
"Could not determine i3 socket path" -- indistinguishable from "this sandbox
has no X server". testsAndMisc's i3-configuration installer failed that way;
with the environment resolved it exits 0 and i3-msg reports {"success":true}.
XAUTHORITY is the awkward half: startx generates /tmp/serverauth.XXXXXXXX
with a fresh random suffix every boot, so it cannot be hardcoded. Resolve it
from the running Xorg's own -auth argument, falling back to a newest-first
glob.
The `[ -f ]` guard on the source line is load-bearing, not defensive
dressing. In POSIX sh, `.` on a missing file is a SPECIAL BUILTIN error that
exits the shell outright, and `|| true` does NOT rescue it -- written that
way, every `vm run` against a base image predating this snippet silently
produced no output at all. Found by doing exactly that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
`_run_exec "$name" "$@"` sat on its own line under `set -euo pipefail`, so a
non-zero return aborted the script before the verdict was ever computed. ssh
returns 255 when its connection dies -- which is exactly what a SUCCESSFUL
guest poweroff causes. The single case this tool exists to report on was the
one case that silently printed nothing.
Caught by the acceptance demo, not by the unit tests: verdict_report was
correct in isolation the whole time and passed 8/8 there. Only running the
real CLI against a real poweroff exposed the wiring.
Verified after the fix, end to end:
==> Running in 'fixtest': sudo systemctl poweroff
Connection to 127.0.0.1 closed by remote host.
ok VERDICT: clean poweroff -- guest powered itself off and reached
'reboot: Power down'
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
The Arch cloud image leaves sshd enabled but never started -- no start line,
no failure line, dependencies satisfied. Four hypotheses (NTP, cloud-init
clean, a socket-disable, a blocking start) were each disproven by measurement,
so rather than keep paying a ~10 min rebuild per guess this adds the transport
the tool should arguably have used all along.
For the case vmbox exists for, serial beats ssh on the merits: it keeps
printing straight through the poweroff that kills an ssh connection, and it
works when sshd does not. `vm run` now picks ssh when it answers and the
serial console otherwise, and recovers the guest exit code either way. Guests
get a second serial port (ttyS1) on a unix socket with a root autologin, kept
separate from ttyS0 so driving a shell never interleaves with the append-only
log the verdict is read from.
serial_exec.py waits for an explicit per-command end marker rather than a
fixed sleep. That is not a detail: an earlier probe used a 4s window, the next
command was typed while the previous one still ran, and the tty echoed that
input -- which reads exactly like output in a transcript and produced a
confident but wrong conclusion. It also strips ANSI CSI and systemd's OSC 3008
shell-integration blocks, verified against a real captured boot log.
Also: a vmbox-sshd-kick unit that forces sshd up after multi-user.target as
belt and braces, and the smoke-test sandbox renamed to a name that actually
passes validate_vm_name -- "__smoke" was rejected, so the gate silently never
ran.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Much of testsAndMisc/linux_configuration cannot be tested on the host: 15
files use `chattr +i` (so a bad test is not simply revertable), the shutdown
installers write units that power the machine off, and boot_recovery installs
pacman hooks. The repo's own CI says as much -- its Ubuntu job runs only the
side-effect-free tests, noting the rest "drive pacman, /etc, systemd or adb".
vmbox runs them in a throwaway Arch guest and verifies the result from the
HOST, which is the part that is not obvious: a successful shutdown kills the
connection carrying the report, and ssh returns 255 for a clean poweroff, a
crash and a dropped link alike. So the verdict comes from two artefacts that
outlive the VM -- a QMP event log written by a recorder attached at launch
(attach later and qemu has already exited), and the serial console log.
Five outcomes are distinguished: clean poweroff (SHUTDOWN guest:true plus a
serial tail reaching "reboot: Power down"), dirty stop, guest-reset (rebooted
instead of powering off), GUEST_PANICKED, and hung. A POWERDOWN event proves
nothing -- it means the request was delivered, not that anything acted on it.
Reset is a qcow2 backing-file trick: the golden image is built once and sealed
(chmod 444 plus a sha256 sidecar verified on every launch, since booting the
base would silently invalidate every overlay), and each sandbox is a thin
overlay, so reset is rm + recreate. No root anywhere -- user-mode networking
with per-VM forwarded ssh ports, and a socket multicast segment for VM-to-VM.
The guest clock is pinned and re-applied on every launch, because the shutdown
installers gate on a 21:00-05:00 window; NTP is masked in the image or it
silently undoes the pin. Repos are cloned from a read-only share scoped to an
explicit directory, never $HOME, and credentials are sentinel values.
`vm build` ends by booting a throwaway sandbox off the sealed image and
refusing to ship if ssh does not answer. Several distinct faults produced an
identical "boots fine, connection reset" symptom; checks inside the build VM
only prove state at seal time, so the gate has to be an end-to-end boot.
Tests: 8 bats cases over the verdict table and name/meta/serial handling;
shellcheck and ruff clean; every file within the 250-line cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
