Testing boot_recovery produced a guest that cannot boot: boot-repair read
/boot/vmlinuz-linux and initramfs-linux.img on this BIOS/GRUB guest as
"orphaned kernel files shadowing the ESP" and deleted them, leaving the
sandbox at a GRUB prompt. That is a legitimate sandbox outcome -- destroying
the guest is what the tool is for -- but vmbox handled it by hanging.
Two independent unbounded waits, both fixed:
- vm_wait_ssh counted only its `sleep 2` calls, ignoring the up-to-5s
ConnectTimeout each failed probe spends inside ssh. A nominal 150s wait ran
past 400s (measured, then cut short by the harness). It now works to a wall
clock deadline: the same guest is now reported unreachable in 156s.
- _run_recover_rc reboots the stopped guest to read the recorded exit code,
and fell through to an unbounded serial console read: 180s for a prompt
that never comes plus 300s for a command that never runs, all AFTER the
verdict had already been printed. Recovery is best-effort by definition, so
both bounds are now short (60s/30s) and overridable.
serial_exec's timeouts become env-overridable to make that possible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Root cause, confirmed causally rather than inferred: masking
systemd-timesyncd (which vmbox does so `-rtc base=` stays pinned, since
timesyncd otherwise resets the guest clock to real time) left
systemd-time-wait-sync waiting for a sync that can never arrive. That unit sat
in "start running" forever and every other job queued behind it as "start
waiting" -- sshd included -- so the guest never reached multi-user.target. It
booted to a login prompt and looked perfectly healthy while being permanently
unreachable. The fix is one line: mask both units together.
Four earlier hypotheses were each disproven by measurement. What finally
worked was asking the guest instead of reading absence-of-log-lines:
`systemctl is-system-running` returned "starting", and `systemctl list-jobs`
put systemd-time-wait-sync at the head of the queue. Stopping it made sshd
active and bound :22 within seconds, and ssh from the host then succeeded.
Rule for next time: a unit that is enabled, has satisfied dependencies, and
neither starts nor fails is QUEUED -- check list-jobs first.
The smoke gate added earlier did its job here: it caught the unreachable image
and refused to seal it, keeping the sandbox so its logs survived.
Also fixes serial_exec markers to be plain alphanumerics -- shell
metacharacters get echoed and line-wrapped by the tty, which made them
unreliable to match and produced spurious timeouts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
The Arch cloud image leaves sshd enabled but never started -- no start line,
no failure line, dependencies satisfied. Four hypotheses (NTP, cloud-init
clean, a socket-disable, a blocking start) were each disproven by measurement,
so rather than keep paying a ~10 min rebuild per guess this adds the transport
the tool should arguably have used all along.
For the case vmbox exists for, serial beats ssh on the merits: it keeps
printing straight through the poweroff that kills an ssh connection, and it
works when sshd does not. `vm run` now picks ssh when it answers and the
serial console otherwise, and recovers the guest exit code either way. Guests
get a second serial port (ttyS1) on a unix socket with a root autologin, kept
separate from ttyS0 so driving a shell never interleaves with the append-only
log the verdict is read from.
serial_exec.py waits for an explicit per-command end marker rather than a
fixed sleep. That is not a detail: an earlier probe used a 4s window, the next
command was typed while the previous one still ran, and the tty echoed that
input -- which reads exactly like output in a transcript and produced a
confident but wrong conclusion. It also strips ANSI CSI and systemd's OSC 3008
shell-integration blocks, verified against a real captured boot log.
Also: a vmbox-sshd-kick unit that forces sshd up after multi-user.target as
belt and braces, and the smoke-test sandbox renamed to a name that actually
passes validate_vm_name -- "__smoke" was rejected, so the gate silently never
ran.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
