shutdown_verdict_demo.sh exercises every branch of the verdict table against a
real guest -- clean poweroff, reboot-instead-of-poweroff, kernel panic, and a
command that leaves the machine running. The failure branches matter as much
as the success one: a verifier that has only ever returned "pass" is untested.
destructive_demo.sh is the acceptance test for the tool as a whole: destroy
/etc and /usr in the guest, assert the guest really is broken, reset with one
command, assert it is pristine again, then assert the HOST is untouched
(/etc/hosts attributes unchanged, base image hash unchanged, /etc and /usr
still present).
Measured on the first real run, which confirms the design's central claim:
QMP reported SHUTDOWN {"guest": true, "reason": "guest-shutdown"} and the
serial log reached "reboot: Power down", so the verdict was "clean poweroff"
with exit 0 -- while ssh saw only "Connection closed by remote host", which
would have been indistinguishable from a crash or a dropped link.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Root cause, confirmed causally rather than inferred: masking
systemd-timesyncd (which vmbox does so `-rtc base=` stays pinned, since
timesyncd otherwise resets the guest clock to real time) left
systemd-time-wait-sync waiting for a sync that can never arrive. That unit sat
in "start running" forever and every other job queued behind it as "start
waiting" -- sshd included -- so the guest never reached multi-user.target. It
booted to a login prompt and looked perfectly healthy while being permanently
unreachable. The fix is one line: mask both units together.
Four earlier hypotheses were each disproven by measurement. What finally
worked was asking the guest instead of reading absence-of-log-lines:
`systemctl is-system-running` returned "starting", and `systemctl list-jobs`
put systemd-time-wait-sync at the head of the queue. Stopping it made sshd
active and bound :22 within seconds, and ssh from the host then succeeded.
Rule for next time: a unit that is enabled, has satisfied dependencies, and
neither starts nor fails is QUEUED -- check list-jobs first.
The smoke gate added earlier did its job here: it caught the unreachable image
and refused to seal it, keeping the sandbox so its logs survived.
Also fixes serial_exec markers to be plain alphanumerics -- shell
metacharacters get echoed and line-wrapped by the tty, which made them
unreliable to match and produced spurious timeouts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Committed by accident: running systemd-ssh-generator --help during
investigation made it treat '--help' as its output path and write generator
units there. Not part of vmbox.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
The Arch cloud image leaves sshd enabled but never started -- no start line,
no failure line, dependencies satisfied. Four hypotheses (NTP, cloud-init
clean, a socket-disable, a blocking start) were each disproven by measurement,
so rather than keep paying a ~10 min rebuild per guess this adds the transport
the tool should arguably have used all along.
For the case vmbox exists for, serial beats ssh on the merits: it keeps
printing straight through the poweroff that kills an ssh connection, and it
works when sshd does not. `vm run` now picks ssh when it answers and the
serial console otherwise, and recovers the guest exit code either way. Guests
get a second serial port (ttyS1) on a unix socket with a root autologin, kept
separate from ttyS0 so driving a shell never interleaves with the append-only
log the verdict is read from.
serial_exec.py waits for an explicit per-command end marker rather than a
fixed sleep. That is not a detail: an earlier probe used a 4s window, the next
command was typed while the previous one still ran, and the tty echoed that
input -- which reads exactly like output in a transcript and produced a
confident but wrong conclusion. It also strips ANSI CSI and systemd's OSC 3008
shell-integration blocks, verified against a real captured boot log.
Also: a vmbox-sshd-kick unit that forces sshd up after multi-user.target as
belt and braces, and the smoke-test sandbox renamed to a name that actually
passes validate_vm_name -- "__smoke" was rejected, so the gate silently never
ran.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Much of testsAndMisc/linux_configuration cannot be tested on the host: 15
files use `chattr +i` (so a bad test is not simply revertable), the shutdown
installers write units that power the machine off, and boot_recovery installs
pacman hooks. The repo's own CI says as much -- its Ubuntu job runs only the
side-effect-free tests, noting the rest "drive pacman, /etc, systemd or adb".
vmbox runs them in a throwaway Arch guest and verifies the result from the
HOST, which is the part that is not obvious: a successful shutdown kills the
connection carrying the report, and ssh returns 255 for a clean poweroff, a
crash and a dropped link alike. So the verdict comes from two artefacts that
outlive the VM -- a QMP event log written by a recorder attached at launch
(attach later and qemu has already exited), and the serial console log.
Five outcomes are distinguished: clean poweroff (SHUTDOWN guest:true plus a
serial tail reaching "reboot: Power down"), dirty stop, guest-reset (rebooted
instead of powering off), GUEST_PANICKED, and hung. A POWERDOWN event proves
nothing -- it means the request was delivered, not that anything acted on it.
Reset is a qcow2 backing-file trick: the golden image is built once and sealed
(chmod 444 plus a sha256 sidecar verified on every launch, since booting the
base would silently invalidate every overlay), and each sandbox is a thin
overlay, so reset is rm + recreate. No root anywhere -- user-mode networking
with per-VM forwarded ssh ports, and a socket multicast segment for VM-to-VM.
The guest clock is pinned and re-applied on every launch, because the shutdown
installers gate on a 21:00-05:00 window; NTP is masked in the image or it
silently undoes the pin. Repos are cloned from a read-only share scoped to an
explicit directory, never $HOME, and credentials are sentinel values.
`vm build` ends by booting a throwaway sandbox off the sealed image and
refusing to ship if ssh does not answer. Several distinct faults produced an
identical "boots fine, connection reset" symptom; checks inside the build VM
only prove state at seal time, so the gate has to be an end-to-end boot.
Tests: 8 bats cases over the verdict table and name/meta/serial handling;
shellcheck and ruff clean; every file within the 250-line cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
