The golden image logs in with agetty+startx and has no RGB controllers,
no NVIDIA card and no sound card, so the night-lockdown scripts — which
mask/stop lightdm.service and getty@.service and shell out to openrgb,
nvidia-smi and amixer — had nothing real to act on in a sandbox and
were being "verified" on the host instead.
- `vm lightdm <name>`: installs lightdm + lightdm-gtk-greeter into the
live overlay (pacman -Syu, not -S: the overlay's package database is
the image's and mirrors have rotated those versions out), configures
autologin -> i3 exactly like the host, turns tty1 back into a plain
getty, reboots the guest and proves lightdm.service is active with an
Xorg it owns. Never rebuilds the base: `vm build --force` re-seals a
new checksum while every existing sandbox still backs onto the old
file's name, which is silent qcow2 corruption.
- `vm shim <name> <tool> [exit[:N]|hang|sleep:S]`: a fake tool ahead of
the real one on PATH, every call logged with an epoch to
/var/log/vmbox-shim/<tool>.log so the host can assert what ran and in
what order. `hang` is the openrgb 1.0-2 that never returned on
2026-09-12 and defeated the curfew.
- The DISPLAY/XAUTHORITY resolver `vm run` sources becomes its own file,
guest/vmbox-x11.sh, installed by the image build and re-installed by
the lightdm switch; it now tests readability, not existence, and tries
~/.Xauthority before the startx serverauth glob, because lightdm's
Xorg -auth file is root-only.
- README: the worked example uses MIDNIGHT_SHUTDOWN_CONFIRM=y (the
installer refuses to run unattended now; `echo y |` no longer works),
and the "i2c RGB stays on the host" line is gone.
- bats: four host-side tests for shim.sh (exit, exit:N, hang, refusals).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015mLnUeoJ3mUwGRTowTUH6d
The previous commit skipped the poweroff settle when `ssh_rc != 255`, on the
reasoning that a successful poweroff kills the ssh connection and ssh reports
255. That holds for the ssh transport and for nothing else. `_run_transport`
falls back to the serial console whenever ssh does not answer -- and the
README calls serial "the transport of record for shutdown tests" -- where the
exit code comes from serial_exec.py and is 0 even while the machine is already
on its way down.
Result, reproduced: `VMBOX_TRANSPORT=serial vm run <vm> 'sudo systemctl
poweroff'` reported "still running" for a guest that had cleanly powered off,
with SHUTDOWN{guest:true} in events.jsonl and "reboot: Power down" in the
serial log. The one thing this tool exists to get right, silently wrong. My
five verdict re-tests all went through ssh, so none of them caught it.
Grepping the serial log for shutdown markers instead does not work either:
they land a second or two after the command returns, so the check loses the
race. The fix waits a short grace period (default 3s) for any sign of a stop
before committing to the 30s wait. Warm runs go from 0s to 3s; that is the
honest price of a correct verdict.
Re-tested: clean poweroff, reboot and panic on BOTH transports, plus
tests/destructive_demo.sh 7/7.
Also, per "a fix that needs a human to remember a flag is not a fix": the X
session wait is now automatic on the branch that actually boots the guest --
the only one that can race tty1's autologin -- instead of requiring
VMBOX_X_WAIT. Warm runs never reach that branch. The env var remains an
override. README numbers corrected to the measured 14s/3s and ~100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
`vm run` took 41s cold / 30s warm to run `true`. Both were vmbox waiting on
itself. Now 12s / 0s, with all five shutdown verdicts re-tested unchanged.
- _run_settle waited up to 30s for the kernel's "reboot: Power down" marker
UNCONDITIONALLY, including on the majority of runs that leave the guest up,
where that marker is never going to appear. It now settles only when the
guest has actually stopped, or when ssh returned 255 (what a successful
poweroff looks like from the host).
- The ssh readiness polls used a 5s ConnectTimeout. QEMU's user-mode
networking ACCEPTS the forwarded connection while the guest still has
nothing on :22, so each probe blocked for the full timeout instead of
failing fast: a guest reachable at ~10s was reported up at ~45s. Fixed in
both the sandbox and build paths, via an env override -- ssh honours the
FIRST value for an option, so a second -o could not have worked.
Making runs fast exposed a race: sshd now answers BEFORE tty1's autologin has
started Xorg, so the X environment could be missing on the first run after a
boot. `VMBOX_X_WAIT=<seconds>` waits for the session; it defaults to 0 so an
ordinary run never pays for a graphical session it is not using.
Build: the host's /var/cache/pacman/pkg is offered to the build guest
read-only (`readonly=on` AND an `ro` mount, because it is the real host cache)
as an extra CacheDir after the guest's own writable one, so a package the host
lacks still downloads rather than failing the build. This removes the 396 MiB
the build used to fetch. Host cache verified byte-for-byte identical across
four builds: same file count, same size, same mtime.
Build time itself moved only 103.8s -> 100.3s. Per-phase timings (also added
here) show why: 47s of it is the build guest's cloud-init first boot, and that
is NOT datasource probing -- `cloud-init analyze blame` totals 2.27s. Left
unexplained rather than guessed at; SESSION_RESULTS.md records the evidence.
Also: `vm scp`, which lib/ssh.sh could always do but the CLI never exposed,
and provision.sh split at the 250-line cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Testing boot_recovery produced a guest that cannot boot: boot-repair read
/boot/vmlinuz-linux and initramfs-linux.img on this BIOS/GRUB guest as
"orphaned kernel files shadowing the ESP" and deleted them, leaving the
sandbox at a GRUB prompt. That is a legitimate sandbox outcome -- destroying
the guest is what the tool is for -- but vmbox handled it by hanging.
Two independent unbounded waits, both fixed:
- vm_wait_ssh counted only its `sleep 2` calls, ignoring the up-to-5s
ConnectTimeout each failed probe spends inside ssh. A nominal 150s wait ran
past 400s (measured, then cut short by the harness). It now works to a wall
clock deadline: the same guest is now reported unreachable in 156s.
- _run_recover_rc reboots the stopped guest to read the recorded exit code,
and fell through to an unbounded serial console read: 180s for a prompt
that never comes plus 300s for a command that never runs, all AFTER the
verdict had already been printed. Recovery is best-effort by definition, so
both bounds are now short (60s/30s) and overridable.
serial_exec's timeouts become env-overridable to make that possible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
The image ships Xorg + i3 on tty1 precisely so locker and i3 tests can run,
but `vm run` executes through `sh -c`, which reads no profile. Every X tool
therefore landed with no DISPLAY and no XAUTHORITY and failed with
"Could not determine i3 socket path" -- indistinguishable from "this sandbox
has no X server". testsAndMisc's i3-configuration installer failed that way;
with the environment resolved it exits 0 and i3-msg reports {"success":true}.
XAUTHORITY is the awkward half: startx generates /tmp/serverauth.XXXXXXXX
with a fresh random suffix every boot, so it cannot be hardcoded. Resolve it
from the running Xorg's own -auth argument, falling back to a newest-first
glob.
The `[ -f ]` guard on the source line is load-bearing, not defensive
dressing. In POSIX sh, `.` on a missing file is a SPECIAL BUILTIN error that
exits the shell outright, and `|| true` does NOT rescue it -- written that
way, every `vm run` against a base image predating this snippet silently
produced no output at all. Found by doing exactly that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
`vm reset` killed the VM and immediately recreated the overlay, without
waiting for the process to actually exit. The next boot then read a
half-detached file and failed with "sandbox did not become reachable" -- a
symptom that appears one step AFTER the cause and says nothing about it.
Reset itself was always correct (verified by hand: after
`rm -rf --no-preserve-root /etc /usr`, a reset guest reports
NAME="Arch Linux" and has /usr/bin/bash back). This only bit when a reset
followed a still-shutting-down VM closely enough to race it, which is exactly
what the destructive demo does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Measured between two sandboxes: 3 packets transmitted, 3 received, 0% packet
loss, 0.23 ms average.
The previous commit called this broken. It was not: the dgram transport worked
all along, but the odd/even pairing computed the wrong peer port for even
indexes, so a VM pointed `remote.port` at a port nothing had bound. Both ends
came up, addresses were right, and zero frames crossed -- indistinguishable
from a dead transport, and nothing in any log said otherwise. The tell was
`ss -lunp`: the bound ports were 12349 and 12350 while the guest was sending
to 12348. The test then compounded it by using indexes 4 and 5, which belong
to DIFFERENT pairs (4 pairs with 3, 5 pairs with 6).
Pairing is (1,2), (3,4), (5,6)... Two sandboxes created back to back are wired
together, which is the common case.
mcast= remains genuinely unusable here and is documented as such so nobody
re-tries it: lo has no MULTICAST flag and there is no multicast route, so qemu
accepts it and drops every frame.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
The peer-address service read a `vmbox.peer=` kernel argument that the
launcher never passed -- the reader was written and the writer forgotten -- so
eth1 came up DOWN with no address on every guest. It now derives the address
from the NIC's own MAC (the host allocates 52:54:00:be:ef:<index>), which
removes the coupling entirely: nothing has to be passed in, so the launcher
and the guest cannot disagree about which index a VM has.
That fixed the addressing but NOT the connectivity. `mcast=` cannot work on
this host at all -- `lo` carries no MULTICAST flag and there is no multicast
route, so qemu accepts the option and silently drops every frame, which is the
same "flags accepted, nothing happens" trap as POWERDOWN. A listen=/connect=
socket hub over loopback TCP was tried next: the hub port listens, both
guests bring eth1 up with the right addresses, and ARP still fails.
Documented as a known limitation rather than papered over. Sandboxes work
fully on their own; two guests cannot talk to each other yet.
Also corrects the destructive demo: a running Linux SURVIVES `rm -rf /usr`
because processes hold their binaries through open file handles, so sshd keeps
answering. The honest proof of destruction is that the machine cannot boot
again, so the demo now reboots it and asserts it does not come back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
9p exports a directory TREE and does not follow a symlink pointing out of it.
The share directory held `testsAndMisc -> /home/kuhy/testsAndMisc`, so the
guest mounted /mnt/hostrepo successfully, listed the symlink, and then got
"No such file or directory" following it -- the target path does not exist
inside the guest. Measured in a live sandbox, not assumed.
`vm share <path>` now bind-mounts the directory into the share and remounts it
read-only, so the guest sees real contents while still being unable to write
to the host copy. `vm unshare <name>` reverses it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
`_run_exec "$name" "$@"` sat on its own line under `set -euo pipefail`, so a
non-zero return aborted the script before the verdict was ever computed. ssh
returns 255 when its connection dies -- which is exactly what a SUCCESSFUL
guest poweroff causes. The single case this tool exists to report on was the
one case that silently printed nothing.
Caught by the acceptance demo, not by the unit tests: verdict_report was
correct in isolation the whole time and passed 8/8 there. Only running the
real CLI against a real poweroff exposed the wiring.
Verified after the fix, end to end:
==> Running in 'fixtest': sudo systemctl poweroff
Connection to 127.0.0.1 closed by remote host.
ok VERDICT: clean poweroff -- guest powered itself off and reached
'reboot: Power down'
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Root cause, confirmed causally rather than inferred: masking
systemd-timesyncd (which vmbox does so `-rtc base=` stays pinned, since
timesyncd otherwise resets the guest clock to real time) left
systemd-time-wait-sync waiting for a sync that can never arrive. That unit sat
in "start running" forever and every other job queued behind it as "start
waiting" -- sshd included -- so the guest never reached multi-user.target. It
booted to a login prompt and looked perfectly healthy while being permanently
unreachable. The fix is one line: mask both units together.
Four earlier hypotheses were each disproven by measurement. What finally
worked was asking the guest instead of reading absence-of-log-lines:
`systemctl is-system-running` returned "starting", and `systemctl list-jobs`
put systemd-time-wait-sync at the head of the queue. Stopping it made sshd
active and bound :22 within seconds, and ssh from the host then succeeded.
Rule for next time: a unit that is enabled, has satisfied dependencies, and
neither starts nor fails is QUEUED -- check list-jobs first.
The smoke gate added earlier did its job here: it caught the unreachable image
and refused to seal it, keeping the sandbox so its logs survived.
Also fixes serial_exec markers to be plain alphanumerics -- shell
metacharacters get echoed and line-wrapped by the tty, which made them
unreliable to match and produced spurious timeouts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
The Arch cloud image leaves sshd enabled but never started -- no start line,
no failure line, dependencies satisfied. Four hypotheses (NTP, cloud-init
clean, a socket-disable, a blocking start) were each disproven by measurement,
so rather than keep paying a ~10 min rebuild per guess this adds the transport
the tool should arguably have used all along.
For the case vmbox exists for, serial beats ssh on the merits: it keeps
printing straight through the poweroff that kills an ssh connection, and it
works when sshd does not. `vm run` now picks ssh when it answers and the
serial console otherwise, and recovers the guest exit code either way. Guests
get a second serial port (ttyS1) on a unix socket with a root autologin, kept
separate from ttyS0 so driving a shell never interleaves with the append-only
log the verdict is read from.
serial_exec.py waits for an explicit per-command end marker rather than a
fixed sleep. That is not a detail: an earlier probe used a 4s window, the next
command was typed while the previous one still ran, and the tty echoed that
input -- which reads exactly like output in a transcript and produced a
confident but wrong conclusion. It also strips ANSI CSI and systemd's OSC 3008
shell-integration blocks, verified against a real captured boot log.
Also: a vmbox-sshd-kick unit that forces sshd up after multi-user.target as
belt and braces, and the smoke-test sandbox renamed to a name that actually
passes validate_vm_name -- "__smoke" was rejected, so the gate silently never
ran.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
Much of testsAndMisc/linux_configuration cannot be tested on the host: 15
files use `chattr +i` (so a bad test is not simply revertable), the shutdown
installers write units that power the machine off, and boot_recovery installs
pacman hooks. The repo's own CI says as much -- its Ubuntu job runs only the
side-effect-free tests, noting the rest "drive pacman, /etc, systemd or adb".
vmbox runs them in a throwaway Arch guest and verifies the result from the
HOST, which is the part that is not obvious: a successful shutdown kills the
connection carrying the report, and ssh returns 255 for a clean poweroff, a
crash and a dropped link alike. So the verdict comes from two artefacts that
outlive the VM -- a QMP event log written by a recorder attached at launch
(attach later and qemu has already exited), and the serial console log.
Five outcomes are distinguished: clean poweroff (SHUTDOWN guest:true plus a
serial tail reaching "reboot: Power down"), dirty stop, guest-reset (rebooted
instead of powering off), GUEST_PANICKED, and hung. A POWERDOWN event proves
nothing -- it means the request was delivered, not that anything acted on it.
Reset is a qcow2 backing-file trick: the golden image is built once and sealed
(chmod 444 plus a sha256 sidecar verified on every launch, since booting the
base would silently invalidate every overlay), and each sandbox is a thin
overlay, so reset is rm + recreate. No root anywhere -- user-mode networking
with per-VM forwarded ssh ports, and a socket multicast segment for VM-to-VM.
The guest clock is pinned and re-applied on every launch, because the shutdown
installers gate on a 21:00-05:00 window; NTP is masked in the image or it
silently undoes the pin. Repos are cloned from a read-only share scoped to an
explicit directory, never $HOME, and credentials are sentinel values.
`vm build` ends by booting a throwaway sandbox off the sealed image and
refusing to ship if ssh does not answer. Several distinct faults produced an
identical "boots fine, connection reset" symptom; checks inside the build VM
only prove state at seal time, so the gate has to be an end-to-end boot.
Tests: 8 bats cases over the verdict table and name/meta/serial handling;
shellcheck and ruff clean; every file within the 250-line cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VdyMBGCrGVGDWtJ9xe48f3
